skip to main content
research-article

Fuzzy Clustering of Crowdsourced Test Reports for Apps

Authors Info & Claims
Published:02 February 2018Publication History
Skip Abstract Section

Abstract

DevOps is a new approach to drive a seamless Application (App) cycle from development to delivery. As a critical part to promote the successful implementation of DevOps, testing can significantly improve team productivity and reliably deliver user experience. However, it is difficult to use traditional testing to cover diverse mobile phones, network environments, operating systems, and so on. Hence, many large companies crowdsource their App testing tasks to workers from open platforms. In crowdsourced testing, test reports submitted by workers may be highly redundant, and their quality may vary sharply. Meanwhile, multi-bug test reports may be submitted, and their root causes are hard to diagnose. Hence, it is a time-consuming and tedious task for developers to manually inspect these test reports. To help developers address the above challenges, we issue the new problem of Fuzzy Clustering Test Reports (FULTER). Aiming to resolve FULTER, a series of barriers need to be overcome. In this study, we propose a new framework named Test Report Fuzzy Clustering Framework (TERFUR) by aggregating redundant and multi-bug test reports into clusters to reduce the number of inspected test reports. First, we construct a filter to remove invalid test reports to break through the invalid barrier. Then, a preprocessor is built to enhance the descriptions of short test reports to break through the uneven barrier. Last, a two-phase merging algorithm is proposed to partition redundant and multi-bug test reports into clusters that can break through the multi-bug barrier. Experimental results over 1,728 test reports from five industrial Apps show that TERFUR can cluster test reports by up to 78.15% in terms of AverageP, 78.41% in terms of AverageR, and 75.82% in terms of AverageF1 and outperform comparative methods by up to 31.69%, 33.06%, and 24.55%, respectively. In addition, the effectiveness of TERFUR is validated in prioritizing test reports for manual inspection.

References

  1. Armin Balalaie, Abbas Heydarnoori, and Pooyan Jamshidi. 2015. Migrating to cloud-native architectures using microservices: An experience report. In Proceedings of the Advances in Service-Oriented and Cloud Computing Workshops (ESOCC’15). Springer, 201--215.Google ScholarGoogle Scholar
  2. Mauro Barni, Vito Cappellini, and Alessandro Mecocci. 1996. Comments on “A possibilistic approach to clusterin.” IEEE Trans. Fuzzy Syst. 4, 3 (1996), 393--396. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Kevin Bartz, Jack W. Stokes, John C. Platt, Ryan Kivett, David Grant, Silviu Calinoiu, and Gretchen Loihle. 2008. Finding similar failures using callstack similarity. In Proceedings of the 3rd Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SysML’08). USENIX Association, San Diego, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Pavel Berkhin. 2006. A survey of clustering data mining techniques. In Grouping Multidimensional Data—Recent Advances in Clustering. 25--71.Google ScholarGoogle Scholar
  5. James C. Bezdek. 2013. Pattern Recognition with Fuzzy Objective Function Algorithms. Springer Science 8 Business Media.Google ScholarGoogle Scholar
  6. Gloria Bordogna and Gabriella Pasi. 2009. Hierarchical-hyperspherical divisive fuzzy C-means (H2D-FCM) clustering for information retrieval. In Proceedings of the 2009 IEEE/WIC/ACM International Conference on Web Intelligence (WI’09). IEEE Computer Society, 614--621. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Yguaratã Cerqueira Cavalcanti, Eduardo Santana de Almeida, Carlos Eduardo Albuquerque da Cunha, Daniel Lucrédio, and Silvio Romero de Lemos Meira. 2010. An initial study on the bug report duplication problem. In Proceedings of the 14th European Conference on Software Maintenance and Reengineering (CSMR’10). IEEE Computer Society, 264--267. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Zhenyu Chen and Bin Luo. 2014. Quasi-crowdsourcing testing for educational projects. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE’14). ACM, 272--275. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. I-Jen Chiang, Charles Chih-Ho Liu, Yi-Hsin Tsai, and Ajit Kumar. 2015. Discovering latent semantics in web documents using fuzzy clustering. IEEE Trans. Fuzzy Syst. 23, 6 (2015), 2122--2134.Google ScholarGoogle ScholarCross RefCross Ref
  10. Hung Chim and Xiaotie Deng. 2008. Efficient phrase-based document similarity for clustering. IEEE Trans. Knowl. Data Eng. 20, 9 (2008), 1217--1229. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Yingnong Dang, Rongxin Wu, Hongyu Zhang, Dongmei Zhang, and Peter Nobel. 2012. ReBucket: A method for clustering duplicate crash reports based on call stack similarity. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE Computer Society, 1084--1093. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Eelco Dolstra, Raynor Vliegendhart, and Johan A. Pouwelse. 2013. Crowdsourcing GUI tests. In Proceedings of the 6th IEEE International Conference on Software Testing, Verification and Validation (ICST’13). IEEE Computer Society, 332--341. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Yang Feng, Zhenyu Chen, James A. Jones, Chunrong Fang, and Baowen Xu. 2015. Test report prioritization to assist crowdsourced testing. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’15). ACM, 225--236. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Yang Feng, James A. Jones, Zhenyu Chen, and Chunrong Fang. 2016. Multi-objective test report prioritization using image understanding. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 202--213. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Fabio Guaiani and Henry Muccini. 2015. Crowd and laboratory testing, can they co-exist? An exploratory study. In Proceedings of the 2nd IEEE/ACM International Workshop on CrowdSourcing in Software Engineering (CSI-SE’15). IEEE Computer Society, 32--37. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Khaled M. Hammouda and Mohamed S. Kamel. 2004. Efficient phrase-based document indexing for web document clustering. IEEE Trans. Knowl. Data Eng. 16, 10 (2004), 1279--1296. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. 14, 6 (2006), 1--4.Google ScholarGoogle Scholar
  18. Michael Httermann. 2012. DevOps for Developers. Apress. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. A. K. Jain, M. N. Murty, and P. J. Flynn. 1999. Data clustering: A review. ACM Comput. Surv. 31, 3 (1999), 264--323. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Jung-Yi Jiang, Ren-Jia Liou, and Shie-Jue Lee. 2011. A fuzzy self-constructing feature clustering algorithm for text classification. IEEE Trans. Knowl. Data Eng. 23, 3 (2011), 335--349. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Sunghun Kim, Thomas Zimmermann, and Nachiappan Nagappan. 2011. Crash graphs: An aggregated view of multiple crashes to improve crash triage. In Proceedings of the 2011 IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’11). IEEE Compute Society, 486--493. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. B. Kirubakaran and Dr. V. Karthikeyani. 2013. Mobile application testing - Challenges and solution approach through automation. In Proceedings of the 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME’13). IEEE Computer Society.Google ScholarGoogle ScholarCross RefCross Ref
  23. Ravikumar Kondadadi and Robert Kozma. 2002. A modified fuzzy art for soft document clustering. In Proceedings of the 2002 International Joint Conference on Neural Networks (IJCNN’02). IEEE Computer Society, 2545--2549.Google ScholarGoogle ScholarCross RefCross Ref
  24. Raghu Krishnapuram, Anupam Joshi, and Liyu Yi. 1999. A fuzzy relative of the K-medoids algorithm with application to web document and snippet clustering. In Proceedings of the IEEE InternationalFuzzy Systems Conference Proceedings (FUZZ-IEEE’99). IEEE Computer Society, 1281--1286.Google ScholarGoogle ScholarCross RefCross Ref
  25. Yves Ledru, Alexandre Petrenko, and Sergiy Boroday. 2009. Using string distances for test case prioritisation. In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE’09). IEEE Computer Society, 510--514. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Yanjun Li, Soon M. Chung, and John D. Holt. 2008. Text document clustering based on frequent word meaning sequences. Data Knowl. Eng. 64, 1 (2008), 381--404. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Yanjun Li, Congnan Luo, and Soon M. Chung. 2008. Text clustering with feature selection by using statistical data. IEEE Trans. Knowl. Data Eng. 20, 5 (2008), 641--652. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. King-Ip Lin and Ravikumar Kondadadi. 2001. A similarity-based soft clustering algorithm for documents. In Proceedings of the 7th International Conference on Database Systems for Advanced Applications (DASFAA’01). IEEE Computer Society, 40--47. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Di Liu, Randolph G. Bias, Matthew Lease, and Rebecca Kuipers. 2012. Crowdsourcing for usability testing. Proc. Am. Soc. Inf. Sci. Technol. 49, 1 (2012), 1--10.Google ScholarGoogle ScholarCross RefCross Ref
  30. Christopher D. Manning and Hinrich Schütze. 2001. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Ke Mao, Licia Capra, Mark Harman, and Yue Jia. 2017. A survey of the use of crowdsourcing in software engineering. Journal of Systems and Software 126 (2017), 57--84.Google ScholarGoogle ScholarCross RefCross Ref
  32. Louis Massey. 2005. Evaluating and comparing text clustering results. In Proceedings of the International Conference on Computational Intelligence. IASTED/ACTA Press, Alberta, Canada, 85--90.Google ScholarGoogle Scholar
  33. M. E. S. Mendes and Lionel Sacks. 2004. Dynamic knowledge representation for E-learning applications. In Enhancing the Power of the Internet. Springer, 259--282.Google ScholarGoogle Scholar
  34. Natwar Modani, Rajeev Gupta, Guy M. Lohman, Tanveer Fathima Syeda-Mahmood, and Laurent Mignet. 2007. Automatically identifying known software problems. In Proceedings of the 23rd International Conference on Data Engineering Workshops (ICDE’07). IEEE Computer Society, 433--441. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Michael Nebeling, Maximilian Speicher, Michael Grossniklaus, and Moira C. Norrie. 2012. Crowdsourced web site evaluation with crowdstudy. In Proceedings of 12th International Conference on Web Engineering (ICWE’12). Springer, 494--497. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Anh Tuan Nguyen, Tung Thanh Nguyen, Tien N. Nguyen, David Lo, and Chengnian Sun. 2012. Duplicate bug report detection with a combination of information retrieval and topic modeling. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE’12). ACM, 70--79. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Rahul R. Papalkar and G. Chandel. 2013. Fuzzy clustering in web text mining and its application in IEEE abstract classification. Int. J. Comput. Sci. Manage. Rese. 2, 2 (2013), 1529--1533.Google ScholarGoogle Scholar
  38. Fabrizio Pastore, Leonardo Mariani, and Gordon Fraser. 2013. CrowdOracles: Can the crowd solve the oracle problem? In Proceedings of the 6th IEEE International Conference on Software Testing, Verification and Validation (ICST’13). IEEE Computer Society, 342--351. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Andy Podgurski, David Leon, Patrick Francis, Wes Masri, Melinda Minch, Jiayang Sun, and Bin Wang. 2003. Automated support for classifying software failure reports. In Proceedings of the 25th International Conference on Software Engineering (ICSE’03). IEEE Compute Society, 465--477. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. M. E. S. Mendes Rodrigues and L. Sacks. 2004. A scalable hierarchical fuzzy clustering algorithm for text mining. In Proceedings of the 5th International Conference on Recent Advances in Soft Computing (RASC’04). IEEE Computer Society, 269--274.Google ScholarGoogle Scholar
  41. Guoping Rong, He Zhang, and Dong Shao. 2016. CMMI guided process improvement for devops projects: An exploratory case study. In Proceedings of the International Conference on Software and Systems Process (ICSSP’16). ACM, 76--85. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Dmitri Roussinov and Hsinchun Chen. 1999. Document clustering for electronic meetings: An experimental comparison of two techniques. Decis. Support Syst. 27, 1--2 (1999), 67--79. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Per Runeson, Magnus Alexandersson, and Oskar Nyholm. 2007. Detection of duplicate defect reports using natural language processing. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07). IEEE Computer Society, 499--510. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo Juzgado. 2015. Are students representatives of professionals in software engineering experiments? In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE’15). IEEE Computer Society, 666--676. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Andrew Skabar and Khaled Abdalgader. 2013. Clustering sentence-level text using a novel fuzzy relational clustering algorithm. IEEE Trans. Knowl. Data Eng. 25, 1 (2013), 62--75. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Chengnian Sun, David Lo, Siau-Cheng Khoo, and Jing Jiang. 2011. Towards more accurate retrieval of duplicate bug reports. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE’11). IEEE Computer Society, 253--262. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Chengnian Sun, David Lo, Xiaoyin Wang, Jing Jiang, and Siau-Cheng Khoo. 2010. A discriminative model approach for accurate duplicate bug report retrieval. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering, Volume 1 (ICSE’10). ACM, 45--54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Amy J. C. Trappey, Charles V. Trappey, Fu-Chiang Hsu, and David W. Hsiao. 2009. A fuzzy ontological knowledge document clustering methodology. IEEE Trans. Syst. Man Cybernet. B 39, 3 (2009), 806--814. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Junjie Wang, Qiang Cui, Qing Wang, and Song Wang. 2016. Towards effectively test report classification to assist crowdsourced testing. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’16). 6:1--6:10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Junjie Wang, Song Wang, Qiang Cui, and Qing Wang. 2016. Local-based active classification of test report to assist crowdsourced testing. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 190--201. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Xiaoyin Wang, Lu Zhang, Tao Xie, John Anvik, and Jiasu Sun. 2008. An approach to detecting duplicate bug reports using natural language and execution information. In Proceedings of the 30th International Conference on Software Engineering (ICSE’08). ACM, 461--470. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Johannes Wettinger, Vasilios Andrikopoulos, and Frank Leymann. 2015. Automated capturing and systematic usage of devops knowledge for cloud applications. In Proceedings of the IEEE International Conference on Cloud Engineering (IC2E’15). IEEE Computer Society, 60--65. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Thaung Thaung Win and LinMon. 2010. Document clustering by fuzzy C-mean algorithm. In Proceedings of the 34th International Conference and Exposition on Advanced Ceramics and Composites (ICACC’10). IEEE Computer Society, 239--242.Google ScholarGoogle Scholar
  54. Jinglin Xu, Junwei Han, Kai Xiong, and Feiping Nie. 2016. Robust and sparse fuzzy K-means clustering. In 25th Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’2016). IJCAI/AAAI Press, New York, 2224--2230. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Jifeng Xuan, He Jiang, Yan Hu, Zhilei Ren, Weiqin Zou, Zhongxuan Luo, and Xindong Wu. 2015. Towards effective bug triage with software data reduction techniques. IEEE Trans. Knowl. Data Eng. 27, 1 (2015), 264--280.Google ScholarGoogle ScholarCross RefCross Ref
  56. Jifeng Xuan, He Jiang, Zhilei Ren, Jun Yan, and Zhongxuan Luo. 2010. Automatic bug triage using semi-supervised text classification. In Proceedings of the 22nd International Conference on Software Engineering 8 Knowledge Engineering (SEKE’10). Knowledge Systems Institute Graduate School, Zurich, Switzerland, 209--214.Google ScholarGoogle Scholar
  57. Jifeng Xuan, He Jiang, Zhilei Ren, and Weiqin Zou. 2012. Developer prioritization in bug repositories. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE Computer Society, 25--35. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Rongrong Zhang, Qingtian Zeng, and Sen Feng. 2010. Data query using short domain question in natural language. In Proceedings of the 2010 IEEE 2nd Symposium on Web Society (SWS’10). IEEE Computer Society, 351--354.Google ScholarGoogle ScholarCross RefCross Ref
  59. Wen Zhang, Taketoshi Yoshida, Xijin Tang, and Qing Wang. 2010. Text clustering using frequent itemsets. Knowl.-Based Syst. 23, 5 (2010), 379--388. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Fuzzy Clustering of Crowdsourced Test Reports for Apps

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Internet Technology
        ACM Transactions on Internet Technology  Volume 18, Issue 2
        Special Issue on Internetware and Devops and Regular Papers
        May 2018
        294 pages
        ISSN:1533-5399
        EISSN:1557-6051
        DOI:10.1145/3182619
        • Editor:
        • Munindar P. Singh
        Issue’s Table of Contents

        Copyright © 2018 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 2 February 2018
        • Accepted: 1 June 2017
        • Revised: 1 February 2017
        • Received: 1 September 2016
        Published in toit Volume 18, Issue 2

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!