Abstract
DevOps is a new approach to drive a seamless Application (App) cycle from development to delivery. As a critical part to promote the successful implementation of DevOps, testing can significantly improve team productivity and reliably deliver user experience. However, it is difficult to use traditional testing to cover diverse mobile phones, network environments, operating systems, and so on. Hence, many large companies crowdsource their App testing tasks to workers from open platforms. In crowdsourced testing, test reports submitted by workers may be highly redundant, and their quality may vary sharply. Meanwhile, multi-bug test reports may be submitted, and their root causes are hard to diagnose. Hence, it is a time-consuming and tedious task for developers to manually inspect these test reports. To help developers address the above challenges, we issue the new problem of Fuzzy Clustering Test Reports (FULTER). Aiming to resolve FULTER, a series of barriers need to be overcome. In this study, we propose a new framework named Test Report Fuzzy Clustering Framework (TERFUR) by aggregating redundant and multi-bug test reports into clusters to reduce the number of inspected test reports. First, we construct a filter to remove invalid test reports to break through the invalid barrier. Then, a preprocessor is built to enhance the descriptions of short test reports to break through the uneven barrier. Last, a two-phase merging algorithm is proposed to partition redundant and multi-bug test reports into clusters that can break through the multi-bug barrier. Experimental results over 1,728 test reports from five industrial Apps show that TERFUR can cluster test reports by up to 78.15% in terms of AverageP, 78.41% in terms of AverageR, and 75.82% in terms of AverageF1 and outperform comparative methods by up to 31.69%, 33.06%, and 24.55%, respectively. In addition, the effectiveness of TERFUR is validated in prioritizing test reports for manual inspection.
- Armin Balalaie, Abbas Heydarnoori, and Pooyan Jamshidi. 2015. Migrating to cloud-native architectures using microservices: An experience report. In Proceedings of the Advances in Service-Oriented and Cloud Computing Workshops (ESOCC’15). Springer, 201--215.Google Scholar
- Mauro Barni, Vito Cappellini, and Alessandro Mecocci. 1996. Comments on “A possibilistic approach to clusterin.” IEEE Trans. Fuzzy Syst. 4, 3 (1996), 393--396. Google Scholar
Digital Library
- Kevin Bartz, Jack W. Stokes, John C. Platt, Ryan Kivett, David Grant, Silviu Calinoiu, and Gretchen Loihle. 2008. Finding similar failures using callstack similarity. In Proceedings of the 3rd Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SysML’08). USENIX Association, San Diego, CA. Google Scholar
Digital Library
- Pavel Berkhin. 2006. A survey of clustering data mining techniques. In Grouping Multidimensional Data—Recent Advances in Clustering. 25--71.Google Scholar
- James C. Bezdek. 2013. Pattern Recognition with Fuzzy Objective Function Algorithms. Springer Science 8 Business Media.Google Scholar
- Gloria Bordogna and Gabriella Pasi. 2009. Hierarchical-hyperspherical divisive fuzzy C-means (H2D-FCM) clustering for information retrieval. In Proceedings of the 2009 IEEE/WIC/ACM International Conference on Web Intelligence (WI’09). IEEE Computer Society, 614--621. Google Scholar
Digital Library
- Yguaratã Cerqueira Cavalcanti, Eduardo Santana de Almeida, Carlos Eduardo Albuquerque da Cunha, Daniel Lucrédio, and Silvio Romero de Lemos Meira. 2010. An initial study on the bug report duplication problem. In Proceedings of the 14th European Conference on Software Maintenance and Reengineering (CSMR’10). IEEE Computer Society, 264--267. Google Scholar
Digital Library
- Zhenyu Chen and Bin Luo. 2014. Quasi-crowdsourcing testing for educational projects. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE’14). ACM, 272--275. Google Scholar
Digital Library
- I-Jen Chiang, Charles Chih-Ho Liu, Yi-Hsin Tsai, and Ajit Kumar. 2015. Discovering latent semantics in web documents using fuzzy clustering. IEEE Trans. Fuzzy Syst. 23, 6 (2015), 2122--2134.Google Scholar
Cross Ref
- Hung Chim and Xiaotie Deng. 2008. Efficient phrase-based document similarity for clustering. IEEE Trans. Knowl. Data Eng. 20, 9 (2008), 1217--1229. Google Scholar
Digital Library
- Yingnong Dang, Rongxin Wu, Hongyu Zhang, Dongmei Zhang, and Peter Nobel. 2012. ReBucket: A method for clustering duplicate crash reports based on call stack similarity. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE Computer Society, 1084--1093. Google Scholar
Digital Library
- Eelco Dolstra, Raynor Vliegendhart, and Johan A. Pouwelse. 2013. Crowdsourcing GUI tests. In Proceedings of the 6th IEEE International Conference on Software Testing, Verification and Validation (ICST’13). IEEE Computer Society, 332--341. Google Scholar
Digital Library
- Yang Feng, Zhenyu Chen, James A. Jones, Chunrong Fang, and Baowen Xu. 2015. Test report prioritization to assist crowdsourced testing. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’15). ACM, 225--236. Google Scholar
Digital Library
- Yang Feng, James A. Jones, Zhenyu Chen, and Chunrong Fang. 2016. Multi-objective test report prioritization using image understanding. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 202--213. Google Scholar
Digital Library
- Fabio Guaiani and Henry Muccini. 2015. Crowd and laboratory testing, can they co-exist? An exploratory study. In Proceedings of the 2nd IEEE/ACM International Workshop on CrowdSourcing in Software Engineering (CSI-SE’15). IEEE Computer Society, 32--37. Google Scholar
Digital Library
- Khaled M. Hammouda and Mohamed S. Kamel. 2004. Efficient phrase-based document indexing for web document clustering. IEEE Trans. Knowl. Data Eng. 16, 10 (2004), 1279--1296. Google Scholar
Digital Library
- Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. 14, 6 (2006), 1--4.Google Scholar
- Michael Httermann. 2012. DevOps for Developers. Apress. Google Scholar
Digital Library
- A. K. Jain, M. N. Murty, and P. J. Flynn. 1999. Data clustering: A review. ACM Comput. Surv. 31, 3 (1999), 264--323. Google Scholar
Digital Library
- Jung-Yi Jiang, Ren-Jia Liou, and Shie-Jue Lee. 2011. A fuzzy self-constructing feature clustering algorithm for text classification. IEEE Trans. Knowl. Data Eng. 23, 3 (2011), 335--349. Google Scholar
Digital Library
- Sunghun Kim, Thomas Zimmermann, and Nachiappan Nagappan. 2011. Crash graphs: An aggregated view of multiple crashes to improve crash triage. In Proceedings of the 2011 IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’11). IEEE Compute Society, 486--493. Google Scholar
Digital Library
- B. Kirubakaran and Dr. V. Karthikeyani. 2013. Mobile application testing - Challenges and solution approach through automation. In Proceedings of the 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME’13). IEEE Computer Society.Google Scholar
Cross Ref
- Ravikumar Kondadadi and Robert Kozma. 2002. A modified fuzzy art for soft document clustering. In Proceedings of the 2002 International Joint Conference on Neural Networks (IJCNN’02). IEEE Computer Society, 2545--2549.Google Scholar
Cross Ref
- Raghu Krishnapuram, Anupam Joshi, and Liyu Yi. 1999. A fuzzy relative of the K-medoids algorithm with application to web document and snippet clustering. In Proceedings of the IEEE InternationalFuzzy Systems Conference Proceedings (FUZZ-IEEE’99). IEEE Computer Society, 1281--1286.Google Scholar
Cross Ref
- Yves Ledru, Alexandre Petrenko, and Sergiy Boroday. 2009. Using string distances for test case prioritisation. In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE’09). IEEE Computer Society, 510--514. Google Scholar
Digital Library
- Yanjun Li, Soon M. Chung, and John D. Holt. 2008. Text document clustering based on frequent word meaning sequences. Data Knowl. Eng. 64, 1 (2008), 381--404. Google Scholar
Digital Library
- Yanjun Li, Congnan Luo, and Soon M. Chung. 2008. Text clustering with feature selection by using statistical data. IEEE Trans. Knowl. Data Eng. 20, 5 (2008), 641--652. Google Scholar
Digital Library
- King-Ip Lin and Ravikumar Kondadadi. 2001. A similarity-based soft clustering algorithm for documents. In Proceedings of the 7th International Conference on Database Systems for Advanced Applications (DASFAA’01). IEEE Computer Society, 40--47. Google Scholar
Digital Library
- Di Liu, Randolph G. Bias, Matthew Lease, and Rebecca Kuipers. 2012. Crowdsourcing for usability testing. Proc. Am. Soc. Inf. Sci. Technol. 49, 1 (2012), 1--10.Google Scholar
Cross Ref
- Christopher D. Manning and Hinrich Schütze. 2001. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge. Google Scholar
Digital Library
- Ke Mao, Licia Capra, Mark Harman, and Yue Jia. 2017. A survey of the use of crowdsourcing in software engineering. Journal of Systems and Software 126 (2017), 57--84.Google Scholar
Cross Ref
- Louis Massey. 2005. Evaluating and comparing text clustering results. In Proceedings of the International Conference on Computational Intelligence. IASTED/ACTA Press, Alberta, Canada, 85--90.Google Scholar
- M. E. S. Mendes and Lionel Sacks. 2004. Dynamic knowledge representation for E-learning applications. In Enhancing the Power of the Internet. Springer, 259--282.Google Scholar
- Natwar Modani, Rajeev Gupta, Guy M. Lohman, Tanveer Fathima Syeda-Mahmood, and Laurent Mignet. 2007. Automatically identifying known software problems. In Proceedings of the 23rd International Conference on Data Engineering Workshops (ICDE’07). IEEE Computer Society, 433--441. Google Scholar
Digital Library
- Michael Nebeling, Maximilian Speicher, Michael Grossniklaus, and Moira C. Norrie. 2012. Crowdsourced web site evaluation with crowdstudy. In Proceedings of 12th International Conference on Web Engineering (ICWE’12). Springer, 494--497. Google Scholar
Digital Library
- Anh Tuan Nguyen, Tung Thanh Nguyen, Tien N. Nguyen, David Lo, and Chengnian Sun. 2012. Duplicate bug report detection with a combination of information retrieval and topic modeling. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE’12). ACM, 70--79. Google Scholar
Digital Library
- Rahul R. Papalkar and G. Chandel. 2013. Fuzzy clustering in web text mining and its application in IEEE abstract classification. Int. J. Comput. Sci. Manage. Rese. 2, 2 (2013), 1529--1533.Google Scholar
- Fabrizio Pastore, Leonardo Mariani, and Gordon Fraser. 2013. CrowdOracles: Can the crowd solve the oracle problem? In Proceedings of the 6th IEEE International Conference on Software Testing, Verification and Validation (ICST’13). IEEE Computer Society, 342--351. Google Scholar
Digital Library
- Andy Podgurski, David Leon, Patrick Francis, Wes Masri, Melinda Minch, Jiayang Sun, and Bin Wang. 2003. Automated support for classifying software failure reports. In Proceedings of the 25th International Conference on Software Engineering (ICSE’03). IEEE Compute Society, 465--477. Google Scholar
Digital Library
- M. E. S. Mendes Rodrigues and L. Sacks. 2004. A scalable hierarchical fuzzy clustering algorithm for text mining. In Proceedings of the 5th International Conference on Recent Advances in Soft Computing (RASC’04). IEEE Computer Society, 269--274.Google Scholar
- Guoping Rong, He Zhang, and Dong Shao. 2016. CMMI guided process improvement for devops projects: An exploratory case study. In Proceedings of the International Conference on Software and Systems Process (ICSSP’16). ACM, 76--85. Google Scholar
Digital Library
- Dmitri Roussinov and Hsinchun Chen. 1999. Document clustering for electronic meetings: An experimental comparison of two techniques. Decis. Support Syst. 27, 1--2 (1999), 67--79. Google Scholar
Digital Library
- Per Runeson, Magnus Alexandersson, and Oskar Nyholm. 2007. Detection of duplicate defect reports using natural language processing. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07). IEEE Computer Society, 499--510. Google Scholar
Digital Library
- Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo Juzgado. 2015. Are students representatives of professionals in software engineering experiments? In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE’15). IEEE Computer Society, 666--676. Google Scholar
Digital Library
- Andrew Skabar and Khaled Abdalgader. 2013. Clustering sentence-level text using a novel fuzzy relational clustering algorithm. IEEE Trans. Knowl. Data Eng. 25, 1 (2013), 62--75. Google Scholar
Digital Library
- Chengnian Sun, David Lo, Siau-Cheng Khoo, and Jing Jiang. 2011. Towards more accurate retrieval of duplicate bug reports. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE’11). IEEE Computer Society, 253--262. Google Scholar
Digital Library
- Chengnian Sun, David Lo, Xiaoyin Wang, Jing Jiang, and Siau-Cheng Khoo. 2010. A discriminative model approach for accurate duplicate bug report retrieval. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering, Volume 1 (ICSE’10). ACM, 45--54. Google Scholar
Digital Library
- Amy J. C. Trappey, Charles V. Trappey, Fu-Chiang Hsu, and David W. Hsiao. 2009. A fuzzy ontological knowledge document clustering methodology. IEEE Trans. Syst. Man Cybernet. B 39, 3 (2009), 806--814. Google Scholar
Digital Library
- Junjie Wang, Qiang Cui, Qing Wang, and Song Wang. 2016. Towards effectively test report classification to assist crowdsourced testing. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’16). 6:1--6:10. Google Scholar
Digital Library
- Junjie Wang, Song Wang, Qiang Cui, and Qing Wang. 2016. Local-based active classification of test report to assist crowdsourced testing. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). 190--201. Google Scholar
Digital Library
- Xiaoyin Wang, Lu Zhang, Tao Xie, John Anvik, and Jiasu Sun. 2008. An approach to detecting duplicate bug reports using natural language and execution information. In Proceedings of the 30th International Conference on Software Engineering (ICSE’08). ACM, 461--470. Google Scholar
Digital Library
- Johannes Wettinger, Vasilios Andrikopoulos, and Frank Leymann. 2015. Automated capturing and systematic usage of devops knowledge for cloud applications. In Proceedings of the IEEE International Conference on Cloud Engineering (IC2E’15). IEEE Computer Society, 60--65. Google Scholar
Digital Library
- Thaung Thaung Win and LinMon. 2010. Document clustering by fuzzy C-mean algorithm. In Proceedings of the 34th International Conference and Exposition on Advanced Ceramics and Composites (ICACC’10). IEEE Computer Society, 239--242.Google Scholar
- Jinglin Xu, Junwei Han, Kai Xiong, and Feiping Nie. 2016. Robust and sparse fuzzy K-means clustering. In 25th Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’2016). IJCAI/AAAI Press, New York, 2224--2230. Google Scholar
Digital Library
- Jifeng Xuan, He Jiang, Yan Hu, Zhilei Ren, Weiqin Zou, Zhongxuan Luo, and Xindong Wu. 2015. Towards effective bug triage with software data reduction techniques. IEEE Trans. Knowl. Data Eng. 27, 1 (2015), 264--280.Google Scholar
Cross Ref
- Jifeng Xuan, He Jiang, Zhilei Ren, Jun Yan, and Zhongxuan Luo. 2010. Automatic bug triage using semi-supervised text classification. In Proceedings of the 22nd International Conference on Software Engineering 8 Knowledge Engineering (SEKE’10). Knowledge Systems Institute Graduate School, Zurich, Switzerland, 209--214.Google Scholar
- Jifeng Xuan, He Jiang, Zhilei Ren, and Weiqin Zou. 2012. Developer prioritization in bug repositories. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE Computer Society, 25--35. Google Scholar
Digital Library
- Rongrong Zhang, Qingtian Zeng, and Sen Feng. 2010. Data query using short domain question in natural language. In Proceedings of the 2010 IEEE 2nd Symposium on Web Society (SWS’10). IEEE Computer Society, 351--354.Google Scholar
Cross Ref
- Wen Zhang, Taketoshi Yoshida, Xijin Tang, and Qing Wang. 2010. Text clustering using frequent itemsets. Knowl.-Based Syst. 23, 5 (2010), 379--388. Google Scholar
Digital Library
Index Terms
Fuzzy Clustering of Crowdsourced Test Reports for Apps
Recommendations
CTRAS: a tool for aggregating and summarizing crowdsourced test reports
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and AnalysisIn this paper, we present CTRAS, a tool for automatically aggregating and summarizing duplicate crowdsourced test reports on the fly. CTRAS can automatically detect duplicates based on both textual information and the screenshots, and further aggregates ...
Test report prioritization to assist crowdsourced testing
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software EngineeringIn crowdsourced testing, users can be incentivized to perform testing tasks and report their results, and because crowdsourced workers are often paid per task, there is a financial incentive to complete tasks quickly rather than well. These reports of ...
Towards Effectively Test Report Classification to Assist Crowdsourced Testing
ESEM '16: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and MeasurementContext: Automatic classification of crowdsourced test reports is important due to their tremendous sizes and large proportion of noises. Most existing approaches towards this problem focus on examining the performance of different machine learning or ...






Comments