ABSTRACT
Commenting source code has long been a common practice in software development. Compared to source code, comments are more direct, descriptive and easy-to-understand. Comments and sourcecode provide relatively redundant and independent information regarding a program's semantic behavior. As software evolves, they can easily grow out-of-sync, indicating two problems: (1) bugs -the source code does not follow the assumptions and requirements specified by correct program comments; (2) bad comments - comments that are inconsistent with correct code, which can confuse and mislead programmers to introduce bugs in subsequent versions. Unfortunately, as most comments are written in natural language, no solution has been proposed to automatically analyze commentsand detect inconsistencies between comments and source code. This paper takes the first step in automatically analyzing commentswritten in natural language to extract implicit program rulesand use these rules to automatically detect inconsistencies between comments and source code, indicating either bugs or bad comments. Our solution, iComment, combines Natural Language Processing(NLP), Machine Learning, Statistics and Program Analysis techniques to achieve these goals. We evaluate iComment on four large code bases: Linux, Mozilla, Wine and Apache. Our experimental results show that iComment automatically extracts 1832 rules from comments with 90.8-100% accuracy and detects 60 comment-code inconsistencies, 33 newbugs and 27 bad comments, in the latest versions of the four programs. Nineteen of them (12 bugs and 7 bad comments) have already been confirmed by the corresponding developers while the others are currently being analyzed by the developers.
Supplemental Material
Available for Download
Supplemental material for /*icomment: bugs or bad comments?*/
- C# XML comments let you build documentation directly from your Visual Studio .NET source files. http://msdn.microsoft.com/msdnmag/issues/02/06/XMLC/.Google Scholar
- CoNLL-2000 shared task web page -- with data, software and systems' outputs availble. http://www.cnts.ua.ac.be/conll/.Google Scholar
- Doxygen -- source code documentation generator tool. http://www.stack.nl/ dimitri/doxygen/.Google Scholar
- FreeBSD problem report database. http://www.freebsd.org/support/bugreports.html.Google Scholar
- Java annotations. http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html.Google Scholar
- Javadoc tool. http://java.sun.com/j2se/javadoc/.Google Scholar
- Lock_Lint -- Static data race and deadlock detection tool for C. http://developers.sun.com/sunstudio/articles/locklint.html.Google Scholar
- MSDN run-time library reference -- SAL annotations. http://msdn2.microsoft.com/en--us/library/ms235402.aspx.Google Scholar
- NLP tools. http://l2r.cs.uiuc.edu/~cogcomp/tools.php.Google Scholar
- RDoc -- documentation from Ruby source files. http://rdoc.sourceforge.net/.Google Scholar
- Sparse -- A semantic parser for C. http://www.kernel.org/pub/software/devel/sparse/.Google Scholar
- M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, 2003. Google Scholar
Digital Library
- J.-D. Choi, M. Burke, and P. Carini. Efficient flow-sensitive interprocedural computation of pointer-induced aliases and side effects. In Proceedings of the 20th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 1993. Google Scholar
Digital Library
- D. L. Detlefs, K. R. M. Leino, G. Nelson, and J. B. Saxe. Extended static checking, SRC research report 159. ftp://gatekeeper.research.compaq.com/pub/DEC/SRC/researchreports/SRC-159.ps.Google Scholar
- D. R. Engler and K. Ashcraft. RacerX: Effective, static detection of race conditions and deadlocks. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, 2003. Google Scholar
Digital Library
- D. R. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In Proceedings of the 18th ACM Symposium on Operating Systems Principles, 2001. Google Scholar
Digital Library
- M. D. Ernst, A. Czeisler, W. G. Griswold, and D. Notkin. Quickly detecting relevant program invariants. In Proceedings of the 22nd International Conference on Software Engineering, 2000. Google Scholar
Digital Library
- D. Evans and D. Larochelle. Improving security using extensible lightweight static analysis. IEEE Software, 2002. Google Scholar
Digital Library
- Y. Even-Zohar and D. Roth. A sequential model for multi class classification. In Proceedings of the Conference on Empirical Methods for Natural Language Processing, 2001.Google Scholar
- S. Hallem, B. Chelf, Y. Xie, and D. R. Engler. A system and language for building system-specific, static analyses. In Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation. Google Scholar
Digital Library
- W. E. Howden. Comments analysis and programming errors. IEEE Transactions on Software Engineering, 1990. Google Scholar
Digital Library
- Z. M. Jiang and A. E. Hassan. Examining the evolution of code comments in PostgreSQL. In Proceedings of the 2006 International Workshop on Mining Software Repositories. Google Scholar
Digital Library
- S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In USENIX Annual Technical Conference, 2005. Google Scholar
Digital Library
- T. Kremenek, P. Twohey, G. Back, A. Y. Ng, and D. R. Engler. From uncertainty to belief: Inferring the specification within. In Proceedings of the 7th USENIX Symposium on Operating System Design and Implementation, 2006. Google Scholar
Digital Library
- T. Li, C. Ellis, A. Lebeck, and D. Sorin. On-demand and semantic-free dynamic deadlock detection with speculative execution. In USENIX Annual Technical Conference, 2005. Google Scholar
Digital Library
- Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In Proceedings of the 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2005. Google Scholar
Digital Library
- C. D. Manning and H. Schütze. Foundations Of Statistical Natural Language Processing. The MIT Press, 2001. Google Scholar
Digital Library
- T. Mitchell. Machine Learning. McGraw Hill, 1997. Google Scholar
Digital Library
- M. Musuvathi, D. Y. W. Park, A. Chou, D. R. Engler, and D. L. Dill. CMC: A pragmatic approach to model checking real code. In Proceedingts of the 5th Symposium on Operating Systems Design and Implementation, 2002. Google Scholar
Digital Library
- A. C. Myers and B. Liskov. Protecting privacy using the decentralized label model. ACM Trans. Softw. Eng. Methodol., 2000. Google Scholar
Digital Library
- S. E. Perl and W. E. Weihl. Performance assertion checking. In Proceedings of the 14th ACM Symposium on Operating Systems Principles, 1993. Google Scholar
Digital Library
- V. Punyakanok and D. Roth. The use of classifiers in sequential inference. In Proceedings of the Conference on Advances in Neural Information Processing Systems, 2001.Google Scholar
- V. Punyakanok, D. Roth, and W. Yih. The necessity of syntactic parsing for semantic role labeling. In Proceedings of the International Joint Conference on Artificial Intelligence, 2005. Google Scholar
Digital Library
- R. J. Quilan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993. Google Scholar
Digital Library
- A. Ratnaparkhi. A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 1996.Google Scholar
- K. Rustan, M. Leino, G. Nelson, and J. B. Saxe. ESC/Java user's manual, SRC technical note 2000-002. http://gatekeeper.dec.com/pub/DEC/SRC/technicalnotes/abstracts/src-tn-2000-002.html.Google Scholar
- S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson. Eraser: A dynamic data race detector for multithreaded programs. ACM Transactions on Computer Systems, 1997. Google Scholar
Digital Library
- B. Steensgaard. Points-to analysis in almost linear time. In Proceedings of the 23rd Annual ACM SIGPLAN--SIGACT Symposium on Principles of Programming Languages, 1996. Google Scholar
Digital Library
- N. Sterling. WARLOCK -- A static data race analysis tool. In USENIX Winter Technical Conference, pages 97--106, 1993.Google Scholar
- S. Teufel and M. Moens. Summarizing scientific articles -- experiments with relevance and rhetorical status. Computational Linguistics, 2002. Google Scholar
Digital Library
- I. H. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques (2nd Ed.). Morgan Kaufmann, 2005. Google Scholar
Digital Library
- S. N. Woodfield, H. E. Dunsmore, and V. Y. Shen. The effect of modularization and comments on program comprehension. In Proceedings of the 5th International Conference on Software Engineering, 1981. Google Scholar
Digital Library
- A. Yaar, A. Perrig, and D. X. Song. Pi: A path identification mechanism to defend against DDoS attack. In IEEE Symposium on Security and Privacy, 2003. Google Scholar
Digital Library
- A. T. T. Ying, J. L. Wright, and S. Abrams. Source code that talks: An exploration of eclipse task comments and their implication to repository mining. In Proceedings of the 2005 International Workshop on Mining Software Repositories. Google Scholar
Digital Library
- C. Zhai, A. Velivelli, and B. Yu. A cross-collection mixture model for comparative text mining. In Proceedings of the 2004 ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining. Google Scholar
Digital Library
- F. Zhou, J. Condit, Z. Anderson, I. Bagrak, R. Ennals, M. Harren, G. Necula, and E. Brewer. SafeDrive: Safe and recoverable extensions using language-based techniques. In Proceedings of the 7th Symposium on Operating System Design and Implementation, 2006. Google Scholar
Digital Library
Index Terms
/*icomment: bugs or bad comments?*/
Recommendations
/*icomment: bugs or bad comments?*/
SOSP '07Commenting source code has long been a common practice in software development. Compared to source code, comments are more direct, descriptive and easy-to-understand. Comments and sourcecode provide relatively redundant and independent information ...
A Comment Analysis Approach for Program Comprehension
SEW '12: Proceedings of the 2012 35th Annual IEEE Software Engineering WorkshopComments are interspersed by the Programmer among code lines, at software development phase, with two main purposes: to help himself during the development phase, to help other programmers later on, during the maintenance phase. The former are memos to ...
@tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies
ICST '12: Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and ValidationCode comments are important artifacts in software. Javadoc comments are widely used in Java for API specifications. API developers write Javadoc comments, and API users read these comments to understand the API, e.g., reading a Javadoc comment for a ...







Comments