skip to main content
10.1145/1294261.1294276acmconferencesArticle/Chapter ViewAbstractPublication PagessospConference Proceedingsconference-collections
Article

/*icomment: bugs or bad comments?*/

Authors Info & Claims
Published:14 October 2007Publication History

ABSTRACT

Commenting source code has long been a common practice in software development. Compared to source code, comments are more direct, descriptive and easy-to-understand. Comments and sourcecode provide relatively redundant and independent information regarding a program's semantic behavior. As software evolves, they can easily grow out-of-sync, indicating two problems: (1) bugs -the source code does not follow the assumptions and requirements specified by correct program comments; (2) bad comments - comments that are inconsistent with correct code, which can confuse and mislead programmers to introduce bugs in subsequent versions. Unfortunately, as most comments are written in natural language, no solution has been proposed to automatically analyze commentsand detect inconsistencies between comments and source code. This paper takes the first step in automatically analyzing commentswritten in natural language to extract implicit program rulesand use these rules to automatically detect inconsistencies between comments and source code, indicating either bugs or bad comments. Our solution, iComment, combines Natural Language Processing(NLP), Machine Learning, Statistics and Program Analysis techniques to achieve these goals. We evaluate iComment on four large code bases: Linux, Mozilla, Wine and Apache. Our experimental results show that iComment automatically extracts 1832 rules from comments with 90.8-100% accuracy and detects 60 comment-code inconsistencies, 33 newbugs and 27 bad comments, in the latest versions of the four programs. Nineteen of them (12 bugs and 7 bad comments) have already been confirmed by the corresponding developers while the others are currently being analyzed by the developers.

Skip Supplemental Material Section

Supplemental Material

Video

References

  1. C# XML comments let you build documentation directly from your Visual Studio .NET source files. http://msdn.microsoft.com/msdnmag/issues/02/06/XMLC/.Google ScholarGoogle Scholar
  2. CoNLL-2000 shared task web page -- with data, software and systems' outputs availble. http://www.cnts.ua.ac.be/conll/.Google ScholarGoogle Scholar
  3. Doxygen -- source code documentation generator tool. http://www.stack.nl/ dimitri/doxygen/.Google ScholarGoogle Scholar
  4. FreeBSD problem report database. http://www.freebsd.org/support/bugreports.html.Google ScholarGoogle Scholar
  5. Java annotations. http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html.Google ScholarGoogle Scholar
  6. Javadoc tool. http://java.sun.com/j2se/javadoc/.Google ScholarGoogle Scholar
  7. Lock_Lint -- Static data race and deadlock detection tool for C. http://developers.sun.com/sunstudio/articles/locklint.html.Google ScholarGoogle Scholar
  8. MSDN run-time library reference -- SAL annotations. http://msdn2.microsoft.com/en--us/library/ms235402.aspx.Google ScholarGoogle Scholar
  9. NLP tools. http://l2r.cs.uiuc.edu/~cogcomp/tools.php.Google ScholarGoogle Scholar
  10. RDoc -- documentation from Ruby source files. http://rdoc.sourceforge.net/.Google ScholarGoogle Scholar
  11. Sparse -- A semantic parser for C. http://www.kernel.org/pub/software/devel/sparse/.Google ScholarGoogle Scholar
  12. M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. J.-D. Choi, M. Burke, and P. Carini. Efficient flow-sensitive interprocedural computation of pointer-induced aliases and side effects. In Proceedings of the 20th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. D. L. Detlefs, K. R. M. Leino, G. Nelson, and J. B. Saxe. Extended static checking, SRC research report 159. ftp://gatekeeper.research.compaq.com/pub/DEC/SRC/researchreports/SRC-159.ps.Google ScholarGoogle Scholar
  15. D. R. Engler and K. Ashcraft. RacerX: Effective, static detection of race conditions and deadlocks. In Proceedings of the 19th ACM Symposium on Operating Systems Principles, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. D. R. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In Proceedings of the 18th ACM Symposium on Operating Systems Principles, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. M. D. Ernst, A. Czeisler, W. G. Griswold, and D. Notkin. Quickly detecting relevant program invariants. In Proceedings of the 22nd International Conference on Software Engineering, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. D. Evans and D. Larochelle. Improving security using extensible lightweight static analysis. IEEE Software, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Y. Even-Zohar and D. Roth. A sequential model for multi class classification. In Proceedings of the Conference on Empirical Methods for Natural Language Processing, 2001.Google ScholarGoogle Scholar
  20. S. Hallem, B. Chelf, Y. Xie, and D. R. Engler. A system and language for building system-specific, static analyses. In Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. W. E. Howden. Comments analysis and programming errors. IEEE Transactions on Software Engineering, 1990. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Z. M. Jiang and A. E. Hassan. Examining the evolution of code comments in PostgreSQL. In Proceedings of the 2006 International Workshop on Mining Software Repositories. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In USENIX Annual Technical Conference, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. T. Kremenek, P. Twohey, G. Back, A. Y. Ng, and D. R. Engler. From uncertainty to belief: Inferring the specification within. In Proceedings of the 7th USENIX Symposium on Operating System Design and Implementation, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. T. Li, C. Ellis, A. Lebeck, and D. Sorin. On-demand and semantic-free dynamic deadlock detection with speculative execution. In USENIX Annual Technical Conference, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In Proceedings of the 13th ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. C. D. Manning and H. Schütze. Foundations Of Statistical Natural Language Processing. The MIT Press, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. T. Mitchell. Machine Learning. McGraw Hill, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. M. Musuvathi, D. Y. W. Park, A. Chou, D. R. Engler, and D. L. Dill. CMC: A pragmatic approach to model checking real code. In Proceedingts of the 5th Symposium on Operating Systems Design and Implementation, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. A. C. Myers and B. Liskov. Protecting privacy using the decentralized label model. ACM Trans. Softw. Eng. Methodol., 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. S. E. Perl and W. E. Weihl. Performance assertion checking. In Proceedings of the 14th ACM Symposium on Operating Systems Principles, 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. V. Punyakanok and D. Roth. The use of classifiers in sequential inference. In Proceedings of the Conference on Advances in Neural Information Processing Systems, 2001.Google ScholarGoogle Scholar
  33. V. Punyakanok, D. Roth, and W. Yih. The necessity of syntactic parsing for semantic role labeling. In Proceedings of the International Joint Conference on Artificial Intelligence, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. R. J. Quilan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. A. Ratnaparkhi. A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 1996.Google ScholarGoogle Scholar
  36. K. Rustan, M. Leino, G. Nelson, and J. B. Saxe. ESC/Java user's manual, SRC technical note 2000-002. http://gatekeeper.dec.com/pub/DEC/SRC/technicalnotes/abstracts/src-tn-2000-002.html.Google ScholarGoogle Scholar
  37. S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson. Eraser: A dynamic data race detector for multithreaded programs. ACM Transactions on Computer Systems, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. B. Steensgaard. Points-to analysis in almost linear time. In Proceedings of the 23rd Annual ACM SIGPLAN--SIGACT Symposium on Principles of Programming Languages, 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. N. Sterling. WARLOCK -- A static data race analysis tool. In USENIX Winter Technical Conference, pages 97--106, 1993.Google ScholarGoogle Scholar
  40. S. Teufel and M. Moens. Summarizing scientific articles -- experiments with relevance and rhetorical status. Computational Linguistics, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. I. H. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques (2nd Ed.). Morgan Kaufmann, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. S. N. Woodfield, H. E. Dunsmore, and V. Y. Shen. The effect of modularization and comments on program comprehension. In Proceedings of the 5th International Conference on Software Engineering, 1981. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. A. Yaar, A. Perrig, and D. X. Song. Pi: A path identification mechanism to defend against DDoS attack. In IEEE Symposium on Security and Privacy, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. A. T. T. Ying, J. L. Wright, and S. Abrams. Source code that talks: An exploration of eclipse task comments and their implication to repository mining. In Proceedings of the 2005 International Workshop on Mining Software Repositories. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. C. Zhai, A. Velivelli, and B. Yu. A cross-collection mixture model for comparative text mining. In Proceedings of the 2004 ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. F. Zhou, J. Condit, Z. Anderson, I. Bagrak, R. Ennals, M. Harren, G. Necula, and E. Brewer. SafeDrive: Safe and recoverable extensions using language-based techniques. In Proceedings of the 7th Symposium on Operating System Design and Implementation, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. /*icomment: bugs or bad comments?*/

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!