What would other programmers do: suggesting solutions to error messages

ABSTRACT

Interpreting compiler errors and exception messages is challenging for novice programmers. Presenting examples of how other programmers have corrected similar errors may help novices understand and correct such errors. This paper introduces HelpMeOut, a social recommender system that aids the debugging of error messages by suggesting solutions that peers have applied in the past. HelpMeOut comprises IDE instrumentation to collect examples of code changes that fix errors; a central database that stores fix reports from many users; and a suggestion interface that, given an error, queries the database for a list of relevant fixes and presents these to the programmer. We report on implementations of this architecture for two programming languages. An evaluation with novice programmers found that the technique can suggest useful fixes for 47% of errors after 39 person-hours of programming in an instrumented environment.

References

1. Ahmadzadeh, M., Elliman, D., and Higgins, C. An analysis of patterns of debugging among novice computer science students. Proceedings of the 10th annual SIGCSE conference on Innovation and technology in computer science education, ACM (2005), 84--88.
2. Birnbaum, B.E. and Goldman, K.J. Achieving Flexibility in Direct-Manipulation Programming Environments by Relaxing the Edit-Time Grammar. Proceedings of the IEEE Symposium on Visual Languages and Human Centric Computing, IEEE Computer Society (2005), 259--266.
3. Boustani, N.E. and Hage, J. Improving type error messages for generic java. Proceedings of the 2009 ACM SIGPLAN workshop on Partial evaluation and program manipulation, ACM (2009), 131--140.
4. Brandt, J., Dontcheva, M., Weskamp, M., and Klemmer, S.R. Example-Centric Programming: Integrating Web Search into the Development Environment. Proceedings of CHI 2010, (2010).
5. Brandt, J., Guo, P.J., Lewenstein, J., Dontcheva, M., and Klemmer, S.R. Opportunistic Programming: Writing Code to Prototype, Ideate, and Discover. IEEE Software 26, 5 (2009), 18--24.
6. Brandt, J., Guo, P.J., Lewenstein, J., Dontcheva, M., and Klemmer, S.R. Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. Proceedings of the 27th international conference on Human factors in computing systems, ACM (2009), 1589--1598.
7. Cheng, L., Souza, C.R.D., Hupfer, S., Patterson, J., and Ross, S. Building Collaboration into IDEs. Queue 1, 9 (2004), 40--50.
8. Dixon, P. pastebin -- collaborative debugging tool. http://pastebin.com/.
9. Domingue, J. and Mulholland, P. Fostering debugging communities on the Web. Communications of the ACM 40, 4 (1997), 65--71.
10. Fitzgerald, S., Lewandowski, G., McCauley, R., et al. Debugging: Finding, Fixing and Flailing, a Multi-Institutional Study of Novice Debuggers. Computer Science Education 18, 2 (2008), 93--116.
11. Gick, M.L. and Holyoak, K.J. Analogical Problem Solving. Cognitive Psychology 12, 3 (1980), 306--55.
12. Gick, M.L. and Holyoak, K.J. Schema induction and analogical transfer. Cognitive Psychology 15, 1 (1983), 1--38.
13. Grabler, F., Agrawala, M., Li, W., Dontcheva, M., and Igarashi, T. Generating photo manipulation tutorials by demonstration. ACM Transactions on Graphics 28, 3 (2009), 1--9.
14. Heckel, P. A technique for isolating differences between files. Communications of the ACM 21, 4 (1978), 264--268.
15. Hilbert, D.M. and Redmiles, D.F. Extracting usability information from user interface events. ACM Computing Surveys 32, 4 (2000), 384--421.
16. Hoffmann, R., Fogarty, J., and Weld, D.S. Assieme: finding and leveraging implicit references in a web search interface for programmers. Proceedings of the 20th annual ACM symposium on User interface software and technology, ACM (2007), 13--22.
17. Jeffery, C.L. Generating LR syntax error messages from examples. ACM Transactions on Programming Languages and Systems 25, 5 (2003), 631--640.
18. Jiang, L., Misherghi, G., Su, Z., and Glondu, S. DECKARD: Scalable and Accurate Tree-Based Detection of Code Clones. Proceedings of the 29th international conference on Software Engineering, IEEE (2007), 96--105.
19. Kim, S., Pan, K., and E. E. James Whitehead, J. Memories of bug fixes. Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, ACM (2006), 35--45.
20. Ko, A.J. and Myers, B.A. Debugging reinvented: asking and answering why and why not questions about program behavior. Proceedings of the 30th international conference on Software engineering, ACM (2008), 301--310.
21. Levenshtein, V.I. Binary codes capable of correcting deletions, insertions and reversals (in Russian). Soviet Physics Doklady 10, 8 (1966), 707--710.
22. Liblit, B., Naik, M., Zheng, A.X., Aiken, A., and Jordan, M.I. Scalable statistical bug isolation. Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, ACM (2005), 15--26.
23. Linton, F. and Schaefer, H. Recommender Systems for Learning: Building User and Expert Models through Long-Term Observation of Application Use. User Modeling and User-Adapted Interaction 10, 2--3 (2000), 181--208.
24. Livshits, B. and Zimmermann, T. DynaMine: finding common error patterns by mining software revision histories. SIGSOFT Software Engineering Notes 30, 5 (2005), 296--305.
25. Matejka, J., Li, W., Grossman, T., and Fitzmaurice, G. CommunityCommands: command recommendations for software applications. Proceedings of the 22nd annual ACM symposium on User interface software and technology, ACM (2009), 193--202.
26. McCauley, R., Fitzgerald, S., Lewandowski, G., et al. Debugging: A Review of the Literature from an Educational Perspective. Computer Science Education 18, 2 (2008).
27. Nardi, B. A small matter of programming. MIT Press, 1993.
28. Nienaltowski, M., Pedroni, M., and Meyer, B. Compiler error messages: what can help novices? Proceedings of the 39th SIGCSE technical symposium on Computer science education, ACM (2008), 168--172.
29. Shiffman, D. Learning Processing: A Beginner's Guide to Programming Images, Animation, and Interaction. Morgan Kaufmann, 2008.
30. Smith, R.B., Wolczko, M., and Ungar, D. From Kansas to Oz: collaborative debugging when a shared world breaks. Communications of the ACM 40, 4 (1997), 72--78.
31. Stylos, J., Faulring, A., Yang, Z., and Myers, B.A. Improving API Documentation Using API Usage Information. Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC'09, (2009).
32. Teitelbaum, T. and Reps, T. The Cornell program synthesizer: a syntax--directed programming environment. Communications of the ACM 24, 9 (1981), 563--573.
33. Terry, M., Kay, M., Vugt, B.V., Slack, B., and Park, T. Ingimp: introducing instrumentation to an end-user open source application. Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, ACM (2008), 607--616.
34. Yeh, R.B., Paepcke, A., and Klemmer, S.R. Iterative design and evaluation of an event architecture for pen-and-paper interfaces. Proceedings of the 21st annual ACM symposium on User interface software and technology, ACM (2008), 111--120.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging