ACM Digital Library home
ACM home
Advanced Search
10.1145/3344429.3372508acmconferencesArticle/Chapter ViewAccess DenialPublication PagesiticseConference Proceedingsconference-collections
research-article

Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research

ITiCSE-WGR '19: Proceedings of the Working Group Reports on Innovation and Technology in Computer Science EducationDecember 2019 Pages 177–210https://doi.org/10.1145/3344429.3372508
Online:18 December 2019Publication History
  • 27citation
  • 952
    • Downloads
Metrics
Total Citations27
Total Downloads952
Last 12 Months362
Last 6 weeks42
ITiCSE-WGR '19: Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education
Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research
Pages 177–210
PreviousChapterNextChapter

ABSTRACT

Diagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty and could be more effective, particularly for novices. Recent years have seen an increased number of papers in the area including studies on the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that can be used to predict student performance, track student progress, and tailor learning plans. Despite this increased interest, the long history of literature is quite scattered and has not been brought together in any digestible form.

In order to help the computing education community (and related communities) to further advance work on programming error messages, we present a comprehensive, historical and state-of-the-art report on research in the area. In addition, we synthesise and present the existing evidence for these messages including the difficulties they present and their effectiveness. We finally present a set of guidelines, curated from the literature, classified on the type of evidence supporting each one (historical, anecdotal, and empirical). This work can serve as a starting point for those who wish to conduct research on compiler error messages, runtime errors, and warnings. We also make the bibtex file of our 300+ reference corpus publicly available. Collectively this report and the bibliography will be useful to those who wish to design better messages or those that aim to measure their effectiveness, more effectively.

Get full access to this Publication

Purchase, subscribe or recommend this publication to your librarian.

Get this Publication

References

  1. Alireza Ahadi, Arto Hellas, Petri Ihantola, Ari Korhonen, and Andrew Petersen. 2016. Replication in Computing Education Research: Researcher Attitudes and Experiences. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research (Koli Calling '16). ACM, New York, NY, USA, 2--11. https://doi.org/10.1145/2999541.2999554Google ScholarGoogle Scholar
  2. Marzieh Ahmadzadeh, Dave Elliman, and Colin Higgins. 2005. An Analysis of Patterns of Debugging Among Novice Computer Science Students. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE '05). ACM, New York, NY, USA, 84--88. https://doi.org/ 10.1145/1067445.1067472Google ScholarGoogle Scholar
  3. Umair Z. Ahmed, Pawan Kumar, Amey Karkare, Purushottam Kar, and Sumit Gulwani. 2018. Compilation Error Repair: For the Student Programs, from the Student Programs. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET '18). ACM, New York, NY, USA, 78--87. https://doi.org/10.1145/3183377.3183383Google ScholarGoogle Scholar
  4. Umair Z. Ahmed, Renuka Sindhgatta, Nisheeth Srivastava, and Amey Karkare. 2019. Targeted Example Generation for Compilation Errors. In Proceedings of the 34th ACM/IEEE International Conference on Automated Software Engineering (ASE '19). ACM, New York, NY, USA, 12.Google ScholarGoogle Scholar
  5. Amjad Altadmri, Michael Kolling, and Neil C. C. Brown. 2016. The Cost of Syntax and How to Avoid It: Text versus Frame-Based Editing. In 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC '16). IEEE, 748--753. https://doi.org/10.1109/COMPSAC.2016.204Google ScholarGoogle Scholar
  6. Paul Ayres and John Sweller. 1990. Locus of Difficulty in Multistage Mathematics Problems. The American Journal of Psychology 103, 2 (1990), 167--193. http: //www.jstor.org/stable/1423141Google ScholarGoogle Scholar
  7. Paul L. Ayres. 2001. Systematic Mathematical Errors and Cognitive Load. Contemporary Educational Psychology 26, 2 (2001), 227 -- 248. https://doi.org/10.1006/ ceps.2000.1051Google ScholarGoogle Scholar
  8. Titus Barik. 2016. How Should Static Analysis Tools Explain Anomalies to Developers? A Communication Theory of Computationally Supporting Developer Self-Explanations for Static Analysis Anomalies. (2016). http://static.barik.net/ barik/proposal/barik_proposal_approved.pdfGoogle ScholarGoogle Scholar
  9. Titus Barik. 2018. Error Messages as Rational Reconstructions. Ph.D. Dissertation. North Carolina State University, Raleigh. https://repository.lib.ncsu.edu/handle/ 1840.20/35439Google ScholarGoogle Scholar
  10. Titus Barik, Denae Ford, Emerson Murphy-Hill, and Chris Parnin. 2018. How Should Compilers Explain Problems to Developers?. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018). ACM, New York, NY, USA, 633--643. https://doi.org/10.1145/3236024.3236040Google ScholarGoogle Scholar
  11. Titus Barik, Kevin Lubick, Samuel Christie, and Emerson Murphy-Hill. 2014. How Developers Visualize Compiler Messages: A Foundational Approach to Notification Construction. In 2014 Second IEEE Working Conference on Software Visualization. IEEE, 87--96. https://doi.org/10.1109/VISSOFT.2014.24Google ScholarGoogle Scholar
  12. Titus Barik, Justin Smith, Kevin Lubick, Elisabeth Holmes, Jing Feng, Emerson Murphy-Hill, and Chris Parnin. 2017. Do Developers Read Compiler Error Messages?. In Proceedings of the 39th International Conference on Software Engineering (ICSE '17). IEEE Press, Piscataway, NJ, USA, 575--585. https: //doi.org/10.1109/ICSE.2017.59Google ScholarGoogle Scholar
  13. D. W. Barron. 1975. A Note on APL. Comput. J. 19, 1 (1975), 93. https://academic. oup.com/comjnl/article-pdf/19/1/93/1058172/190093.pdfGoogle ScholarGoogle Scholar
  14. Brett A. Becker. 2015. An Exploration Of The Effects Of Enhanced Compiler Error Messages For Computer Programming Novices. Masters Thesis. Dublin Institute of Technology. https://doi.org/10.13140/RG.2.2.26637.13288Google ScholarGoogle Scholar
  15. Brett A. Becker. 2016. An Effective Approach to Enhancing Compiler Error Messages. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE '16). ACM, New York, NY, USA, 126--131. https://doi. org/10.1145/2839509.2844584Google ScholarGoogle Scholar
  16. Brett A. Becker. 2016. A New Metric to Quantify Repeated Compiler Errors for Novice Programmers. In Proceedings of the 21st ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16). ACM, New York, NY, USA, 296--301. https://doi.org/10.1145/2899415.2899463Google ScholarGoogle Scholar
  17. Brett A. Becker. 2019. Parlez-vous Java? Bonjour La Monde != Hello World: Barriers to Programming Language Acquisition for Non-Native English Speakers. In Proceedings of the 30th Annual Conference of the Psychology of Programming Interest Group (PPIG '19). http://www.ppig.org/sites/ppig.org/files/ 2019-PPIG-30th-becker.pdfGoogle ScholarGoogle Scholar
  18. Brett A. Becker, Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, and Catherine Mooney. 2016. Effective Compiler Error Message Enhancement for Novice Programming Students. Computer Science Education 26, 2--3 (2016), 148--175. https://doi.org/10.1080/08993408.2016.1225464Google ScholarGoogle Scholar
  19. Brett A. Becker, Kyle Goslin, and Graham Glanville. 2018. The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 640--645. https://doi.org/10.1145/3159450.3159461Google ScholarGoogle Scholar
  20. Brett A. Becker and Catherine Mooney. 2016. Categorizing Compiler Error Messages with Principal Component Analysis. In Proceedings of the 12th China- Europe International Symposium on Software Engineering Education (CEISEE '16). Shenyang, China, 1--8. https://researchrepository.ucd.ie/handle/10197/7889Google ScholarGoogle Scholar
  21. Brett A. Becker, Cormac Murray, Tianyi Tao, Changheng Song, Robert McCartney, and Kate Sanders. 2018. Fix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 634--639. https://doi.org/10.1145/3159450.3159453Google ScholarGoogle Scholar
  22. Brett A. Becker and Keith Quille. 2019. 50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA, 338--344. https://doi.org/10.1145/3287324.3287432Google ScholarGoogle Scholar
  23. T.R. Beelders and Jean-Pierre L. du Plessis. 2016. The Influence of Syntax Highlighting on Reading and Comprehending Code. Journal of Eye Movement Research Beelders 91, 1 (2016), 1--11. https://doi.org/10.16910/jemr.9.1.1Google ScholarGoogle Scholar
  24. Mordechai (Moti) Ben-Ari. 2007. Compile and Runtime Errors in Java. (2007). http://www.weizmann.ac.il/sci-tea/benari/sites/sci-tea.benari/files/ uploads/softwareAndLearningMaterials/errors.pdfGoogle ScholarGoogle Scholar
  25. Joe Bergin, Achla Agarwal, and Krishna Agarwal. 2003. Some Deficiencies of C++ in Teaching CS1 and CS2. ACM SIGPLAN Notices 38, 6 (2003), 9--13. https://doi.org/10.1145/885638.885642Google ScholarGoogle Scholar
  26. Sahil Bhatia and Rishabh Singh. 2016. Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks. In 2nd Indian Workshop on Machine Learning (IWML '16). https://www2.cse.iitk.ac.in/~iwml/ 2016/papers/iWML_2016_paper_5.pdfGoogle ScholarGoogle Scholar
  27. Michael W. Bigrigg, Russell Bortz, Shyamal Chandra, David Reed, Jared Sheehan, and Sara Smith. 2003. An Evaluation of the Usefulness of Compiler Error Messages. Technical Report. http://www.ices.cmu.edu/reports/040903.pdfGoogle ScholarGoogle Scholar
  28. Dennis Bouvier, Ellie Lovellette, John Matta, Bedour Alshaigy, Brett A. Becker, Michelle Craig, Jana Jackova, Robert McCartney, Kate Sanders, and Mark Zarb. 2016. Novice Programmers and the Problem Description Effect. In Proceedings of the 2016 ITiCSE Working Group Reports (ITiCSE-WGR '16). ACM, New York, NY, USA, 103--118. https://doi.org/10.1145/3024906.3024912Google ScholarGoogle Scholar
  29. Bernd Braßel. 2004. Typehope: There is Hope for Your Type Errors. In Int. Workshop on Implementation of Functional Languages.Google ScholarGoogle Scholar
  30. Pearl Brereton, Barbara A. Kitchenham, David Budgen, Mark Turner, and Mohamed Khalil. 2007. Lessons from Applying the Systematic Literature Review Process Within the Software Engineering Domain. J. Syst. Softw. 80, 4 (April 2007), 571--583. https://doi.org/10.1016/j.jss.2006.07.009Google ScholarGoogle Scholar
  31. Neil C. C. Brown and Amjad Altadmri. 2017. Novice Java Programming Mistakes: Large-Scale Data vs. Educator Beliefs. ACM Transactions on Computing Education 17, 2, Article 7 (May 2017), 21 pages. https://doi.org/10.1145/2994154Google ScholarGoogle Scholar
  32. Neil C. C. Brown, Amjad Altadmri, Sue Sentance, and Michael Kölling. 2018. Blackbox, Five Years On. In Proceedings of the 2018 ACM Conference on International Computing Education Research - ICER '18. ACM, Espoo, Finland, 196--204. https://doi.org/10.1145/3230977.3230991Google ScholarGoogle Scholar
  33. Neil C. C. Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: A Large Scale Repository of Novice Programmers' Activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE '14). ACM, New York, NY, USA, 223--228. https://doi.org/10.1145/2538862.2538924Google ScholarGoogle Scholar
  34. Neil C. C. Brown and Greg Wilson. 2018. Ten Quick Tips For Teaching Programming. PLoS Computational Biology 14, 4 (Apr 2018), e1006023. https: //doi.org/10.1371/journal.pcbi.1006023Google ScholarGoogle Scholar
  35. P. J. Brown. 1982. My System Gives Excellent Error Messages - Or Does It? Software: Practice and Experience 12, 1 (Jan 1982), 91--94. https://doi.org/10.1002/ spe.4380120110Google ScholarGoogle Scholar
  36. P. J. Brown. 1983. Error Messages: The Neglected Area of the Man/Machine Interface. Commun. ACM 26, 4 (Apr 1983), 246--249. https://doi.org/10.1145/2163. 358083Google ScholarGoogle Scholar
  37. Andrew Buxton and Lesley Trenner. 1987. An Experiment to Assess the Friendliness of Error Messages from Interactive Information Retrieval Systems. Journal of Information Science 13, 4 (Aug 1987), 197--209. https://doi.org/10.1177/ 016555158701300403Google ScholarGoogle Scholar
  38. Adam S. Carter, Christopher D. Hundhausen, and Olusola Adesope. 2015. The Normalized Programming State Model: Predicting Student Performance in Computing Courses Based on Programming Behavior. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research (ICER '15). ACM, New York, NY, USA, 141--150. https://doi.org/10.1145/2787622. 2787710Google ScholarGoogle Scholar
  39. A. T. Chamillard and William C. Hobart, Jr. 1997. Transitioning to Ada in an Introductory Course for Non-majors. In Proceedings of the Conference on TRI-Ada (TRI-Ada '97). ACM, New York, NY, USA, 37--40. https://doi.org/10.1145/269629. 269634Google ScholarGoogle Scholar
  40. G. V. Cormack. 1989. An LR Substring Parser for Noncorrecting Syntax Error Recovery. In Proceedings of the ACM SIGPLAN 1989 Conference on Programming Language Design and Implementation (PLDI '89). ACM, New York, NY, USA, 161--169. https://doi.org/10.1145/73141.74832Google ScholarGoogle Scholar
  41. Natalie J Coull. 2008. SNOOPIE: Development of a Learning Support Tool for Novice Programmers within a Conceptual Framework. Ph.D. Dissertation. University of St Andrews, St Andrews, Scotland. http://hdl.handle.net/10023/522Google ScholarGoogle Scholar
  42. Natalie J. Coull and Ishbel M.M. Duncan. 2011. Emergent Requirements for Supporting Introductory Programming. Innovations in Teaching and Learning in Information and Computer Sciences (ITaLICS) 10, 1 (2011), 78--85. https://doi.org/ 10.11120/ital.2011.10010078Google ScholarGoogle Scholar
  43. Paul Cress, Paul Dirksen, andWesley J Graham. 1970. FORTRAN IV WithWATFOR and WATFIV. Prentice-Hall, Englewood Cliffs, New Jersey.Google ScholarGoogle Scholar
  44. Edgar Dale and Jeanne S Chall. 1949. The Concept of Readability. Elementary English 26, 1 (1949), 19--26.Google ScholarGoogle Scholar
  45. E A Davis, M C Linn, and M Clancy. 1995. Learning to Use Parentheses and Quotes in LISP. Computer Science Education 6, 1 (1995), 15--31. https://doi.org/10. 1080/0899340950060102Google ScholarGoogle Scholar
  46. Carla De Lira. 2017. Improving the Learning Experiences of First-Year Computer Science Students with Empathetic IDEs. In Proceedings of the 2017 ACM Conference on International Computing Education Research (ICER '17). ACM, New York, NY, USA, 293--294. https://doi.org/10.1145/3105726.3105742Google ScholarGoogle Scholar
  47. Morris Dean. 1982. How a Computer Should Talk To People. IBM Systems Journal 21, 4 (1982), 424--453. https://doi.org/10.1147/sj.214.0424Google ScholarGoogle Scholar
  48. Paul Denny, Brett A. Becker, Michelle Craig, Greg Wilson, and Piotr Banaszkiewicz. 2019. Research This! Questions That Computing Educators Most Want Computing Education Researchers to Answer. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER '19). ACM, New York, NY, USA, 259--267. https://doi.org/10.1145/3291279.3339402Google ScholarGoogle Scholar
  49. Paul Denny, Brian Hanks, and Beth Simon. 2010. PeerWise: Replication Study of a Student-collaborative Self-testing Web Service in a U.S. Setting. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education (SIGCSE '10). ACM, New York, NY, USA, 421--425. https://doi.org/10.1145/1734263.1734407Google ScholarGoogle Scholar
  50. Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing Syntax Error Messages Appears Ineffectual. In Proceedings of the 19th Conference on Innovation and Technology in Computer Science Education (ITiCSE '14). ACM, New York, NY, USA, 273--278. https://doi.org/10.1145/2591708.2591748Google ScholarGoogle Scholar
  51. Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2012. All Syntax Errors Are Not Equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE '12). ACM, New York, NY, USA, 75--80. https://doi.org/10.1145/2325296.2325318Google ScholarGoogle Scholar
  52. Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. CodeWrite: Supporting Student-driven Practice of Java. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (SIGCSE '11). ACM, New York, NY, USA, 471--476. https://doi.org/10.1145/1953163.1953299Google ScholarGoogle Scholar
  53. Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. Understanding the Syntax Barrier for Novices. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education (ITiCSE '11). ACM, New York, NY, USA, 208--212. https://doi.org/10.1145/1999747. 1999807Google ScholarGoogle Scholar
  54. Paul Denny, James Prather, Brett A. Becker, Zachary Albrecht, Dastyni Loksa, and Raymond Pettit. 2019. A Closer Look at Metacognitive Scaffolding: Solving Test Cases Before Programming. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research (Koli Calling '19). ACM, New York, NY, USA. https://doi.org/10.1145/3364510.3366170Google ScholarGoogle Scholar
  55. Gergely Dévai, Dániel Leskó, and Máté Tejfel. 2013. The EDSL's Struggle for Their Sources. In: Zsók V., Horváth Z., Csató L. (eds) Central European Functional Programming School. CEFP 2013. Lecture Notes in Computer Science, Vol. 8606. Springer, Cham. 300--335 pages. https://doi.org/10.1007/978--3--319--15940--9_7Google ScholarGoogle Scholar
  56. Tao Dong and Kandarp Khandwala. 2019. The Impact of "Cosmetic" Changes on the Usability of Error Messages. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1--6. https://doi.org/10.1145/3290607. 3312978Google ScholarGoogle Scholar
  57. Benedict du Boulay and Ian Matthew. 1984. Fatal Error in Pass Zero: How Not to Confuse Novices. Behaviour and Information Technology 3, 2 (1984), 109--118. https://doi.org/10.1080/01449298408901742Google ScholarGoogle Scholar
  58. William H DuBay. 2007. Smart Language: Readers, Readability, and the Grading of Text. ERIC.Google ScholarGoogle Scholar
  59. Thomas Dy and Ma. Mercedes Rodrigo. 2010. A Detector for Non-literal Java Errors. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling '10). ACM, New York, NY, USA, 118--122. https://doi.org/10.1145/1930464.1930485Google ScholarGoogle Scholar
  60. Eddie Antonio Santos, Joshua Charles Campbell, Dhvani Patel, Abram Hindle and José Nelson Amaral. 2018. Syntax and Sensibility: Using Language Models to Detect and Correct Syntax Errors. 311--322 pages.Google ScholarGoogle Scholar
  61. Marc Eisenstadt and Matthew W. Lewis. 2018. Errors in an Interactive Programming Environment: Causes and Cures. In Novice Programming Environments, Mark Eisenstadt, Mark T. Keane, and Tim Rajan (Eds.). Routledge, London, Chapter 5, 111--131.Google ScholarGoogle Scholar
  62. Nabil El Boustani and Jurriaan Hage. 2010. Corrective Hints for Type Incorrect Generic Java Programs. In Proceedings of the 2010 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation (PEPM '10). ACM, New York, NY, USA, 5--14. https://doi.org/10.1145/1706356.1706360Google ScholarGoogle Scholar
  63. Nabil El Boustani and Jurriaan Hage. 2011. Improving Type Error Messages for Generic Java. In Higher-Order and Symbolic Computation, Vol. 24. Savannah, GA, 3--39. https://doi.org/10.1007/s10990-011--9070--3Google ScholarGoogle Scholar
  64. U. Engelmann and H. P. Meinzer. 1985. Rules for the Design of End User Languages. In Medical Informatics Europe 85, F. H. Roger, P. Grönroos, R. Tervo- Pellikka, and R. O'Moore (Eds.). Springer, Berlin, Heidelberg, Helsinki, Finland, 240--245. https://doi.org/10.1007/978--3--642--93295--3_48Google ScholarGoogle Scholar
  65. Anneli Eteläpelto. 1993. Metacognition and the Expertise of Computer Program Comprehension. Scandinavian Journal of Educational Research 37, 3 (1993), 243-- 254. https://doi.org/10.1080/0031383930370305Google ScholarGoogle Scholar
  66. Andrew Ettles, Andrew Luxton-Reilly, and Paul Denny. 2018. Common Logic Errors Made by Novice Programmers. In Proceedings of the 20th Australasian Computing Education Conference (ACE '18). ACM, New York, NY, USA, 83--89. https://doi.org/10.1145/3160489.3160493Google ScholarGoogle Scholar
  67. Georgios Evangelidis, Vassilios Dagdilelis, Maria Satratzemi, and Vassilios Efopoulos. 2001. X-compiler: Yet Another Integrated Novice Programming Environment. In Proceedings IEEE International Conference on Advanced Learning Technologies. IEEE Comput. Soc, 166--169. https://doi.org/10.1109/ICALT.2001.943890Google ScholarGoogle Scholar
  68. S. Fincher. 1999. What are we doing when we teach programming?. In FIE'99 Frontiers in Education. 29th Annual Frontiers in Education Conference. Designing the Future of Science and Engineering Education. Conference Proceedings (IEEE Cat. No.99CH37011, Vol. 1. 12A4/1--12A4/5 vol.1. https://doi.org/10.1109/FIE.1999. 839268Google ScholarGoogle Scholar
  69. Robert Bruce Findler, John Clements, Cormac Flanagan, Matthew Flatt, Shriram Krishnamurthi, Paul Steckler, and Matthias Felleisen. 2002. DrScheme: A Programming Environment for Scheme. Journal of Functional Programming 12, 2 (March 2002), 159--182. https://doi.org/10.1017/S0956796801004208Google ScholarGoogle Scholar
  70. Allan Fisher and Jane Margolis. 2002. Unlocking the Clubhouse: The Carnegie Mellon Experience. SIGCSE Bull. 34, 2 (June 2002), 79--83. https://doi.org/10.1145/ 543812.543836Google ScholarGoogle Scholar
  71. Julie Fisher. 1999. The Importance of User Message Text and Why Professional Writers Should Be Involved. Australian Computer Journal 31 (Nov 1999), 118--123.Google ScholarGoogle Scholar
  72. Thomas Flowers, Curtis Carver, and James Jackson. 2004. Empowering students and building confidence in novice programmers through gauntlet. In 34th ASEE/IEEE Annual Frontiers in Education, 2004. FIE 2004. IEEE, Savannah, GA, USA, T3H10--13. https://doi.org/10.1109/fie.2004.1408551Google ScholarGoogle Scholar
  73. Edward B Fry. 2006. Readability: Reading Hall of Fame Book. Newark. DE: International Reading Association (2006).Google ScholarGoogle Scholar
  74. Richard Furuta and P. Michael Kemp. 1979. Experimental Evaluation of Programming Language Features: Implications for Introductory Programming Languages. In Proceedings of the 10th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '79). ACM, New York, NY, USA, 18--21. https: //doi.org/10.1145/800126.809544Google ScholarGoogle Scholar
  75. Susan L. Graham and Steven P. Rhodes. 1973. Practical Syntactic Error Recovery in Compilers. In Proceedings of the 1st Annual ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (POPL '73). ACM, New York, NY, USA, 52--58. https://doi.org/10.1145/512927.512932Google ScholarGoogle Scholar
  76. David Gries. 1968. Use of Transition Matrices in Compiling. Commun. ACM 11, 1 (Jan. 1968), 26--34. https://doi.org/10.1145/362851.362872Google ScholarGoogle Scholar
  77. David Gries. 1974. What Should We Teach in an Introductory Programming Course?. In Proceedings of the Fourth SIGCSE Technical Symposium on Computer Science Education (SIGCSE '74). ACM, New York, NY, USA, 81--89. https://doi.org/ 10.1145/800183.810447Google ScholarGoogle Scholar
  78. Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. DeepFix: Fixing Common C Language Errors by Deep Learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI'17). AAAI Press, 1345--1351. http://dl.acm.org/citation.cfm?id=3298239.3298436Google ScholarGoogle Scholar
  79. Jurriaan Hage and Heeren Bastiaan. 2006. Heuristics for Type Error Discovery and Recovery. In 18th International Conference on Implementation and Application of Functional Languages (IFL '06). 199--216. https://link.springer.com/chapter/10. 1007/978--3--540--74130--5_12Google ScholarGoogle Scholar
  80. Devon Harker. 2017. Examining the Effects of Enhanced Compilers on Student Productivity. Masters Thesis. University of Northern British Columbia. https: //unbc.arcabc.ca/islandora/object/unbc%3A58897Google ScholarGoogle Scholar
  81. Jan Lee Harrington. 1984. The Effect of Error Messages on Learning Computer Programming by Individuals Without Prior Programming Experience. PhD Thesis. Drexel University.Google ScholarGoogle Scholar
  82. Theodore L Harris and Richard E Hodges. 1995. The Literacy Dictionary: The Vocabulary of Reading and Writing. ERIC.Google ScholarGoogle Scholar
  83. Björn Hartmann, Daniel MacDougall, Joel Brandt, and Scott R. Klemmer. 2010. What Would Other Programmers Do: Suggesting Solutions to Error Messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1019--1028. https://doi.org/10.1145/1753326. 1753478Google ScholarGoogle Scholar
  84. J. Hartz, Adam. 2012. CAT-SOOP: A Tool for Automatic Collection and Assessment of Homework Exercises. Master's thesis. Massachusetts Institute of Technology. https://dspace.mit.edu/bitstream/handle/1721.1/77086/825763362-MIT.pdf? sequence=2Google ScholarGoogle Scholar
  85. Brian Harvey. 1982. Why Logo? Byte 7, 8 (Aug 1982), 163--195. http://cmkfutures. com/wp-content/uploads/2017/06/Why-Logo-by-Brian-Harvey.pdfGoogle ScholarGoogle Scholar
  86. Bastiaan Heeren, Jurriaan Hage, and S. Doaitse Swierstra. 2003. Scripting the Type Inference Process. In Proceedings of the 8th ACM SIGPLAN International Conference on Functional Programming (ICFP '03). ACM, New York, NY, USA, 3--13. https://doi.org/10.1145/944705.944707Google ScholarGoogle Scholar
  87. Bastiaan Heeren, Daan Leijen, and Arjan van IJzendoorn. 2003. Helium, for Learning Haskell. In Proceedings of the 2003 ACM SIGPLAN Workshop on Haskell (Haskell '03). ACM, New York, NY, USA, 62--71. https://doi.org/10.1145/871895. 871902Google ScholarGoogle Scholar
  88. Zef Hemel, Danny M. Groenewegen, Lennart C.L. Kats, and Eelco Visser. 2011. Static Consistency Checking of Web Applications with WebDSL. Journal of Symbolic Computation 46, 2 (2011), 150--182. https://doi.org/10.1016/j.jsc.2010.08. 006Google ScholarGoogle Scholar
  89. Richard Hill. 2008. Developing a Teaching Compiler for Students Learning the C Programming Language. Bachelor of Science Dissertation. University of Bath.Google ScholarGoogle Scholar
  90. James J Horning. 1976. What the Compiler Should Tell the User. In Compiler Construction: An Advanced Course, G Goos and J Hartmanis (Eds.). Springer- Verlag, Berlin-Heidelberg, 525--548.Google ScholarGoogle Scholar
  91. C. D. Hundhausen, D. M. Olivares, and A. S. Carter. 2017. IDE-Based Learning Analytics for Computing Education: A Process Model, Critical Review, and Research Agenda. ACM Transactions on Computing Education 17, 3, Article 11 (Aug. 2017), 26 pages. https://doi.org/10.1145/3105759Google ScholarGoogle Scholar
  92. Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H. Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel Ángel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll. 2015. Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies. In Proceedings of the 2015 ITiCSE Working Group Reports (ITICSE-WGR '15). ACM, New York, NY, USA, 41--63. https://doi.org/10.1145/2858796.2858798Google ScholarGoogle Scholar
  93. Barbara S. Isa, James M. Boyle, Alan S. Neal, and Roger M. Simons. 1983. A Methodology for Objectively Evaluating Error Messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '83). ACM, New York, NY, USA, 68--71. https://doi.org/10.1145/800045.801583Google ScholarGoogle Scholar
  94. ISO/IEC 14882:2011 2011. Information Technology -- Programming languages -- C++. Technical Report. https://www.iso.org/standard/50372.htmlGoogle ScholarGoogle Scholar
  95. ISO/IEC TS 19217:2015 2015. Information Technology -- Programming Languages -- C++ Extensions for Concepts. Technical Report. https://www.iso.org/standard/ 64031.htmlGoogle ScholarGoogle Scholar
  96. J. Jackson, M. Cobb, and C. Carver. 2005. Identifying Top Java Errors for Novice Programmers. In 35th Annual Frontiers in Education Conference (FIE '05). T4C--24 -- T4C--27. https://doi.org/10.1109/fie.2005.1611967Google ScholarGoogle Scholar
  97. Matthew C Jadud. 2005. A First Look at Novice Compilation Behaviour Using BlueJ. Computer Science Education 15, 1 (2005), 25--40. https://doi.org/10.1080/ 08993400500056530 arXiv:https://doi.org/10.1080/08993400500056530Google ScholarGoogle Scholar
  98. Matthew C. Jadud. 2006. An Exploration of Novice Compilation Behaviour in BlueJ. Ph.D. Dissertation. University of Kent at Canterbury. https://jadud.com/dl/pdf/ jadud-dissertation.pdfGoogle ScholarGoogle Scholar
  99. Mathias Johan Johansen. 2015. Errors and Misunderstandings Among Novice Programmers Assessing the Student Not the Program. Masters Thesis. University of Oslo. https://www.duo.uio.no/handle/10852/49045Google ScholarGoogle Scholar
  100. Eliezer Kantorowitz and H. Laor. 1986. Automatic Generation of Useful Syntax Error Messages. Software: Practice and Experience 16, 7 (1986), 627--640.Google ScholarGoogle Scholar
  101. Ioannis Karvelas. 2019. Investigating Novice Programmers' Interaction with Programming Environments. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '19). ACM, New York, NY, USA, 336--337. https://doi.org/10.1145/3304221.3325596Google ScholarGoogle Scholar
  102. Caitlin Kelleher, Dennis Cosgrove, and David Culyba. 2002. Alice2: Programming Without Syntax Errors. User Interface Software and Technology - UIST 2002 (2002), 3--4. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.4640&rep= rep1&type=pdfGoogle ScholarGoogle Scholar
  103. B. Kitchenham and S. Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering, version 2.3. (2007).Google ScholarGoogle Scholar
  104. Amy J Ko, Brad A Myers, and Htet Htet Aung. 2004. Six Learning Barriers in End-User Programming Systems Prior Research on Learning Barriers A Study of Visual Basic . NET 2003. In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing (VLHCC '04). 199--206. https://doi.org/10.1109/ VLHCC.2004.47Google ScholarGoogle Scholar
  105. Tobias Kohn. 2017. Teaching Python Programming to Novices: Addressing Misconceptions and Creating a Development Environment. PhD Thesis. ETH Zürich.Google ScholarGoogle Scholar
  106. Tobias Kohn. 2019. The Error Behind The Message: Finding the Cause of Error Messages in Python. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA, 524--530. https://doi.org/10.1145/3287324.3287381Google ScholarGoogle Scholar
  107. Michael Kölling. 1999. The Design of an Object-Oriented Environment and Language for Teaching. Ph.D. Dissertation. University of Sydney. https://kar.kent.ac. uk/21868/1/the_design_of_an_object-oriented_kolling.pdfGoogle ScholarGoogle Scholar
  108. Michael Kölling. 2015. Lessons from the Design of Three Educational Programming Environments: Blue, BlueJ and Greenfoot. International Journal of People- Oriented Programming 4, 1 (2015), 5--32. https://doi.org/10.4018/ijpop.2015010102Google ScholarGoogle Scholar
  109. Michael Kölling, Bruce Quig, Andrew Patterson, and John Rosenberg. 2003. The BlueJ System and its Pedagogy. Computer Science Education 13, 4 (Dec 2003), 249--268. https://doi.org/10.1076/csed.13.4.249.17496Google ScholarGoogle Scholar
  110. Sarah K. Kummerfeld and Judy Kay. 2003. The Neglected Battle Fields of Syntax Errors. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20 (ACE '03). Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 105--111. http://dl.acm.org/citation.cfm?id=858403.858416Google ScholarGoogle Scholar
  111. Thomas Kurtz. 1978. BASIC. ACM SIGPLAN Notices - Special issue: History of programming languages conference 13, 8 (1978), 103--118. https://doi.org/10.1145/ 960118.808376Google ScholarGoogle Scholar
  112. Nicolas Laurent. 2017. Red Shift: Procedural Shift-reduce Parsing (Vision Paper). In Proceedings of the 10th ACM SIGPLAN International Conference on Software Language Engineering (SLE '17). ACM, New York, NY, USA, 38--42. https://doi. org/10.1145/3136014.3136036Google ScholarGoogle Scholar
  113. Michael J. Lee and Amy J. Ko. 2011. Personifying Programming Tool Feedback Improves Novice Programmers' Learning. In Proceedings of the Seventh International Workshop on Computing Education Research (ICER '11). ACM, New York, NY, USA, 109--116. https://doi.org/10.1145/2016911.2016934Google ScholarGoogle Scholar
  114. Oukseh Lee and Kwangkeun Yi. 1998. Proofs About a Folklore Let-polymorphic Type Inference Algorithm. ACM Trans. Program. Lang. Syst. 20, 4 (July 1998), 707--723. https://doi.org/10.1145/291891.291892Google ScholarGoogle Scholar
  115. Ronald Paul Leinius. 1970. Error Detection and Recovery for Syntax Directed Compiler Systems. Ph.D. Dissertation. AAI7024758.Google ScholarGoogle Scholar
  116. Benjamin S. Lerner, Matthew Flower, Dan Grossman, and Craig Chambers. 2007. Searching for Type-error Messages. In Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '07). ACM, New York, NY, USA, 425--434. https://doi.org/10.1145/1250734.1250783Google ScholarGoogle Scholar
  117. Stuart Lewis and Gaius Mulley. 1998. A Comparison Between Novice and Experienced Compiler Users in a Learning Environment. In Proceedings of the 6th Annual Conference on the Teaching of Computing and the 3rd Annual Conference on Integrating Technology into Computer Science Education: Changing the Delivery of Computer Science Education (ITiCSE '98). ACM, New York, NY, USA, 157--161. https://doi.org/10.1145/282991.283106Google ScholarGoogle Scholar
  118. William Lidwell, Kritina Holden, and Jill Butler. 2010. Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design. Rockport Publishers.Google ScholarGoogle Scholar
  119. Derrell Lipman. 2014. LearnCS! a Browser-Based Research Platform for CS1 and Studying the Role of Instruction of Debugging from Early in the Course. Ph.D. Dissertation. University of Massachusetts Lowell.Google ScholarGoogle Scholar
  120. Charles R. Litecky and Gordon B. Davis. 1976. A Study of Errors, Errorproneness, and Error Diagnosis in Cobol. Commun. ACM 19, 1 (1976), 33--38. https://doi.org/10.1145/359970.359991Google ScholarGoogle Scholar
  121. Dastyni Loksa, Amy J. Ko, Will Jernigan, Alannah Oleson, Christopher J. Mendez, and Margaret M. Burnett. 2016. Programming, Problem Solving, and Self- Awareness: Effects of Explicit Guidance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1449--1461. https://doi.org/10.1145/2858036.2858252Google ScholarGoogle Scholar
  122. Glenn R Luecke, James Coyle, James Hoekstra, Marina Kraeva, and Ying Xu. 2009. The Importance of Run-time Error Detection. In Tools for High Performance Computing 2009. 145--155. https://doi.org/10.1007/978--3--642--11261--4Google ScholarGoogle Scholar
  123. Glenn R. Luecke, James Coyle, James Hoekstra, Marina Kraeva, Ying Xu, Elizabeth Kleiman, and Olga Weiss. 2009. Evaluating Error Detection Capabilities of UPC Run-time Systems. In Proceedings of the Third Conference on Partitioned Global Address Space Programing Models (PGAS '09). ACM, New York, NY, USA, Article 7, 4 pages. https://doi.org/10.1145/1809961.1809971Google ScholarGoogle Scholar
  124. Harri Luoma, Essi Lahtinen, and Hannu-Matti Järvinen. 2007. CLIP, a Command Line Interpreter for a Subset of C++. In Proceedings of the Seventh Baltic Sea Conference on Computing Education Research - Volume 88 (Koli Calling '07). Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 199--202. http://dl.acm.org/citation.cfm?id=2449323.2449351Google ScholarGoogle Scholar
  125. Andrew Luxton-Reilly. 2016. Learning to Program is Easy. In Proceedings of the 21st ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '16). ACM, New York, NY, USA, 284--289. https://doi.org/10.1145/2899415. 2899432Google ScholarGoogle Scholar
  126. Andrew Luxton-Reilly, Brett A. Becker, Yingjun Cao, Roger McDermott, Claudio Mirolo, Andreas Mühling, Andrew Petersen, Kate Sanders, Simon, and Jacqueline Whalley. 2017. Developing Assessments to Determine Mastery of Programming Fundamentals. In Proceedings of the 2017 ITiCSE Conference on Working Group Reports (ITiCSE-WGR '17). ACM, New York, NY, USA, 47--69. https://doi.org/10. 1145/3174781.3174784Google ScholarGoogle Scholar
  127. Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory Programming: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '18). ACM, New York, NY, USA, 55--106. https://doi.org/10.1145/3293881.3295779Google ScholarGoogle Scholar
  128. Celeste S. Magers. 1983. An Experimental Evaluation of On-line HELP for Non-programmers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '83). ACM, New York, NY, USA, 277--281. https: //doi.org/10.1145/800045.801626Google ScholarGoogle Scholar
  129. Qusay H. Mahmoud, Wlodek Dobosiewicz, and David Swayne. 2004. Redesigning Introductory Computer Programming with HTML, JavaScript, and Java. SIGCSE Bull. 36, 1 (March 2004), 120--124. https://doi.org/10.1145/1028174.971344Google ScholarGoogle Scholar
  130. John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The Scratch Programming Language and Environment. ACM Transactions on Computing Education 10, 4, Article 16 (Nov. 2010), 15 pages. https://doi.org/10.1145/1868358.1868363Google ScholarGoogle Scholar
  131. Murali Mani and Quamrul Mazumder. 2013. Incorporating Metacognition into Learning. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (SIGCSE '13). ACM, New York, NY, USA, 53--58. https://doi.org/10. 1145/2445196.2445218Google ScholarGoogle Scholar
  132. Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011. Measuring the Effectiveness of Error Messages Designed for Novice Programmers. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (SIGCSE '11). ACM, New York, NY, USA, 499--504. https://doi.org/10.1145/1953163.1953308Google ScholarGoogle Scholar
  133. Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011. Mind Your Language: On Novices' Interactions with Error Messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2011). ACM, New York, NY, USA, 3--18. https://doi.org/10.1145/2048237.2048241Google ScholarGoogle Scholar
  134. Samiha Marwan, Nicholas Lytle, Joseph Jay Williams, and Thomas Price. 2019. The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '19). ACM, New York, NY, USA, 520--526. https://doi.org/10.1145/3304221.3319759Google ScholarGoogle Scholar
  135. Richard E. Mayer. 2004. Teaching of Subject Matter. Annual Review of Psychology 55, 1 (Feb 2004), 715--744. https://doi.org/10.1146/annurev.psych.55.082602.133124Google ScholarGoogle Scholar
  136. G Harry Mc Laughlin. 1969. SMOG Grading -- A New Readability Formula. Journal of reading 12, 8 (1969), 639--646.Google ScholarGoogle Scholar
  137. Bruce J. McAdam. 1998. On the Unification of Substitutions in Type Inference. Technical Report. 1--23 pages. http://www.lfcs.inf.ed.ac.uk/reports/98/ ECS-LFCS-98--384/ECS-LFCS-98--384.pdfGoogle ScholarGoogle Scholar
  138. Davin Mccall. 2016. Novice Programmer Errors-Analysis and Diagnostics. Ph.D. Dissertation. The University of Kent. https://kar.kent.ac.uk/id/eprint/61340Google ScholarGoogle Scholar
  139. Davin McCall and Michael Kölling. 2014. Meaningful Categorisation of Novice Programmer Errors. In IEEE Frontiers in Education Conference (FIE '14). 1--8. https://doi.org/10.1109/FIE.2014.7044420Google ScholarGoogle Scholar
  140. Davin McCall and Michael Kölling. 2019. A New Look at Novice Programmer Errors. ACM Transactions on Computing Education 19, 4 (2019), 1--30. https: //doi.org/10.1145/3335814 arXiv:10.1145/3335814Google ScholarGoogle Scholar
  141. Michael McCracken, Vicki Almstrum, Danny Diaz, Mark Guzdial, Dianne Hagan, Yifat Ben-David Kolikant, Cary Laxer, Lynda Thomas, Ian Utting, and Tadeusz Wilusz. 2001. A Multi-national, Multi-institutional Study of Assessment of Programming Skills of First-year CS Students. SIGCSE Bull. 33, 4 (Dec. 2001), 125--180. https://doi.org/10.1145/572139.572181Google ScholarGoogle Scholar
  142. L. McIver and D. Conway. 1996. Seven Deadly Sins of Introductory Programming Language Design. In International Conference on Software Engineering: Education and Practice (SEEP '96). IEEE Computer Society, Dunedin, New Zealand, 309--316. https://doi.org/10.1109/SEEP.1996.534015Google ScholarGoogle Scholar
  143. Linda Kathryn McIver. 2001. Syntactic and Semantic Issues in Introductory Programming Education. PhD Thesis. Monash University.Google ScholarGoogle Scholar
  144. R. P. Medeiros, G. L. Ramalho, and T. P. Falcão. 2019. A Systematic Literature Review on Teaching and Learning Introductory Programming in Higher Education. IEEE Transactions on Education 62, 2 (May 2019), 77--90. https: //doi.org/10.1109/TE.2018.2864133Google ScholarGoogle Scholar
  145. Rolf Molich and Jakob Nielsen. 1990. Improving a Human-computer Dialogue. Commun. ACM 33, 3 (March 1990), 338--348. https://doi.org/10.1145/77481.77486Google ScholarGoogle Scholar
  146. P. G. Moulton and M. E. Muller. 1967. DITRAN - A Compiler Emphasizing Diagnostics. Commun. ACM 10, 1 (1967), 45--52. https://doi.org/10.1145/363018. 363060Google ScholarGoogle Scholar
  147. F Mulder. 2016. Awesome Error Messages for Dotty. (Oct 2016). https://www. scala-lang.org/blog/2016/10/14/dotty-errors.htmlGoogle ScholarGoogle Scholar
  148. Christian Murphy, Eunhee Kim, Gail Kaiser, and Adam Cannon. 2008. Backstop: A Tool for Debugging Runtime Errors. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '08). ACM, New York, NY, USA, 173--177. https://doi.org/10.1145/1352135.1352193Google ScholarGoogle Scholar
  149. Emerson Murphy-Hill, Titus Barik, and Andrew P. Black. 2013. Interactive Ambient Visualizations For Soft Advice. Information Visualization 12, 2 (2013), 107--132. https://doi.org/10.1177/1473871612469020Google ScholarGoogle Scholar
  150. Emerson Murphy-Hill and Andrew P. Black. 2012. Programmer-Friendly Refactoring Errors. IEEE Transactions on Software Engineering 38, 6 (2012), 1417--1431. https://doi.org/10.1109/TSE.2011.110Google ScholarGoogle Scholar
  151. Scott Nesbitt. 2017. How to Write Better Error Messages. (Aug 2017). https: //opensource.com/article/17/8/write-effective-error-messagesGoogle ScholarGoogle Scholar
  152. Eric Niebler. 2007. Proto: A Compiler Construction Toolkit for DSELs. In Proceedings of the 2007 Symposium on Library-Centric Software Design (LCSD '07). ACM, New York, NY, USA, 42--51. https://doi.org/10.1145/1512762.1512767Google ScholarGoogle Scholar
  153. Marie-Hélène Nienaltowski, Michela Pedroni, and Bertrand Meyer. 2008. Compiler Error Messages: What Can Help Novices?. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '08). ACM, New York, NY, USA, 168--172. https://doi.org/10.1145/1352135.1352192Google ScholarGoogle Scholar
  154. Sagar Parihar, Ziyaan Dadachanji, Praveen Kumar Singh, Rajdeep Das, Amey Karkare, and Arnab Bhattacharya. 2017. Automatic Grading and Feedback Using Program Repair for Introductory Programming Courses. In Proceedings of the 22nd ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '17). ACM, New York, NY, USA, 92--97. https://doi.org/10.1145/3059009. 3059026Google ScholarGoogle Scholar
  155. Miranda C. Parker, Mark Guzdial, and Shelly Engleman. 2016. Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment. In Proceedings of the 2016 ACM Conference on International Computing Education Research (ICER '16). ACM, New York, NY, USA, 93--101. https://doi.org/10.1145/ 2960310.2960316Google ScholarGoogle Scholar
  156. Terence Parr and Kathleen Fisher. 2011. LL(*): The Foundation of the ANTLR Parser Generator. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '11). ACM, New York, NY, USA, 425--436. https://doi.org/10.1145/1993498.1993548Google ScholarGoogle Scholar
  157. D. N. Perkins and Fay Martin. 1986. Fragile Knowledge and Neglected Strategies in Novice Programmers. In Papers Presented at the First Workshop on Empirical Studies of Programmers on Empirical Studies of Programmers. Ablex Publishing Corp., Norwood, NJ, USA, 213--229. http://dl.acm.org/citation.cfm?id=21842. 28896Google ScholarGoogle Scholar
  158. Raymond S. Pettit, John Homer, and Roger Gee. 2017. Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive.. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE '17). ACM, New York, NY, USA, 465--470. https://doi.org/10.1145/3017680.3017768Google ScholarGoogle Scholar
  159. Phitchaya Mangpo Phothilimthana and Sumukh Sridhara. 2017. High-Coverage Hint Generation for Massive Courses: Do Automated Hints Help CS1 Students?. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '17). ACM, New York, NY, USA, 182--187. https://doi.org/10.1145/3059009.3059058Google ScholarGoogle Scholar
  160. James Prather. 2018. Beyond Automated Assessment: Building Metacognitive Awareness in Novice Programmers in CS1. Ph.D. Dissertation. Nova Southeastern University.Google ScholarGoogle Scholar
  161. James Prather, Raymond Pettit, Brett A. Becker, Paul Denny, Dastyni Loksa, Alani Peters, Zachary Albrecht, and Krista Masci. 2019. First Things First: Providing Metacognitive Scaffolding for Interpreting Problem Prompts. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE '19). ACM, New York, NY, USA, 531--537. https://doi.org/10.1145/3287324.3287374Google ScholarGoogle Scholar
  162. James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research (ICER '18). ACM, New York, NY, USA, 41--50. https://doi.org/10.1145/3230977.3230981Google ScholarGoogle Scholar
  163. James Prather, Raymond Pettit, Kayla Holcomb McMurry, Alani Peters, John Homer, Nevan Simone, and Maxine Cohen. 2017. On Novices' Interaction with Compiler Error Messages: A Human Factors Approach. In Proceedings of the 2017 ACM Conference on International Computing Education Research (ICER '17). ACM, New York, NY, USA, 74--82. https://doi.org/10.1145/3105726.3106169Google ScholarGoogle Scholar
  164. David Pritchard. 2015. Frequency Distribution of Error Messages. In Proceedings of the 6th Workshop on Evaluation and Usability of Programming Languages and Tools. 1--8. https://doi.org/10.1145/nnnnnnn.nnnnnnn arXiv:1509.07238v1Google ScholarGoogle Scholar
  165. Yizhou Qian and James Lehman. 2017. Students' Misconceptions and Other Difficulties in Introductory Programming: A Literature Review. ACM Transactions on Computing Education 18, 1, Article 1 (Oct. 2017), 24 pages. https: //doi.org/10.1145/3077618Google ScholarGoogle Scholar
  166. Keith Quille, Roisin Faherty, Susan Bergin, and Brett A. Becker. 2018. Second Level Computer Science: The Irish K-12 Journey Begins. In Proceedings of the 18th Koli Calling International Conference on Computing Education Research (Koli Calling '18). ACM, New York, NY, USA, Article 22, 5 pages. https://doi.org/10. 1145/3279720.3279742Google ScholarGoogle Scholar
  167. Timothy Rafalski, P. Merlin Uesbeck, Cristina Panks-Meloney, Patrick Daleiden, William Allee, Amelia Mcnamara, and Andreas Stefik. 2019. A Randomized Controlled Trial on the Wild Wild West of Scientific Computing with Student Learners. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER '19). ACM, New York, NY, USA, 239--247. https://doi. org/10.1145/3291279.3339421Google ScholarGoogle Scholar
  168. Vincent Rahli, Joe Wells, John Pirie, and Fairouz Kamareddine. 2015. Skalpel: A Type Error Slicer for Standard ML. Electronic Notes in Theoretical Computer Science 312 (2015), 197--213. https://doi.org/10.1016/j.entcs.2015.04.012Google ScholarGoogle Scholar
  169. Vincent Rahli, Joe Wells, John Pirie, and Fairouz Kamareddine. 2017. Skalpel: A Constraint-based Type Error Slicer for Standard ML. Journal of Symbolic Computation 80 (May 2017), 164--208. https://doi.org/10.1016/j.jsc.2016.07.013Google ScholarGoogle Scholar
  170. Kyle Reestman and Brian Dorn. 2019. Native Language's Effect on Java Compiler Errors. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER '19). ACM, New York, NY, USA, 249--257. https://doi. org/10.1145/3291279.3339423Google ScholarGoogle Scholar
  171. Charles Reis and Robert Cartwright. 2004. Taming a Professional IDE for the Classroom. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '04). ACM, New York, NY, USA, 156--160. https://doi. org/10.1145/971300.971357Google ScholarGoogle Scholar
  172. H. G. Rice. 1953. Classes of Recursively Enumerable Sets and Their Decision Problems. Trans. Amer. Math. Soc. 74, 2 (1953), 358--366. http://www.jstor.org/ stable/1990888Google ScholarGoogle Scholar
  173. Liam Rigby, Paul Denny, and Andrew Luxton-Reilly. 2020. A Miss is as Good as a Mile: Off-By-One Errors and Arrays in an Introductory Programming Course. In Proceedings of the 22nd Australasian Computing Education Conference (ACE '20). ACM, New York, NY, USA. https://doi.org/10.1145/3373165.3373169Google ScholarGoogle Scholar
  174. Peter C. Rigby and Suzanne Thompson. 2005. Study of Novice Programmers Using Eclipse and Gild. In Proceedings of the 2005 OOPSLA Workshop on Eclipse Technology eXchange (eclipse '05). ACM, New York, NY, USA, 105--109. https: //doi.org/10.1145/1117696.1117718Google ScholarGoogle Scholar
  175. Anthony Robins, Janet Rountree, and Nathan Rountree. 2003. Learning and Teaching Programming: A Review and Discussion. Computer Science Education 13, 2 (2003), 137--172. https://doi.org/10.1076/csed.13.2.137.14200Google ScholarGoogle Scholar
  176. Christine Rogerson and Elsje Scott. 2017. The Fear Factor: How It Affects Students Learning to Program in a Tertiary Environment. Journal of Information Technology Education: Research 9 (2017), 147--171. https://doi.org/10.28945/1183Google ScholarGoogle Scholar
  177. Saul Rosen, Robert A. Spurgeon, and Joel K. Donnelly. 1965. PUFFT - The Purdue University Fast FORTRAN Translator. Commun. ACM 8, 11 (nov 1965), 661--666. https://doi.org/10.1145/365660.365671Google ScholarGoogle Scholar
  178. Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg, and Collin Winter. 2015. Tricorder: Building a Program Analysis Ecosystem. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE '15). IEEE Press, Piscataway, NJ, USA, 598--608. http://dl.acm.org/citation.cfm?id= 2818754.2818828Google ScholarGoogle Scholar
  179. Advait Sarkar. 2015. The Impact of Syntax Colouring on Program Comprehension. Proceedings of the 26th Annual Conference of the Psychology of Programming Interest Group (PPIG '15) (2015), 49--58. http://www.ppig.org/library/paper/ impact-syntax-colouring-program-comprehensionGoogle ScholarGoogle Scholar
  180. Thomas Schilling. 2012. Constraint-Free Type Error Slicing. In Proceedings of the 12th International Conference on Trends in Functional Programming (TFP'11). Springer-Verlag, Berlin, Heidelberg, 1--16. https://doi.org/10.1007/ 978--3--642--32037--8_1Google ScholarGoogle Scholar
  181. Jean Scholtz and Susan Wiedenbeck. 1993. Using Unfamiliar Programming Languages: The Effects on Expertise. Interacting with Computers 5, 1 (1993), 13--30.Google ScholarGoogle Scholar
  182. Tom Schorsch. 1995. CAP: An Automated Self-assessment Tool to Check Pascal Programs for Syntax, Logic and Style Errors. In Proceedings of the Twenty-sixth SIGCSE Technical Symposium on Computer Science Education (SIGCSE '95). ACM, New York, NY, USA, 168--172. https://doi.org/10.1145/199688.199769Google ScholarGoogle Scholar
  183. Hyunmin Seo, Caitlin Sadowski, Sebastian Elbaum, Edward Aftandilian, and Robert Bowdidge. 2014. Programmers' Build Errors: A Case Study (at Google). In Proceedings of the 36th International Conference on Software Engineering - ICSE 2014. ACM Press, New York, New York, USA, 724--734. https://doi.org/10.1145/ 2568225.2568255Google ScholarGoogle Scholar
  184. Alejandro Serrano and Jurriaan Hage. 2016. Type Error Diagnosis for Embedded DSLs by Two-Stage Specialized Type Rules. In Proceedings of the 25th European Symposium on Programming Languages and Systems - Volume 9632. Springer- Verlag New York, Inc., New York, NY, USA, 672--698. https://doi.org/10.1007/ 978--3--662--49498--1_26Google ScholarGoogle Scholar
  185. Dale Shaffer, Wendy Doube, and Juhani Tuovinen. 2003. Applying Cognitive Load Theory to Computer Science Education.. In Proceedings of the 15th Annual Workshop of the Psychology of Programming Interest Group (PPIG '03). 333--346. http://ppig.org/library/paper/ applying-cognitive-load-theory-computer-science-educationGoogle ScholarGoogle Scholar
  186. W. J. Shaw. 1989. Making APL Error Messages Kinder and Gentler. In Conference Proceedings on APL As a Tool of Thought (APL '89). ACM, New York, NY, USA, 320--324. https://doi.org/10.1145/75144.75188Google ScholarGoogle Scholar
  187. Ben Shneiderman. 1982. Designing Computer System Messages. Commun. ACM 25, 9 (1982), 610--611. https://doi.org/10.1145/358628.358639Google ScholarGoogle Scholar
  188. Ben Shneiderman and Catherine Plaisant. 2004. Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th ed.). Pearson Addison Wesley.Google ScholarGoogle Scholar
  189. M. E. Sime, A. T. Arblaster, and T. R. Green. 1977. Structuring the Programmer's Task. Journal of Occupational Psychology 50, 3 (sep 1977), 205--216. https://doi. org/10.1111/j.2044--8325.1977.tb00376.xGoogle ScholarGoogle Scholar
  190. Jaime Spacco, Paul Denny, Brad Richards, David Babcock, David Hovemeyer, James Moscola, and Robert Duvall. 2015. Analyzing StudentWork Patterns Using Programming Exercise Data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE '15). ACM, New York, NY, USA, 18--23. https://doi.org/10.1145/2676723.2677297Google ScholarGoogle Scholar
  191. Andreas Stefik and Stefan Hanenberg. 2014. The Programming Language Wars: Questions and Responsibilities for the Programming Language Community. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (Onward! 2014). ACM, New York, NY, USA, 283--299. https://doi.org/10.1145/2661136.2661156Google ScholarGoogle Scholar
  192. Andreas Stefik, Bonita Sharif, Brad. A. Myers, and Stefan Hanenberg. 2018. Evidence About Programmers for Programming Language Design (Dagstuhl Seminar 18061). Dagstuhl Reports 8, 2 (2018), 1--25. https://doi.org/10.4230/ DagRep.8.2.1Google ScholarGoogle Scholar
  193. Andreas Stefik and Susanna Siebert. 2013. An Empirical Investigation into Programming Language Syntax. ACM Transactions on Computing Education 13, 4, Article 19 (Nov. 2013), 40 pages. https://doi.org/10.1145/2534973Google ScholarGoogle Scholar
  194. S Suhailan, S Abdul Samad, and M A Burhanuddin. 2014. A Perspective of Automated Programming Error Feedback Approaches in Problem Solving Exercises. Journal of Theoretical and Applied Information Technology 70, 1 (2014), 121--129. http://www.jatit.org/volumes/Vol70No1/16Vol70No1.pdfGoogle ScholarGoogle Scholar
  195. John Sweller. 1988. Cognitive Load During Problem Solving: Effects on Learning. Cognitive science 12, 2 (1988), 257--285. https://www.sciencedirect.com/science/ article/pii/0364021388900237Google ScholarGoogle Scholar
  196. Emily S. Tabanao, Ma. Mercedes T. Rodrigo, and Matthew C. Jadud. 2011. Predicting At-risk Novice Java Programmers Through the Analysis of Online Protocols. In Proceedings of the 7th International Workshop on Computing Education Research (ICER '11). ACM, New York, NY, USA, 85--92. https: //doi.org/10.1145/2016911.2016930Google ScholarGoogle Scholar
  197. Tim Teitelbaum and Thomas Reps. 1981. The Cornell Program Synthesizer: A Syntax-Directed Programming Environment. Commun. ACM 24, 9 (sep 1981), 563--573. https://doi.org/10.1145/358746.358755Google ScholarGoogle Scholar
  198. Warren Teitelman and Larry Masinter. 1981. The Interlisp Programming Environment. Computer 14, 4 (1981), 25--33. https://doi.org/10.1109/C-M.1981.220410Google ScholarGoogle Scholar
  199. Emillie Thiselton and Christoph Treude. 2019. Enhancing Python Compiler Error Messages via Stack Overflow. In Proceedings of the 19th International Symposium on Empirical Software Engineering and Measurement (ESEM '19). arXiv:1906.11456. http://arxiv.org/abs/1906.11456Google ScholarGoogle Scholar
  200. Suzanne Marie Thompson. 2006. An Exploratory Study of Novice Programming Experiences and Errors. Masters Thesis. University of Victoria.Google ScholarGoogle Scholar
  201. Warren Toomey. 2011. Quantifying The Incidence of Novice Programmers' Errors. (2011), 5 pages. https://minnie.tuhs.org/Programs/BlueJErrors/arjen_ draft.pdfGoogle ScholarGoogle Scholar
  202. V. Javier Traver. 2010. On Compiler Error Messages: What They Say and What They Mean. Advances in Human-Computer Interaction 2010, Article 3 (Jan. 2010), 26 pages. https://doi.org/10.1155/2010/602570Google ScholarGoogle Scholar
  203. Kota Uchida and Katsuhiko Gondow. 2016. C-Helper: C Latent-error Static/Heuristic Checker for Novice Programmers. In Proceedings of the 8th International Conference on Computer Supported Education (CSEDU 2016). SciTePress - Science and Technology Publications, Lda, Portugal, 321--329. https://doi.org/10. 5220/0005797703210329Google ScholarGoogle Scholar
  204. Miguel Ulloa. 1983. A survey of run-time and logic errors in a classroom environment. ACM SIGCUE Outlook 17, 3 (1983), 21--25. https://doi.org/10.1145/ 1045078.1045081Google ScholarGoogle Scholar
  205. Leo C. Ureel II and Charles Wallace. 2019. Automated Critique of Early Programming Antipatterns. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 738--744. https://doi.org/10.1145/3287324.3287463Google ScholarGoogle Scholar
  206. Christopher Watson, Frederick W.B. Li, and Jamie L. Godwin. 2014. No Tests Required: Comparing Traditional and Dynamic Predictors of Programming Success. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE '14). ACM, New York, NY, USA, 469--474. https://doi.org/10.1145/2538862.2538930Google ScholarGoogle Scholar
  207. Christopher Watson, Frederick W. B. Li, and Jamie L. Godwin. 2012. BlueFix: Using Crowd-sourced Feedback to Support Programming Students in Error Diagnosis and Repair. In Proceedings of the 11th International Conference on Advances in Web-Based Learning (ICWL'12). Springer-Verlag, Berlin, Heidelberg, 228--239. https://doi.org/10.1007/978--3--642--33642--3_25Google ScholarGoogle Scholar
  208. Richard L.Wexelblat. 1976. Maxims for Malfeasant Designers, or How to Design Languages to Make Programming As Difficult As Possible. In Proceedings of the 2nd International Conference on Software Engineering (ICSE '76). IEEE Computer Society Press, Los Alamitos, CA, USA, 331--336. http://dl.acm.org/citation.cfm? id=800253.807695Google ScholarGoogle Scholar
  209. Emily Wilska. 2004. Non-Fatal Errors : Creating Usable , Effective Error Messages. http://www.writersua.com/articles/message/index.html. (2004).Google ScholarGoogle Scholar
  210. Niklaus Wirth. 1968. PL360, a Programming Language for the 360 Computers. J. ACM 15, 1 (Jan. 1968), 37--74. https://doi.org/10.1145/321439.321442Google ScholarGoogle Scholar
  211. Alexander William Wong, Amir Salimi, Shaiful Chowdhury, and Abram Hindle. 2019. Syntax and Stack Overflow: A Methodology for Extracting a Corpus of Syntax Errors and Fixes. (Jul 2019). arXiv:1907.07803 http://arxiv.org/abs/1907. 07803Google ScholarGoogle Scholar
  212. John Wrenn and Shriram Krishnamurthi. 2017. Error Messages Are Classifiers: A Process to Design and Evaluate Error Messages. In Proceedings of the 2017 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2017). ACM, New York, NY, USA, 134--147. https://doi.org/10.1145/3133850.3133862Google ScholarGoogle Scholar
  213. Baijun Wu, John Peter Campora III, and Sheng Chen. 2017. Learning User Friendly Type-error Messages. Proc. ACM Program. Lang. 1, OOPSLA, Article 106 (Oct. 2017), 29 pages. https://doi.org/10.1145/3133930Google ScholarGoogle Scholar
  214. Stelios Xinogalos, Maya Satratzemi, and Vassilios Dagdilelis. 2006. An Introduction to Object-Oriented Programming with a Didactic Microworld: objectKarel. Computers and Education 47, 2 (2006), 148--171. https://doi.org/10.1016/j.compedu. 2004.09.005Google ScholarGoogle Scholar
  215. Stelios Xinogalos, Maya Satratzemi, and Christos Malliarakis. 2017. Microworlds, Games, Animations, Mobile Apps, Puzzle Editors and More: What is Important for an Introductory Programming Environment? Education and Information Technologies 22, 1 (Jan. 2017), 145--176. https://doi.org/10.1007/s10639-015--9433--1Google ScholarGoogle Scholar
  216. Andreas Zeller. 1999. Yesterday, My Program Worked. Today, It Does Not. Why?. In Proceedings of the 7th European Software Engineering Conference Held Jointly with the 7th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE '99). Springer-Verlag, Berlin, Heidelberg, 253--267. http://dl.acm.org/citation.cfm?id=318773.318946Google ScholarGoogle Scholar
  217. Danfeng Zhang, Andrew C. Myers, Dimitrios Vytiniotis, and Simon Peyton- Jones. 2015. Diagnosing Type Errors with Class. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '15). ACM, New York, NY, USA, 12--21. https://doi.org/10.1145/2737924.2738009Google ScholarGoogle Scholar
  218. Lu Zhang. 2012. Empirical Design and Analysis of a Defect Taxonomy for Novice Programmers. Masters Thesis (by Research). The University ofWestern Australia. https://research-repository.uwa.edu.au/files/3232803/Zhang_Lu_2012.pdfGoogle ScholarGoogle Scholar
  219. Daniel Zingaro, Michelle Craig, Leo Porter, Brett A. Becker, Yingjun Cao, Phill Conrad, Diana Cukierman, Arto Hellas, Dastyni Loksa, and Neena Thota. 2018. Achievement Goals in CS1: Replication and Extension. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 687--692. https://doi.org/10.1145/3159450.3159452Google ScholarGoogle Scholar

Index Terms

  1. Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research

                Comments

                Login options

                Check if you have access through your login credentials or your institution to get full access on this article.

                Sign in

                Full Access

                Get this Publication

                • Published in

                  ACM Conferences cover image
                  ITiCSE-WGR '19: Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education
                  December 2019
                  218 pages
                  ISBN:9781450375672
                  DOI:10.1145/3344429

                  Copyright © 2019 ACM

                  Publisher

                  Association for Computing Machinery

                  New York, NY, United States

                  Author Tags

                  Qualifiers

                  • research-article
                View Table of Contents
                About Cookies On This Site

                We use cookies to ensure that we give you the best experience on our website.

                Learn more

                Got it!