Abstract
We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong.
We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to Programming course at MIT (6.00) and MITx (6.00x). Our results show that relatively simple error models can correct on average 64% of all incorrect submissions in our benchmark set.
- A. Adam and J.-P. H. Laurent. LAURA, A System to Debug Student Programs. Artif. Intell., 15(1-2):75--122, 1980.Google Scholar
Digital Library
- U. Ahmed, S. Gulwani, and A. Karkare. Automatically generating problems and solutions for natural deduction. In IJCAI, 2013.Google Scholar
Digital Library
- R. Alur, L. D'Antoni, S. Gulwani, D. Kini, and M. Viswanathan. Automated grading of dfa constructions. In IJCAI, 2013.Google Scholar
Digital Library
- E. Andersen, S. Gulwani, and Z. Popovic. A trace-based framework for analyzing and synthesizing educational progressions. In CHI, 2013. Google Scholar
Digital Library
- A. Arcuri. On the automation of fixing software bugs. In ICSE Companion, 2008. Google Scholar
Digital Library
- T. Ball, M. Naik, and S. K. Rajamani. From symptom to cause: localizing errors in counterexample traces. In POPL, 2003. Google Scholar
Digital Library
- M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In UIST, 2010. Google Scholar
Digital Library
- R. Bodík, S. Chandra, J. Galenson, D. Kimelman, N. Tung, S. Barman, and C. Rodarmor. Programming with angelic nondeterminism. In POPL, 2010. Google Scholar
Digital Library
- C. Daly. Roboprof and an introductory computer programming course. ITiCSE, 1999. Google Scholar
Digital Library
- V. Debroy and W. Wong. Using mutation to automatically suggest fixes for faulty programs. In ICST, 2010. Google Scholar
Digital Library
- C. Douce, D. Livingstone, and J. Orwell. Automatic test-based assessment of programming: A review. J. Educ. Resour. Comput., 5(3), Sept. 2005. Google Scholar
Digital Library
- P. Ertmer, J. Richardson, B. Belland, D. Camin, P. Connolly, G. Coulthard, K. Lei, and C. Mong. Using peer feedback to enhance the quality of student online postings: An exploratory study. Journal of Computer-Mediated Communication, 12(2):412--433, 2007.Google Scholar
Cross Ref
- R. G. Farrell, J. R. Anderson, and B. J. Reiser. An interactive computer-based tutor for lisp. In AAAI, 1984.Google Scholar
- S. Forrest, T. Nguyen, W. Weimer, and C. L. Goues. A genetic programming approach to automated software repair. In GECCO, 2009. Google Scholar
Digital Library
- A. Groce, S. Chaki, D. Kroening, and O. Strichman. Error explanation with distance metrics. STTT, 8(3):229--247, 2006. Google Scholar
Digital Library
- S. Gulwani, S. Srivastava, and R. Venkatesan. Program analysis as constraint solving. In PLDI, 2008. Google Scholar
Digital Library
- S. Gulwani, V. A. Korthikanti, and A. Tiwari. Synthesizing geometry constructions. In PLDI, 2011. Google Scholar
Digital Library
- S. Gulwani, W. R. Harris, and R. Singh. Spreadsheet data manipulation using examples. In CACM, 2012. Google Scholar
Digital Library
- P. Hawkins, A. Aiken, K. Fisher, M. C. Rinard, and M. Sagiv. Data representation synthesis. In PLDI, 2011. Google Scholar
Digital Library
- P. Hawkins, A. Aiken, K. Fisher, M. C. Rinard, and M. Sagiv. Concurrent data representation synthesis. In PLDI, 2012. Google Scholar
Digital Library
- J. B. Hext and J.W.Winings. An automatic grading scheme for simple programming exercises. Commun. ACM, 12(5), May 1969. Google Scholar
Digital Library
- D. Jackson and M. Usher. Grading student programs using assyst. SIGCSE, 1997. Google Scholar
Digital Library
- B. Jobstmann, A. Griesmayer, and R. Bloem. Program repair as a game. In CAV, pages 226--238, 2005. Google Scholar
Digital Library
- W. L. Johnson and E. Soloway. Proust: Knowledge-based program understanding. IEEE Trans. Software Eng., 11(3):267--275, 1985. Google Scholar
Digital Library
- M. Jose and R. Majumdar. Cause clue clauses: error localization using maximum satisfiability. In PLDI, 2011. Google Scholar
Digital Library
- U. Junker. QUICKXPLAIN: preferred explanations and relaxations for over-constrained problems. In AAAI, 2004. Google Scholar
Digital Library
- R. Könighofer and R. P. Bloem. Automated error localization and correction for imperative programs. In FMCAD, 2011. Google Scholar
Digital Library
- C. Kulkarni and S. R. Klemmer. Learning design wisdom by augmenting physical studio critique with online self-assessment. Technical report, Stanford University, 2012.Google Scholar
- V. Kuncak, M. Mayer, R. Piskac, and P. Suter. Complete functional synthesis. PLDI, 2010. Google Scholar
Digital Library
- G. Little, L. B. Chilton, M. Goldman, and R. C. Miller. Turkit: human computation algorithms on mechanical turk. In UIST, 2010. Google Scholar
Digital Library
- W. R. Murray. Automatic program debugging for intelligent tutoring systems. Computational Intelligence, 3:1--16, 1987.Google Scholar
Cross Ref
- W. Sack, E. Soloway, and P. Weingrad. From PROUST to CHIRON: Its design as iterative engineering: Intermediate results are important! In In J.H. Larkin and R.W. Chabay (Eds.), Computer-Assisted Instruction and Intelligent Tutoring Systems: Shared Goals and Complementary Approaches., pages 239--274, 1992.Google Scholar
- R. Singh and S. Gulwani. Learning semantic string transformations from examples. PVLDB, 5, 2012. Google Scholar
Digital Library
- R. Singh and A. Solar-Lezama. Synthesizing data structure manipulations from storyboards. In SIGSOFT FSE, 2011. Google Scholar
Digital Library
- R. Singh, S. Gulwani, and S. K. Rajamani. Automatically generating algebra problems. In AAAI, 2012.Google Scholar
- R. Singh, S. Gulwani, and A. Solar-Lezama. Automated semantic grading of programs. CoRR, abs/1204.1751, 2012.Google Scholar
- A. Solar-Lezama. Program Synthesis By Sketching. PhD thesis, EECS Dept., UC Berkeley, 2008. Google Scholar
Digital Library
- A. Solar-Lezama, R. Rabbah, R. Bodik, and K. Ebcioglu. Programming by sketching for bit-streaming programs. In PLDI, 2005. Google Scholar
Digital Library
- E. Soloway, B. P. Woolf, E. Rubin, and P. Barth. Meno-II: An Intelligent Tutoring System for Novice Programmers. In IJCAI, 1981. Google Scholar
Digital Library
- S. Srivastava, S. Gulwani, and J. Foster. From program verification to program synthesis. POPL, 2010. Google Scholar
Digital Library
- S. S. Staber, B. Jobstmann, and R. P. Bloem. Finding and fixing faults. In Correct Hardware Design and Verification Methods, Lecture notes in computer science, pages 35--49, 2005. Google Scholar
Digital Library
- M. Vechev, E. Yahav, and G. Yorsh. Abstraction-guided synthesis of synchronization. In POPL, 2010. Google Scholar
Digital Library
- D. S. Weld, E. Adar, L. Chilton, R. Hoffmann, and E. Horvitz. Personalized online education - a crowdsourcing challenge. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.Google Scholar
- A. Zeller and R. Hildebrandt. Simplifying and isolating failureinducing input. IEEE Transactions on Software Engineering, 28:183--200, 2002. Google Scholar
Digital Library
Recommendations
Search, align, and repair: data-driven feedback generation for introductory programming exercises
PLDI 2018: Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and ImplementationThis paper introduces the “Search, Align, and Repair” data-driven program repair framework to automate feedback generation for introductory programming exercises. Distinct from existing techniques, our goal is to develop an efficient, fully automated, ...
Automatic Grading and Feedback using Program Repair for Introductory Programming Courses
ITiCSE '17: Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science EducationWe present GradeIT, a system that combines the dual objectives of automated grading and program repairing for introductory programming courses (CS1). Syntax errors pose a significant challenge for testcase-based grading as it is difficult to ...
Automated feedback generation for introductory programming assignments
PLDI '13: Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and ImplementationWe present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that ...







Comments