Abstract
Providing feedback on programming assignments is a tedious task for the instructor, and even impossible in large Massive Open Online Courses with thousands of students. Previous research has suggested that program repair techniques can be used to generate feedback in programming education. In this paper, we present a novel fully automated program repair algorithm for introductory programming assignments. The key idea of the technique, which enables automation and scalability, is to use the existing correct student solutions to repair the incorrect attempts. We evaluate the approach in two experiments: (I) We evaluate the number, size and quality of the generated repairs on 4,293 incorrect student attempts from an existing MOOC. We find that our approach can repair 97% of student attempts, while 81% of those are small repairs of good quality. (II) We conduct a preliminary user study on performance and repair usefulness in an interactive teaching setting. We obtain promising initial results (the average usefulness grade 3.4 on a scale from 1 to 5), and conclude that our approach can be used in an interactive setting.
Supplemental Material
- Anne Adam and Jean-Pierre Laurent. 1980. LAURA, a system to debug student programs. Artificial Intelligence 15, 1âĂŞ2 (1980), 75 – 122. Google Scholar
Digital Library
- Andrea Arcuri. 2008. On the Automation of Fixing Software Bugs. In Companion of the 30th International Conference on Software Engineering (ICSE Companion ’08) . ACM, New York, NY, USA, 1003–1006. Google Scholar
Digital Library
- D. Beyer, A. Cimatti, A. Griggio, M. E. Keremoglu, S. F. University, and R. Sebastiani. 2009. Software model checking via large-block encoding. In 2009 Formal Methods in Computer-Aided Design. 25–32.Google Scholar
- Satish Chandra, Emina Torlak, Shaon Barman, and Rastislav Bodik. 2011. Angelic Debugging. In Proceedings of the 33rd International Conference on Software Engineering (ICSE ’11) . ACM, New York, NY, USA, 121–130. Google Scholar
Digital Library
- Loris D’Antoni, Roopsha Samanta, and Rishabh Singh. 2016. Qlose: Program Repair with Quantitative Objectives. In Computer Aided Verification - 28th International Conference, CAV 2016, Toronto, ON, Canada, July 17-23, 2016, Proceedings, Part II . 383–401.Google Scholar
- Rajdeep Das, Umair Z. Ahmed, Amey Karkare, and Sumit Gulwani. 2016. Prutor: A System for Tutoring CS1 and Collecting Student Programs for Analysis. CoRR abs/1608.03828 (2016). http://arxiv.org/ abs/1608.03828Google Scholar
- V. Debroy and W.E. Wong. 2010. Using Mutation to Automatically Suggest Fixes for Faulty Programs. In Software Testing, Verification and Validation (ICST), 2010 Third International Conference on . 65–74. Google Scholar
Digital Library
- Yulia Demyanova, Helmut Veith, and Florian Zuleger. 2013. On the concept of variable roles and its use in software analysis. In Formal Methods in Computer-Aided Design, FMCAD 2013, Portland, OR, USA, October 20-23, 2013 . 226–230. http://ieeexplore.ieee.org/xpl/freeabs_all. jsp?arnumber=6679414Google Scholar
- A. Drummond, Y. Lu, S. Chaudhuri, C. Jermaine, J. Warren, and S. Rixner. 2014. Learning to Grade Student Programs in a Massive Open Online Course. In Data Mining (ICDM), 2014 IEEE International Conference on . 785–790. Google Scholar
Digital Library
- Stephanie Forrest, ThanhVu Nguyen, Westley Weimer, and Claire Le Goues. 2009. A Genetic Programming Approach to Automated Software Repair. In Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation (GECCO ’09) . ACM, New York, NY, USA, 947–954. Google Scholar
Digital Library
- Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip Guo, and Robert Miller. 2014. OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale. In Proceedings of the Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST’14 Adjunct) . ACM, New York, NY, USA, 129–130. Google Scholar
Digital Library
- Divya Gopinath, Muhammad Zubair Malik, and Sarfraz Khurshid. 2011. Specification-based Program Repair Using SAT. In Proceedings of the 17th International Conference on Tools and Algorithms for the Construction and Analysis of Systems: Part of the Joint European Conferences on Theory and Practice of Software (TACAS’11/ETAPS’11) . SpringerVerlag, Berlin, Heidelberg, 173–188. http://dl.acm.org/citation.cfm? id=1987389.1987408 Google Scholar
Digital Library
- C. Le Goues, N. Holtschulte, E. K. Smith, Y. Brun, P. Devanbu, S. Forrest, and W. Weimer. 2015. The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs. IEEE Transactions on Software Engineering 41, 12 (Dec 2015), 1236–1256.Google Scholar
Digital Library
- Sumit Gulwani, Ivan Radiček, and Florian Zuleger. 2014. Feedback Generation for Performance Problems in Introductory Programming Assignments. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014) . ACM, New York, NY, USA, 41–51. Google Scholar
Digital Library
- Sumit Gulwani, Ivan Radiček, and Florian Zuleger. 2018. Automated Clustering and Program Repair for Introductory Programming Assignments. CoRR abs/1603.03165 (2018). arXiv: 1603.03165 http://arxiv.org/abs/1603.03165 Google Scholar
Digital Library
- Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D’Antoni, and Björn Hartmann. 2017. Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale ([email protected] ’17) . ACM, New York, NY, USA, 89–98. Google Scholar
Digital Library
- Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of Recent Systems for Automatic Assessment of Programming Assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling ’10) . ACM, New York, NY, USA, 86–93. Google Scholar
Digital Library
- Barbara Jobstmann, Andreas Griesmayer, and Roderick Bloem. 2005. Program Repair As a Game. In Proceedings of the 17th International Conference on Computer Aided Verification (CAV’05) . Springer-Verlag, Berlin, Heidelberg, 226–238. Google Scholar
Digital Library
- Shalini Kaleeswaran, Anirudh Santhiar, Aditya Kanade, and Sumit Gulwani. 2016. Semi-supervised Verified Feedback Generation. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2016) . ACM, New York, NY, USA, 739–750. Google Scholar
Digital Library
- Yalin Ke, Kathryn T. Stolee, Claire Le Goues, and Yuriy Brun. 2015. Repairing Programs with Semantic Code Search (T). In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (ASE ’15) . IEEE Computer Society, Washington, DC, USA, 295–306.Google Scholar
Digital Library
- Dohyeong Kim, Yonghwi Kwon, Peng Liu, I. Luk Kim, David Mitchel Perry, Xiangyu Zhang, and Gustavo Rodriguez-Rivera. 2016. Apex: Automatic Programming Assignment Error Explanation. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2016) . ACM, New York, NY, USA, 311–327. Google Scholar
Digital Library
- Robert Könighofer and Roderick Bloem. 2011. Automated Error Localization and Correction for Imperative Programs. In Proceedings of the International Conference on Formal Methods in ComputerAided Design (FMCAD ’11) . FMCAD Inc, Austin, TX, 91–100. http: //dl.acm.org/citation.cfm?id=2157654.2157671 Google Scholar
Digital Library
- Fan Long and Martin Rinard. 2016. Automatic Patch Generation by Learning Correct Code. SIGPLAN Not. 51, 1 (Jan. 2016), 298–312. Google Scholar
Digital Library
- Ken Masters. 2011. A Brief Guide To Understanding MOOCs. The Internet Journal of Medical Education 1, 2 (2011).Google Scholar
- Robin Milner. 1971. An Algebraic Definition of Simulation Between Programs . Technical Report. Stanford, CA, USA. Google Scholar
- Andy Nguyen, Christopher Piech, Jonathan Huang, and Leonidas Guibas. 2014. Codewebs: Scalable Homework Search for Massive Open Online Programming Courses. In Proceedings of the 23rd International Conference on World Wide Web (WWW ’14) . ACM, New York, NY, USA, 491–502. Google Scholar
Digital Library
- Kelly Rivers and Kenneth R. Koedinger. 2017. Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor. International Journal of Artificial Intelligence in Education 27, 1 (01 Mar 2017), 37–64.Google Scholar
Cross Ref
- Reudismam Rolim, Gustavo Soares, Loris D’Antoni, Oleksandr Polozov, Sumit Gulwani, Rohit Gheyi, Ryo Suzuki, and Björn Hartmann. 2017. Learning Syntactic Program Transformations from Examples. In Proceedings of the 39th International Conference on Software Engineering (ICSE ’17) . IEEE Press, Piscataway, NJ, USA, 404–415. Google Scholar
Digital Library
- Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. 2013. Automated Feedback Generation for Introductory Programming Assignments. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’13) . ACM, New York, NY, USA, 15–26. Google Scholar
Digital Library
- Stefan Staber, Barbara Jobstmann, and Roderick Bloem. 2005. Finding and Fixing Faults. In Correct Hardware Design and Verification Methods, Dominique Borrione and Wolfgang Paul (Eds.). Lecture Notes in Computer Science, Vol. 3725. Springer Berlin Heidelberg, 35–49. Google Scholar
Digital Library
- Michael Striewe and Michael Goedicke. 2011. Using run time traces in automated programming tutoring. In ITiCSE. 303–307. Google Scholar
Digital Library
- Michael Striewe and Michael Goedicke. 2013. Trace Alignment for Automated Tutoring. In CAA.Google Scholar
- Ryo Suzuki, Gustavo Soares, Elena Glassman, Andrew Head, Loris D’Antoni, and Björn Hartmann. 2017. Exploring the Design Space of Automatically Synthesized Hints for Introductory Programming Assignments. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17) . ACM, New York, NY, USA, 2951–2958. Google Scholar
Digital Library
- Kuo-Chung Tai. 1979. The Tree-to-Tree Correction Problem. J. ACM 26, 3 (July 1979), 422–433. Google Scholar
Digital Library
- Nikolai Tillmann, Judith Bishop, R. Nigel Horspool, Daniel Perelman, and Tao Xie. 2014. Code Hunt: Searching for Secret Code for Fun. Proceedings of the International Conference on Software Engineering (Workshops) (June 2014). http://research.microsoft.com/apps/pubs/ default.aspx?id=210651 Google Scholar
Digital Library
- Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani, and Judith Bishop. 2013. Teaching and Learning Programming and Software Engineering via Interactive Gaming. In Proc. 35th International Conference on Software Engineering (ICSE 2013), Software Engineering Education (SEE) . http://www.cs.illinois.edu/homes/taoxie/publications/ icse13see-pex4fun.pdf Google Scholar
Digital Library
- Takeaki Uno. 1997. Algorithms for Enumerating All Perfect, Maximum and Maximal Matchings in Bipartite Graphs. In ISAAC. 92–101. Google Scholar
Digital Library
- Jooyong Yi, Umair Z. Ahmed, Amey Karkare, Shin Hwei Tan, and Abhik Roychoudhury. 2017. A Feasibility Study of Using Automated Program Repair for Introductory Programming Assignments. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017) . ACM, New York, NY, USA, 740–751. Google Scholar
Digital Library
- K. Zhang and D. Shasha. 1989. Simple Fast Algorithms for the Editing Distance Between Trees and Related Problems. SIAM J. Comput. 18, 6 (Dec. 1989), 1245–1262. Google Scholar
Digital Library
Index Terms
Automated clustering and program repair for introductory programming assignments
Recommendations
Automated clustering and program repair for introductory programming assignments
PLDI 2018: Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and ImplementationProviding feedback on programming assignments is a tedious task for the instructor, and even impossible in large Massive Open Online Courses with thousands of students. Previous research has suggested that program repair techniques can be used to ...
A feasibility study of using automated program repair for introductory programming assignments
ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software EngineeringDespite the fact an intelligent tutoring system for programming (ITSP) education has long attracted interest, its widespread use has been hindered by the difficulty of generating personalized feedback automatically. Meanwhile, automated program repair (...
Model and Program Repair via SAT Solving
Special Issue on MEMCODE 2015 and Regular Papers (Diamonds)We consider the subtractive model repair problem: given a finite Kripke structure M and a CTL formula η, determine if M contains a substructure M′ that satisfies η. Thus, M can be “repaired” to satisfy eta by deleting some transitions and states. We map ...







Comments