Understanding Programming Expertise: An Empirical Study of Phasic Brain Wave Changes

Abstract

Recent decades have seen a resurgence of interest in electroencephalography (EEG), as neuroscience develops new models of cognition and refines old ones, associating them with detectable indicators of brain activity. This article presents a more direct measure of programmer expertise, derived from noninvasive observation of the brain’s electrical activity. This article provides a foundational approach for investigating the role of expertise in programming language comprehension, showing that this electrical activity in the brain can indicate (1) prior programming experience by class level (current state of progression through an undergraduate computer science program), and (2) self-reported experience levels.

References

1. B. Adelson. 1984. When novices surpass experts: the difficulty of a task may increase with expertise. J. Exp. Psychol. 10, 3 (1984), 483--495.
2. E. W. Anderson, K. C. Potter, L. E. Matzen, J. F. Shepherd, G. a. Preston, and C. T. Silva. 2011. A user study of visualization effectiveness using EEG and cognitive load. Comput. Graph. Forum 30, 3 (June 2011), 791--800. DOI:http://dx.doi.org/10.1111/j.1467-8659.2011.01928.x
3. Pavlo Antonenko, Fred Paas, Roland Grabner, and Tamara van Gog. 2010. Using electroencephalography to measure cognitive load. Edu. Psychol. Rev 22, 4 (2010), 425--438.
4. Erik Arisholm, Hans Gallis, Tore Dyba, and Dag Sjoberg. 2007. Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Tran. Softw. Eng. 33, 2 (February 2007), 65--86. DOI:http://dx.doi.org/10.1109/TSE.2007.17
5. Alan D. Baddeley. 1992. Working memory: the interface between memory and cognition. J. Cogn. Neurosci. 4, 3 (January 1992), 281--8. DOI:http://dx.doi.org/10.1162/jocn.1992.4.3.281
6. Alan D. Baddeley. 2001. Is working memory still working? Am. Psychologist 56, 11 (2001), 851.
7. Alan D. Baddeley and Graham Hitch. 1974. Working memory. Psychol. Learn. Motivation 8 (1974), 47--89.
8. Stuart N. Baker. 2007. Oscillatory interactions between sensorimotor cortex and the periphery. Current Opinion Neurobiol. 17, 6 (2007), 649--655.
9. E. Basar. 1999. Brain Function and Oscillations. Integrative Brain Functions. Neurophysiology and Cognitive Processes, Vol. 2. Springer, Berlin.
10. Hans Berger. 1929. Über das elektroenkephalogramm des menschen. Archiv für Psychiatrie und Nervenkrankheiten 87 (1929), 527--570.
11. Chris Berka, Daniel J. Levendowski, Caitlin K. Ramsey, Gene Davis, Michelle N. Lumicao, Kay Stanney, Leah Reeves, Susan H. Regli, Patrice D. Tremoulet, and Kathleen Stibler. 2005. Evaluation of an EEG workload model in an Aegis simulation environment. In Biomonitoring for Physiological and Cognitive Performance During Military Operations, John A. Caldwell and Nancy Jo Wesensten (Eds.), Vol. 5797. 90--99. DOI:http://dx.doi.org/10.1117/12.598555
12. C. F. Bertholf and J. Scholtz. 1986. Program comprehension of literate programs by novice programmers. In Empirical Studies of Programmers: 5th Workshop. Ablex, Norwood, NJ, 28--47.
13. Richard Bornat, Saeed Dehnadi, and Simon. 2008. Mental models, consistency and programming aptitude. In Proceedings of the 7th Conference on Australasian Computing Education - Volume 78 (ACE’08). Australian Computer Society, Inc., Darlinghurst, Australia, 53--61. http://dl.acm.org/citation.cfm?id=1379249.1379253
14. Roland Brunken, Jan L. Plass, and Detlev Leutner. 2003. Direct measurement of cognitive load in multimedia learning. Educ. Psychologist 38, 1 (2003), 53--61.
15. Roberto Cabeza and Lars Nyberg. 2000. Imaging cognition II: an empirical review of 275 PET and fMRI studies. J. Cogn. Neurosci. 12, 1 (2000), 1--47.
16. William G. Chase and Herbert A. Simon. 1973. Perception in chess. Cogn. Psychol. 4, 1 (1973), 55--81.
17. Michelene T. H. Chi, Paul J. Feltovich, and Robert Glaser. 1981. Categorization and representation of physics problems by experts and novices*. Cogn. Sci. 5, 2 (1981), 121--152.
18. Andrew R. A. Conway, Nelson Cowan, Michael F. Bunting, David J. Therriault, and Scott R. B. Minkoff. 2002. A latent variable analysis of working memory capacity, short-term memory capacity, processing speed, and general fluid intelligence. Intelligence 30, 2 (2002), 163--183.
19. Nicholas R. Cooper, Adrian P. Burgess, Rodney J. Croft, and John H Gruzelier. 2006. Investigating evoked and induced electroencephalogram activity in task-related alpha power increases during an internally directed attention task. Neuroreport 17, 2 (February 2006), 205--208.
20. Joseph T. Coyne, Carryl Baldwin, Anna Cole, Ciara Sibley, and Daniel M. Roberts. 2009. Applying real time physiological measures of cognitive load to improve training. In Proceedings of the 5th International Conference on Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience: Held As Part of HCI International 2009 (FAC’09). Springer, Berlin, 469--478. DOI:http://dx.doi.org/10.1007/ 978-3-642-02812-0_55
21. Simon P. Davies. 1990. The nature and development of programming plans. Int. J. Man-Mach. Stud. 32, 4 (1990), 461--481.
22. Françoise Détienne and Frank Bott. 2002. Software Design--Cognitive Aspects. Springer, London.
23. Adrianus de Groot. 1965. Thought and Choice in Chess. Mouton, The Hague.
24. Dennis Egan and Barry Schwartz. 1979. Chunking in recall of symbolic drawings. Memory Cogn. 7, 2 (1979), 149--158.
25. Randall W. Engle, Stephen W. Tuholski, James E. Laughlin, and Andrew R. A. Conway. 1999. Working memory, short-term memory, and general fluid intelligence: a latent-variable approach. Journal of Experimental Psychology: General 128, 3 (1999), 309--331.
26. K. Anders Ericsson, Ralf Th. Krampe, and Clemens Tesch-romer. 1993. The role of deliberate practice in the acquisition of expert performance. Psychological Rev. (1993), 363--406.
27. Thomas Fritz, Andrew Begel, Sebastian C. Müller, Serap Yigit-Elliott, and Manuela Züger. 2014. Using psycho-physiological measures to assess task difficulty in software development. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, 402--413. DOI:http://dx.doi.org/10.1145/2568225.2568266
28. Laurent George, Maud Marchal, Loeiz Glondu, and L. Anatole. 2012. Combining brain-computer interfaces and haptics : detecting mental workload to adapt haptic assistance. EuroHaptics (2012), 1--12.
29. Alan Gevins, Michael E. Smith, Linda McEvoy, and Daphne Yu. 1997. High-resolution EEG mapping of cortical activation related to working memory: Effects of task difficulty, type of processing, and practice. Cerebral Cortex 7, 4 (1997), 374--385.
30. D. J. Gilmore and T. R. G. Green. 1988. Programming plans and programming expertise. Quart. J. of Exp. Psychol. 40, 3 (1988), 423--442.
31. F. Gobet. 1998. Expert memory: A comparison of four theories. Cognition 66, 2 (May 1998), 115--152.
32. R. H. Grabner, A. C. Neubauer, and E. Stern. 2006. Superior performance and neural efficiency: The impact of intelligence and expertise. Brain Res. Bull. 69, 4 (April 2006), 422--439.
33. R. H. Grabner, E. Stern, and A. C. Neubauer. 2003. When intelligence loses its impact: Neural efficiency during reasoning in a familiar area. Int. J. Psychophysiol. 49, 2 (Aug 2003), 89--98.
34. David Grimes, Desney S. Tan, Scott E. Hudson, Pradeep Shenoy, and Rajesh P. N. Rao. 2008. Feasibility and pragmatics of classifying working memory load with an electroencephalograph. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, New York, NY, 835--844. DOI:http://dx.doi.org/10.1145/1357054.1357187
35. Michael E. Hansen, Andrew Lumsdaine, and Robert L. Goldstone. 2012. Cognitive architectures: A way forward for the psychology of programming. In Proceedings of the ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. ACM, 27--38.
36. Sandra G. Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 904--908. DOI:http://dx.doi.org/10.1037/e577632012-009
37. B. D. Hatfield, A. J. Haufler, T. M. Hung, and T. W. Spalding. 2004. Electroencephalographic studies of skilled psychomotor performance. J. Clin. Neurophysiol. 21, 3 (2004), 144--156.
38. J. Allan Hobson and Edward F. Pace-Schott. 2002. The cognitive neuroscience of sleep: Neuronal systems, consciousness and learning. Nat. Rev. Neurosci. 3, 9 (2002), 679--693.
39. Michael Hoppe and Stefan Hanenberg. 2013. Do developers benefit from generic types?: An empirical comparison of generic and raw types in java. SIGPLAN Not. 48, 10 (October 2013), 457--474. DOI:http://dx.doi.org/10.1145/2544173.2509528
40. Ole Jensen, Jack Gelfand, John Kounios, and John E. Lisman. 2002. Oscillations in the alpha band (9--12 Hz) increase with memory load during retention in a short-term memory task. Cerebral Cortex 12, 8 (August 2002), 877--882. DOI:http://dx.doi.org/10.1093/cercor/12.8.877
41. Lloyd Kaufman, Barry Schwartz, Carlo Salustri, and Samuel J. Williamson. 1990. Modulation of spontaneous brain activity during mental imagery. J. Cogn. Neurosci. 2, 2 (January 1990), 124--132. DOI:http://dx.doi.org/10.1162/jocn.1990.2.2.124
42. Wolfgang Klimesch. 1996. Memory processes, brain oscillations and EEG synchronization. Int. J. Psychophysiol. 24, 1--2 (November 1996), 61--100.
43. Wolfgang Klimesch. 1999. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res. Rev. 29, 2--3 (April 1999), 169--95.
44. Wolfgang Klimesch, Paul Sauseng, and Christian Gerloff. 2003. Enhancing cognitive performance with repetitive transcranial magnetic stimulation at human individual alpha frequency. Eur. J. Neurosci. 17, 5 (March 2003), 1129--1133.
45. Wolfgang Klimesch, Paul Sauseng, and Simon Hanslmayr. 2007. EEG alpha oscillations: the inhibition-timing hypothesis. Brain Res. Rev. 53, 1 (January 2007), 63--88. DOI:http://dx.doi.org/10.1016/ j.brainresrev.2006.06.003
46. Wolfgang Klimesch, Bärbel Schack, and Paul Sauseng. 2005. The functional significance of theta and upper alpha oscillations. Exp. Psychol. 52, 2 (2005), 99--108.
47. Jeff Klingner, Barbara Tversky, and Pat Hanrahan. 2011. Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. Psychophysiology 48, 3 (March 2011), 323--32. DOI:http://dx.doi.org/10.1111/j.1469-8986.2010.01069.x
48. Andrew Jensen Ko and Bob Uttl. 2003. Individual differences in program comprehension strategies in unfamiliar programming systems. In Proceedings of the 11th IEEE International Workshop on Program Comprehension. IEEE, 2003, 175--184.
49. Kyle E. Mathewson, Chandramallika Basak, Edward L. Maclin, Kathy A. Low, Walter R. Boot, Arthur F. Kramer, Monica Fabiani, and Gabriele Gratton. 2012. Different slopes for different folks: Alpha and delta EEG power predict subsequent video game learning rate and improvements in cognitive control tasks. Psychophysiology 49, 12 (December 2012), 1558--1570. DOI:http://dx.doi.org/10.1111/ j.1469-8986.2012.01474.x
50. Clemens Mayer, Stefan Hanenberg, Romain Robbes, Éric Tanter, and Andreas Stefik. 2012. An empirical study of the influence of static type systems on the usability of undocumented software. SIGPLAN Not. 47, 10 (October 2012), 683--702.
51. Anneliese von Mayrhouser and A. Marie Vans. 1994. Comprehension processes during large scale maintenance. In ICSE’94: Proceedings of the 16th International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos, CA, 39--48.
52. Anneliese von Mayrhouser and A. Marie Vans. 1996. Identification of dynamic comprehension processes during large scale maintenance. IEEE Trans. Softw. Eng. 22, 6 (1996), 424--437. DOI:http://dx.doi. org/10.1109/32.508315
53. Anneliese von Mayrhouser and A. Marie Vans. 1997. Program understanding behavior during debugging of large scale software. In ESP’97: Papers Presented at the 7th Workshop on Empirical Studies of Programmers. ACM Press, New York, NY, 157--179. DOI:http://dx.doi.org/10.1145/266399.266414
54. George A. Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychol. Rev. 63, 2 (1956), 81--97.
55. Marvin Minsky. 1974. A framework for representing knowledge. (June 1974). MIT-AI Laboratory Memo 306, Massachusetts Institute of Technology Cambridge, MA.
56. Takao Nakagawa, Yasutaka Kamei, Hidetake Uwano, Akito Monden, Kenichi Matsumoto, and Daniel M. German. 2014. Quantifying programmers’ mental workload during program comprehension based on cerebral blood flow measurement: A controlled experiment. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE Companion 2014). ACM, New York, NY, 448--451. DOI:http://dx.doi.org/10.1145/2591062.2591098
57. Aljoscha C. Neubauer and Andreas Fink. 2003. Fluid intelligence and neural efficiency: effects of task complexity and sex. Personality and Individual Differences 35, 4 (2003), 811--827.
58. Fred G. Paas. 1992. Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach. J. Educ. Psychol. 84, 4 (1992), 429--434.
59. Chris Parnin. 2011. Subvocalization—toward hearing the inner thoughts of developers. In Proceedings of the 2011 IEEE 19th International Conference on Program Comprehension (ICPC’11). IEEE Computer Society, Washington, DC, 197--200. DOI:http://dx.doi.org/10.1109/ICPC.2011.49
60. Nancy Pennington. 1987a. Comprehension strategies in programming. In Empirical Studies of Programmers: Second Workshop, Gary M. Olson, Sylvia Sheppard, Elliot Soloway, and Ben Shneiderman (Eds.). Greenwood Publishing Group, Inc., Westport, CT, 100--113.
61. Nancy Pennington. 1987b. Stimulus structures and mental representations in expert comprehension of computer programs. Cogn. Psychol. 19 (1987), 295--341.
62. G. Pfurtscheller and A. Aranibar. 1977. Event-related cortical desynchronization detected by power measurements of scalp EEG. Electroencephalogr. Clin. Neurophysiol. 42, 6 (June 1977), 817--826.
63. G. Pfurtscheller, C. Neuper, and W. Mohl. 1994. Event-related desynchronization (ERD) during visual processing. Int. J. Psychophysiol. 16, 2--3 (May 1994), 147--153.
64. Vennila Ramalingam and Susan Wiedenbeck. 1997. An empirical study of novice program comprehension in the imperative and object-oriented styles. In Presented at the 7th Workshop on Empirical Studies of Programmers. ACM, 124--139.
65. W. J. Ray and H. W. Cole. 1985. EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science 228, 4700 (May 1985), 750--752.
66. Judith S. Reitman. 1976. Skilled perception in GO: Deducing memory structures from interresponse times. Cogn. Psychol. 8, 3(1976), 336--356.
67. Tonia A. Rihs, Christoph M. Michel, and Gregor Thut. 2007. Mechanisms of selective inhibition in visual spatial attention are indexed by alpha-band EEG synchronization. Eur. J. Neurosci. 25, 2 (January 2007), 603--610. DOI:http://dx.doi.org/10.1111/j.1460-9568.2007.05278.x
68. Robert S. Rist. 1986. Plans in programming: Definition, demonstration, and development. In Empirical Studies of Programmers. Ablex, Norwood, NJ, 28--47.
69. B. Schack, W. Klimesch, and P. Sauseng. 2005. Phase synchronization between theta and upper alpha oscillations in a working memory task. Int. J. Psychophysiol. 57, 2 (2005), 105--114.
70. Yu Shi, Natalie Ruiz, Ronnie Taib, Eric Choi, and Fang Chen. 2007. Galvanic skin response (GSR) as an index of cognitive load. In CHI’07 Extended Abstracts on Human Factors in Computing Systems - CHI’07. ACM Press, New York, NY, 2651. DOI:http://dx.doi.org/10.1145/1240866.1241057
71. Richard M. Shiffrin and Walter Schneider. 1977. Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. Psychol. Rev. 84, 2 (1977), 127--190.
72. Janet Siegmund, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2013. Measuring and modeling programming experience. Empirical Softw. Eng. 9, 5 (December 2013), 1299--1334. DOI:http://dx.doi.org/10.1007/s10664-013-9286-4
73. Janet Siegmund, Christian Kästner, Sven Apel, Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. 2014. Understanding understanding source code with functional magnetic resonance imaging. In ICSE 2014: Proceedings of the 36th International Conference on Software Engineering. ACM, 378--389.
74. D. P. Simon and H. A. Simon. 2004. Individual differences in solving physics problems. In Children’s Thinking: What Develops? R. S. Siegler (Ed.). Lawrence Erlbaum Associates, Inc., Hillsdale, NJ.
75. Edward E. Smith and John Jonides. 1999. Storage and executive processes in the frontal lobes. Science 283, 5408 (1999), 1657--1661.
76. M. E. Smith, L. K. McEvoy, and A. Gevins. 1999. Neurophysiological indices of strategy development and skill acquisition. Brain Res. Cogn. Brain Res. 7, 3 (January 1999), 389--404.
77. Elliot Soloway and Kate Ehrlich. 1984. Empirical studies of programming knowledge. IEEE Trans. Softw. Eng. 5 (1984), 595--609.
78. Andreas Stefik and Susanna Siebert. 2013. An empirical investigation into programming language syntax. Trans. Comput. Educ. 13, 4, Article 19 (November 2013), 40 pages.
79. A. Stipacek, R. H. Grabner, C. Neuper, A. Fink, and A. C. Neubauer. 2003. Sensitivity of human EEG alpha band desynchronization to different working memory components and increasing levels of memory load. Neurosci. Lett. 353, 3 (December 2003), 193--196.
80. John Sweller. 2005. Implications of cognitive load theory for multimedia learning. In The Cambridge Handbook of Multimedia Learning. Cambridge University Press, Cambridge, U.K., 19--30.
81. John Sweller, Paul Ayres, and Slava Kalyuga. 2011. Cognitive Load Theory. Springer, New York. http://books. google.com/books?id=sSAwbd8qOAAC
82. W. Paul Vogt. 2007. Quantitative Research Methods for Professionals. Pearson/Allyn and Bacon. http://books. google.com/books?id=NWIUAQAAMAAJ.
83. Peter D. Welch. 1967. The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15, 2 (June 1967), 70--73. DOI:http://dx.doi.org/10.1109/TAU.1967.1161901
84. P. Zhuang, C. Toro, J. Grafman, P. Manganotti, L. Leocani, and M. Hallett. 1997. Event-related desynchronization (ERD) in the alpha frequency during development of implicit and explicit learning. Electroencephalogr. Clin. Neurophysiol. 102, 4 (April 1997), 374--381.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI