AR-C&P: A Tangible Programming for Children Based Augmented Reality

Augmented Reality (AR) technology enables continuous visual feedback and varied digital scenes on the real world. This paper presents AR-C&P, an AR-based tangible programming tool for young children. It allows children to create AR maze map with low-cost materials and plan the route by constructing tangible program sequences. Meanwhile, a variety of real-time visual feedback from the screen will be superimposed on the real scenes of the whole process. AR-C&P is intuitive and intriguing which could enrich children’s creating experience, develops their cognitive developmental skills and helps children learn fundamental programming concepts. We conducted two experiments. The first experiment reveals the relationship between task completion and cognitive developmental skills. The second experiment reveals the effect of AR on learning programming.


INTRODUCTION
A number of countries have brought programming courses back to their elementary education.Programming learning has been proved to have positive effects on children in many areas, such as developing computational thinking, problem-solving and design strategies [28].In particular, computational thinking has been described by Wing as a fundamental skill for everyone, not just for computer scientists [49].In addition, using interactive programming tools in computer teaching can increase students' enthusiasm and participation [33].Therefore, developing an appropriate tool of interactive programming for children becomes necessary.
Most of the existing programming languages are text-based, which is difficult for children to comprehend without sufficient context.But Graphical User Interface (GUI) and Tangible User Interface (TUI) offer flexible interfaces, helping children to handle the code more easily.GUI-based kits usually run in the virtual applications in PC and/or mobile-device without physical components.For example, ScratchJR [7] provides a programming learning platform for children to create and run their programs.Research has shown that TUI is more attractive and conducive to collaborative interaction than GUI, and encourage children to explore and learn better [15].In particular, tangible languages have the potential to simplify the understanding of complex syntax, enhance the learning effect [12], and improve children's involvement in programming [14].As a result, there have been extensive researches on tangible programming languages [50], such as Project Bloks developed by Google [1], roBlock developed by CMU [38] and TLogic developed by University of Chinese Academy of Sciences [5] which aim at ameliorating the user experience in programming learning.
Augmented Reality (AR) is an interactive technology emerging in recent years.It enhances users' programming immersion and learning experience through rich 3D virtual scenes and visual-based real-time feedback [13].Relevant study has shown that the use of interactive AR in education can help attract students' interest in learning and thus improve study efficiency [47][24] [19].Existing research focuses on using AR to bring virtual content into the real world and create exciting and unique experiences for young users [8][21] [18] [40].Some researches have realized the combination of AR technology and physical programming, but few people use AR technology to provide continuous, real-time and intuitive feedback in programming process.
In this paper, we present a tangible creation and programming tool AR-C&P (Figure 1).This system supports AR-based 3D scenes creation and sorts of real-time feedback.For example, if the rules of programming grammar are violated, text and image tips will be played, thus reminding the user to address the program error.The system combines virtual system with low-cost physical materials.
And it allows children to create and program in a physical form and executes the generated code in virtual environments with a coherent view of the operating space.This system lowers down the entry barriers of AR creation and programming for children.And it makes it possible for children to learn fundamental programming concepts while cultivating creativity, computational thinking and spatial cognitive skills.To verify the effect of AR on learning programming and the relationship between the task completion and children's cognitive developmental skills, we conduct two experiments.

RELATED WORK 2.1 Programming Tool
Tangible Programming gets rid of the pure computer environment Programming, enables children to participate in programming activities in the real physical space, and helps them understand programming knowledge [25].Several excellent tangible creations and programming tools have been developed for children.Electronic Blocks [51], KIBO [43] ,TanProRobot 2.0 [48] and KareNao [9], are robotics toolkit designed for children with tangible I/O system and no digital screen.On the market, there are also some of excellent tangible programming tools, such as Matatalab [? ], Ozobot [? ] and Project Blocks [1] that allow children to control a target object to move on a map.In the above programming tools, children can run the program, verify and correct the program with the observation of electronic devices (such as machine components), during the process of these tasks.However, these participants could merely rely on the physical interface itself (such as adding a voice sensor or LED as feedback), which increases the cost of programming tools and reduces the visual form of feedback.
Combining tangible programming interface with screen display interface can compensate for the lack of visual feedback.StarLoop [26], Strawbies [16], CoProStory [6], Loopo [23] and TLogic [5] combine tangible language and virtual display interface.Users connect wood blocks or electronic slices to form program control flow, and get programming feedback through screen display.Among them, TLogic uses visual and audio feedback displayed after a successful landmark acquisition.The evaluation of it demonstrated that tangible programming blocks combined with virtual display interface can reduce children's cognitive load.StarLoop, Strawbies and Loopo do not provide clear real-time feedback.CoProStory uses real-time feedback of text and confirmed that real-time feedback is helpful for children to complete programming tasks.These tangible languages bring the programming process closer to the real world, making programming learning easier to start for novices.And the feedback from electronic screen reduces the cost of the physical parts.But it still not convenient enough, because it requires children to get feedback through the mapping of physical languages and the screen display to check the running results of the programs.

AR Technology
The AR technology with mature development offers a brand-new way of interaction and blending between real-world and virtual environment.There already have some attempts to apply AR technology in programming education.Code Bits [11], Code Notes [35] and ARcadia [22] are tangible programming toolkits, on which programs are created by paper-based materials and programming effect can be observed through the AR-based screen.MagiPlay [41] is a children's programming tool for smart home environment integrating augmented reality.StoryMakAR [10] has combined physical prototyping and Augmented Reality, which is a new AR-IoT system for children making and storytelling.Draw2Code [17] allows the user to design Sprite, Action, and Event blocks on cards with special markings.Using action blocks and event blocks to form a programming sequence, the user can control the target character enhanced by the sprite blocks.Kao et al. [20] adopted a high interactive AR operating mode with puzzle cards to further enhance the interactivity between the learner and the learning content, which could encourage the spirit of "learning by doing" to engage learners.These tangible tools combining AR facilitate novices' programming learning and improve their enjoyment.However, most of them mentioned above assign the augmented reality effect as the results of program running, failing to provide real-time interactive feedback or physical-based tips during coding period.
We implement AR-C&P combining between AR technology and tangible programming interface, which can superimpose real-time AR feedback directly on the real object interface throughout the programming process.We hope to introduce programming concepts to the children with more intuitive and instructive feedback (feedback is described in detail in Section 3.5), as well as enrich their programming experiences with 3D AR scenes.

Evaluation Methods
Pearson correlation analysis is a common method of analysis used in statistics.It can effectively analyze the relationship between two random variables [3].For this reason, Amanda Strawhacker et al. [42] put this approach into the evaluation of children's programming, aiming to determine the correlation between student performance and their abilities in terms of block identification.
Lautaro Cabrera et al [4], Theodosios Sapounidis et al [37], Anja Thieme et al [44], Xiaozhou Deng et al [6] use a comparative research method in the evaluation of programming tools for children.In this method, children are divided into two groups and asked to complete a set of tasks.The researchers record video of the process, the time it take to complete the task, the number of revisions, and the children's real-time reactions during the task.Through the comparison and analysis of these data, the evaluation results of programming tools for children are obtained.Among them, Theodosios Sapounidis et al. [37] and Xiaozhou Deng et al. [6] also combined another evaluation method in their study.This method lists several specific questions according to the tasks assigned in the experiment.Each question is scored by a 5-level Likert scale.
We refer to the evaluation methods of above research.In our first experiment, we use the Pearson correlation test to analyse the relationship between children's cognitive developmental skills and their performance in completing the children's programming tasks.In our second experiment, the children are divided into two groups (one of which is using an AR-C&P that doesn't have a real-time feedback function).Both groups are asked to complete an identical task.After that, children need fill out a questionnaire that some questions are scored by a 5-level Likert scale.According to the questionnaire, we further explore the influence of AR on learning programming.

Creating Board
The Creating Board consists of an A4-sized magnetic board with grid square and a series of 2.5cm*2.5cmpaper tokens of diminutive magnetic sheet on back side.Each of the paper token is imprinted with a circular barcode called TopCode [? ], to determine the position, orientation, and type of each token.In the meantime, every token is classified by color and stripe.In all, there are three types of blocks (Figure 4).As shown in (Figure 3), six kinds of tokens can be divided into three categories: Feasible Path Tokens (grey or blue): to construct the feasible maze path.Grey tokens are the normal path and blue ones are the prominent path.
Start/End Point Tokens (yellow): to mark the initial position of virtual characters and the destination of the maze map.
Decoration Tokens (pink or green): these tokens allow children to decorate their maze map.

Location Map
Location Map is a self-made picture card, designed to locate the position of the virtual map.AR-C&P game allows the detection and tracking of Location Map based on existing 3D models.As the position and angle of the Location Map change, the attached virtual map will be modified accordingly.Location Map also guarantees the coherence of the creating stage and subsequent stages (programming and running stages).In the creating stage, children place paper tokens on the creating board to complete customized maze map.When the save button is clicked, the map in the creating board will be scale-reduced and attached to the Location Map (Figure 5).In the programming stage, the program is constructed based on the Location Map.Then the virtual maze traces the Location Map shown on the screen in the running stage.It allows users to examine whether the characters act and move as they programmed in the programming stage.

Programming Blocks
Programming Blocks are made by 3cm wooden brick cube with TopCode texture.Magnets are arranged in the Programming Blocks to help children keep the correct building direction.Moreover, these magnets optimize the Programming Blocks' rotation.In order to make the program metaphor much clearer, we added pictures and text to indicate the functionality of the block.There are four faces in each block, which can be used to express four distinct semantics, and to reduce the number of Programming Blocks, thus lowering cost of the system.According to the semantics and functions, we divided the blocks into three types (Figure 4): Start/End Blocks:to mark the beginning or end of theprogram.Programmer should place it at the beginning or the end of the sequence, which will wrap the content of program in the middle part.Motion Blocks: to control the movement of the game character.Motion Blocks are divided into Jump Blocks, Turn Blocks and Forward Blocks.For example, Jump Blocks include Jump Up/Down.Turn Blocks include Turn Right/Left, and Forward Blocks contain forward one to four steps.Children can change the parameters of the action by rotating the programming Block.Loop Blocks: to execute a part of program repeatedly, which have different times (from one to three) of repetitions and the end tag of loop on each face.Loop Blocks can be used to optimize the program, and teach children how to reuse a program or code segment.

AR-C&P Mobile Application
This system is implemented in Unity3D and Qualcomm Vuforia [? ] platforms using a designated image (Location Map) as a marker and displaying a 3D scene on the top.There are three stages in AR-C&P, including creating stage, programming stage and running stage.
In the creating stage, it is required to place the mobile device above the creating board and make sure the board is in the camera vision scope.System recognizes the position of every paper token placed on the grid of magnetic white board, then generates the map derive from those tokens and projects them on the cognition picture next to it instantly.Meanwhile, the content of the Location Map on the left side will be constructed concurrently.
In the programming stage, the system will provide users with a continuous real-time visual feedback that is based on AR.In our system, there are three types of feedback.First, when putting a new programming block, an arrow appears to indicate the position and direction of current program.Second, each programming block placed in the field of view has an augmented reality shape superimposed on the top.In this way, when the placed blocks are not recognized by the system due to lighting or the edge of vision, programming sequence can be adjusted.Third, if the character is not on the bricks of maze or the rules of programming grammar are violated, text and image tips will be played, thus reminding the user to address the program error (Figure 6).
In the running stage, the virtual maze traces the Location Map shown on the screen.It allows users to observe whether the character acts and moves as they coded in the programming stage.Meanwhile, the image tips show the location of the current programming block when the program is executed.Audio and image tips will also notify children which programming block is running currently (Figure 7).In addition, there are four types of feedback to indicate the status of Programming Blocks (Figure 8).These feedback would help children to understand program execution procedure flows.

Others
With relatively small hands, children may feel uncomfortable to hold the device during the game.Considering this, a bracket is embedded to support the mobile device, freeing their both hands and providing them with an immersive AR programming experience.

USER STUDY
To verify our design targets, we conducted two experiments.The first experiment is aimed to examine the relationship between the task completion and children's cognitive developmental skills, such as direction perception, fine motor skills and spatial cognitive.Task completion included the time of programming and the number of modifications.The second experiment is aimed to verify the effect of AR on learning programming.

The First Experiment
In this experiment, we used an earlier version of AR-C&P, where the motion path of the character is two-dimensional (Figure 6).16 children aged between 5 and 8 (10 boys and 6 girls, mean age of 6.35) took part in the first experiment.

Task Design.
One practice task and two formal tasks were designed.The purpose of practice task is to familiarize children with AR-C&P system.In every formal task, children need to find a feasible path correctly   and construct the solution sequence with Programming Blocks to control the virtual characters to complete the maze.In the first task, children need to complete the maze task through a series of step-by-step instructions (Figure 9).Then in the second task, the children need to use Loop Block to optimize the code in the first task (Figure 10).To finish this task, participants need to detect the potential pattern of circular structure and create a suitable loop structure to solve the corresponding task.

Process.
In the first experiment, researchers adjust the height of the pad according to each children's height in advance and introduce them to the whole system.Firstly, children would have a pre-questionnaire.Secondly, a video presents the whole system and introduces the use of all components to children.Then, we asked the children to complete practice task, during which they were encouraged to ask all kinds of relevant questions, which is aimed to ensure that the child has mastered the system.Next, the children would start formal tasks.

Results.
In the following, we analyze the relationship between the task completion and children's cognitive developmental skills based on the questionnaire and children's performance, which used the Pearson correlation test to verify.1. Task Completion: In the experiment, all the children successfully completed the two tasks.As shown in Figure 12 and Figure 13, the average number of revisions for the first task is 4.1, while the average completion time is 134.5s.The average number of revisions for the second task is 3.6, while the average completion time is 81.11s.2. Direction Perception: For the first task, there is significant inverse correlation between direction perception test and the time of programming (r=-0.34,p<0.01) as well as the number of modification (r=-0.42,p<0.01).As mentioned earlier, task 1 does not need to use loop blocks, which means that the sequence of the programming is much longer.Therefore, task 1 needs a better direction perception to determine the position and direction of the game character.For the second task, there was no significant correlation between this test and children's task completion.Because task 2 requires more logic and abstract skills to modify the existing sequence instead of forming a new programming path.3. Fine Motor Skills: For the first task, there is no significant correlation between Fine Motor Skills test and the time of programming, but there is significant positive correlation between this test and the number of modifications (r=0.30,p<0.01).For the second task, there was no significant correlation between this test and the time of programming, but there is significant positive correlation between it and the number of modifications (r=0.46,p<0.01).By analyzing the video, we found that children who got higher score of Fine Motor Skills skill change blocks more frequently and prefer to try different placement positions to detect effects.This process requires hand-eye coordination and the ability to manipulate objects, and it also leads to an increase in the number of modifications.4. Block Construction: For the first task, there is significant inverse correlation between Block Construction test and the time of programming (r=-0.34,p<0.05) as well as the number of modification (r=-0.23,p<0.01).For the second task, there is significant inverse correlation between this test and the time of programming (r=-0.40,p<0.05) as well as the number of modification (r=-0.33,p<0.01).These results indicate that task 2 requires better three-dimensional structure construction capabilities.In addition, to create a loop structure, children's spatial reasoning ability is needed.
The results above suggest that children need to use different cognitive developmental skills to interact with tangible programming in AR environment, but not all higher score is correlated with better task performance.Compared to task 1, task 2 requires more logic and abstract skills to modify the existing sequence instead of forming a new programming path.In addition, task 2 requires better three-dimensional structure construction capabilities and spatial reasoning ability.

The Second Experiment
In order to verify that real-time feedback is helpful for children's programming, we designed the second experiment to explore.21 children with grade 1-3 and age range of 6 to 9 took part in the second experiment.In the comparative study, we divided 21 participants into two groups (Table 1).In G1 (3 girls and 7 boys, mean age of 7.20 and SD=1.03), children created programs with provided physical Programming Blocks and Location Map (Figure 14).After program creation phase, children scanned these blocks by the camera compiler of the Android app.Upon scanning, the app recognized the blocks and then sequenced them into executive actions that could finish the AR maze task (Figure 15).That means the version of AR-C&P used in G1 have no instant AR feedback when children programming.It was only used to run the program.In G2 (4 girls and 7 boys, mean age of 7.36 and SD=1.12), children programmed with identical Location Map and relied on the feedback of AR-C&P to debug the program.

Task Design.
In the second experiment, children need to complete the same task as in the first experiment, including one practice task and two formal tasks.But the difference is that we designed several feasible paths from initial position to endpoint and children need to plan a three-dimensional maze path (Figure 15), in which they would use Jump Blocks.The process of the second experiment was similar to that of the first experiment, but we collected children's demographic data and prior relevant experience in pre-questionnaire rather than cognitive developmental skills.After children completed tasks, children in G2 were asked to finish a post-questionnaire which includes three quantitative questions and three subjective questions (Table 2),  while children in G1 were only asked to finish a post-questionnaire which includes three quantitative questions.The three quantitative questions aim at finding out whether the children have acquired programming knowledge.The first quantitative question gives a character's initial state and the operation command of programming blocks, and lets children find the character's state after block's programming.To accomplish this, the child needs to be able to correctly understand the function of each programming block.As for the second question, a non-loop sequence with three errors ("rotate error", "steps error", "syntax error (without end block)") was given to children and we asked them to point out all these mistakes.The third question involves loop concept.We offer a complex sequence containing three repeat structures and ask kids to optimize the code segment with loop.
Three subjective questions are used to evaluate the helpfulness of AR-based feedback.As for subjective questionnaire, we used the modified rating scale based on the Smiley-o-Meter, with which children could answer the questions by pointing at a specific smiley [36] [34].The questionnaire uses the Likert-type scale with score range of 1 to 5, where one denotes the minimum score and five denotes the maximum score.Finally, a post-experiment interview could be conducted for children to collect how they felt and what they liked or disliked.Q1 Was the text feedback helpful to accomplish the tasks?Q2 Was the image tips helpful to accomplish the tasks?Q3 Was the image feedback helpful to accomplish the tasks?

Results.
We get some results, shown in following, reveal effect of AR on learning programming.As for quantitative questions, the first quantitative question tests comprehension about the function of programming blocks.The average accuracy rate of children in G1 was 90%(1 child failed) and that of children in G2 was 100%, indicating that children in G2 might have a better comprehension about the function of programming blocks.The second quantitative question is to find out whether the children master the programming sequence concept.The children in G1 got accuracy rate of 80%, and the accuracy rate of G2 group was 90.1%.The most frequent mistakes they made are the corner error.The third quantitative question involves loop concept.In this part, Only 40% of the children completed the revision question successfully in G1.Instead, there are 81.2% of the children in G2 could complete it well.
As for subjective questionnaire, we provide three types of AR feedback to help children accomplish their tasks only for G2.The results of subjective questionnaire of G2 are shown in (Figure 16).Due to the fact that some children are impaired in reading on the screen, the results of Q1 got an average of 2.58 (SD=0.88).A child giving low rating (P13, 9 years old, boy) on this question thought that the text feedback was useless since he is not capable to read the feedback words on the screen.But this kid gave relative higher score to the image tips (Q2) which help point out the correctness of program and remind the error block.Compared to Q1, the results of Q2 received a higher score with an average of 4.44 (SD=0.81).The image tips may enhance the children's confidence to keep learning programming knowledge in the programming stage.The Q3 primarily focuses on the image feedback, which indicates where the character would move towards with arrows.In our observation, all children could debug their programs by the image feedback.And this question received an average of 4.7 (SD=0.48),which demonstrates that the image feedback is most conducive and image tips are more significant compared with text feedback.
In general, children can get better effect of learning programming when using AR-C&P with AR support which can give children realtime feedback.

DISCUSSION
We present a tangible programming tool based on AR technology, which can help children create AR maze game and learn the programming concepts.This programming tool has multiple real-time feedback and continuous AR experience from game creation to program running.Our work has some key advantages that it has an intuitive feedback on the process of programming which could enrich children's creating experience and lower down the entry barriers of programming for children.The system needs low-cost materials, but enable children to get an improvement of programming knowledge.

Children's behavior
When we inquired children about their most fulfilling part during the experiment, almost all the children (N=9 in G1 and N=11 in G2) referred to the AR technology: "The 3D map looks so real that I can observe it from every perspective" (P9, 8years old, girl).In addition, three of the children in G1 and two of the children in G2 mentioned the functionality of the programming blocks: "forward blocks are great because the 4 step blocks could make the character faster towards the destination" (P7, 9 years old, boy) and "I think loop blocks are very interesting because they could reduce the length of sequence" (P21, 8 years old, boy).According to the analysis of the videotape, children were more engaged when the system entered running stage.Children will become so excited when they watch their programmed character running correctly and reaching the endpoint eventually.
During the observations of children's interactions in the game, we found that the most common mistake they made lied in the Turn Block.An interesting strategy made by three children (P1, 8, boy; P4, 7, girl; P9, 8, girl) in G1 is keeping rotating location map to determine the next action of character.For example, when the character's current orientation was the front, P4 would rotate the location map counterclockwise 90 degrees after using the Turn Right Block.Under this strategy, she could ensure that the character would always facing forward.These three children spent less time to complete the task, compared with others in G1.

Limitation
Using subjective questionnaire may result in the case that children inclined to choose what favors the tester, which biased experiment stats.Brown et al. [2] recommended several changes in user study to help moderate theses effects, which includes diversifying methods and emphasizing the unique characteristics of participant.In this work, we maintained these recommendations and applied the comparison experiment of two groups to obtain meaningful and representative information from the collected data.
Moreover, during the experiment, children may accidentally bump into the device-supporting bracket, which cost some time for device adjustment.Most of these were minor issues and the researches nearby can quickly adjust the equipment and fix it.

CONCLUSION AND FUTURE WORK
In this paper, we design and implement a children's tangible programming system based on AR technology -AR-C&P.In this system, children can create 3D maze game mapping between virtual and real world.They program with tangible language, and the running results will be shown in a virtual world.Our results from user study demonstrate that AR-C&P is an edutainment tool with great attraction which could provide a better programming experience for children.Meanwhile, it can help children learn programming knowledge and better understand programming concepts.
In the future, we will continue to improve our system.More programming concepts, such as functions and branches, will be added.In addition, the physical arrangement and tracking effect of location maps needs further improvement, and more interaction will be extended in the running stage with AR technology.
Furthermore, people might have stereotype that regard girls not as good as boys in STEM.However, it has been noted that they deserve to be treated identical in programming learning [15] [24].Since the number of people in this experiment is relatively small, we didn't do the analysis of gender difference.In future work, we will consider whether children have gender-related specific behaviors in the physical programming environment of AR.
Finally, tangible programming environment was proved to be better for cooperation, which deserves further investigation.When applying AR to tangible programming environment, the size of the device's screen and the camera's reach may cause a shortage of space for children to collaborate, but the physical interface still can realize leading people to socially-motivated interaction.ShareTable [46] and RobotIST [39] used projection technology to complete the project, which inspired us to apply projection to expand the coverage of the device screen and device camera mentioned above.So in the following research, cooperation elements will be considered in the system.

Figure 1 .
Figure 1.A child is playing with AR-C&P.
AR-C&P is a Child-oriented tool for tangible creation and programming based on AR technology.AR-C&P mobile application will provide fully AR support.Physical interface contains Creating Board which includes types of paper tokens, three kinds of Programming Blocks and Location Map (Figure2).The primary design targets of AR-C&P are: 1. Cognitive developmental ability: cognitive development refers to the process in which individuals' cognition of things and their thinking mode and ability performance in the face of problem situations change with age.It includes direction perception, fine motor skills and spatial cognitive.2. Effect of learning programming: children could learn basic programming knowledge; the effect of different feedback on process of programming.

Figure 3 .
Figure 3. Six kinds of paper tokens.

Figure 4 .
Figure 4. Three types of programming blocks.

Figure 5 .
Figure 5.Using Location Map in the creating stage.

Figure 8 .
Figure 8. a) The block is not running.b) The block has completed run.c) The block is currently running.d) The block is wrong.

Figure 11 .
Figure 11.Examples of cognitive developmental skills tests.

Table 1 :
Group sizes and demographics in second experiment

Table 2 :
Subjective questions in second experiment.