A Self-Regulated Learning Framework using Generative AI and its Application in CS Educational Intervention Design

Self-regulation refers to the ability to plan, monitor, control and reflect on one's problem-solving process. Prior research has shown that self-regulated learning (SRL) strategies help improve novice performance in solving programming problems. However, with the advent of LLM tools like ChatGPT, novices can generate fairly accurate code by just providing the problem prompt, and hence may forego applying essential self-regulation strategies such as planning and reflection to solve the problem. In this position paper, we discuss challenges and opportunities that generative AI technologies pose for novices' self-regulation strategies in the context of programming problem solving. We believe that the key challenge facing educators is that such technologies may hamper novices' ability to regulate their programming problem solving process. On the other hand, these technologies also open up the possibility to design new interventions that promote better SRL strategies in learners. We draw on generic and domain-specific self-regulated learning theories as the basis of our work, and propose an SRL framework that incorporates use of generative AI tools in programming problem solving. We illustrate how the proposed framework guides exploration of the design space of interventions that integrate generative AI in CS education.


INTRODUCTION
As students and instructors learnt about LLM tools, many possible uses of such tools and worries about their use in CS education surfaced in the community.In an early study, Lau and Guo [17] collected a snapshot of tool capabilties and instructor responses and proposed research opportunities for computing education research, such as improved scaffolding, building educator tailored tools, scalable assessments, feedback, and personalized instruction.Another early study [11] identified the possibilities of generating exercises, solution variants and refactoring problems.These ideas for future work are being addressed in papers on exercise generation, code explanations, improvements of error messages [18,19,36], and so on.Such investigations of the possible uses of LLMs have emerged organically, driven by community perceptions of possibilities and worries.
In this position paper, we propose a more principled way to think about the ways LLMs can be used in computing education, using student learning as the central driving point.We believe that the primary issue with LLMs is that by directly providing solutions to problems on demand, they hinder the development of independent problem solving ability among students.Thus students may never develop the self-regulation that is needed to solve complex problems in practice [22].To address this issue, we propose a new framework for self-regulated learning (SRL) that builds on existing frameworks, and adds new elements that take into account the capabilities of LLM tools.
A useful model of learning with LLMs is to imagine it as pair thinking, something more than just pair programming, that describes the conversation between the student and a seemingly omniscient expert.Our framework lays out a roadmap for methodical investigation of educational interventions that lend structure to such 'pair thinking' in a way that centers on student learning.
The paper is organized as follows.In Sec. 2, we examine related work in self-regulation, use of LLM tools in computing education, and prior research about calculator use in education.Sec. 3 outlines the central position of this paper, which forms the basis of subsequent sections.In Sec. 4, we provide details of the proposed SRL framework for programming problem solving using generative AI tools, drawing on theories described in Sec. 2. In Sec. 5, we show how our proposed framework serves as the basis for designing learning interventions for fostering self-regulation in students.Sec.6 concludes by highlighting our contributions.

RELATED WORK
In this section, we discuss prior work used in formulating our position, including the literature on self-regulated learning, on uses of LLMs in education, and on the somewhat analogous historical issue of using calculators in educational contexts.
Metacognition and Self-Regulation: Metacognition refers to the ability to think about and reflect upon one's thought process [30].Self-regulation typically refers to learners' ability to plan, monitor and control their cognitive processes while problem solving [22].In computing education, Bergin et al. [4] highlight the importance of self-regulation in programming.Loksa and Ko [21] demonstrate that explicit instruction in making the steps of problem solving manifest improves performance.Prather et al. [32] consider the effect of metacognitive scaffolding for interpreting a problem and report positive effects.Falkner et al. [10] show how the appropriate use of general and CS specific specific SRL strategies grows as a student matures from first year to final year.The survey by Loksa et al. [22] presents an overview of theories of self regulation in the educational psychology literature and describes two CS domain-specific variants.
Unlike these works that predate conversational AI, we are interested in the effect of LLMs on problem solving (PS) and selfregulation (SR).We use the pioneering work of Polya [31], the SRL frameworks of Flavell [12], Pintrich [29] and Zimmerman [41], and draw upon Loksa and Ko [21] who add CS specific elements of PS and SR.Our framework interprets elements mentioned in these works in a way that caters to the ubiquitous presence of LLMs.
Using LLM tools in CSEd: Since LLM tools work very effectively with languages, they make possible new kinds of educational interventions [11].Sarsa et al. generate exercises using LLMs [36], finding that generated exercises are sensible and novel, and the problem statements are properly contextualized.They show that LLMs also generate suitable test cases.LLMs generate code explanations [24] that are easy to understand [18] and helpful [23].Other uses include improvement of error messages [19,35] that are sometimes better than the originals.
Perhaps the most interesting new aspect of LLM based tools is that they allow conversation.The idea of 'prompt engineering', the iterative redesign of problems, is an LLM specific skill explored in the design of constraining tools like Promptly [7].That overreliance on LLM based tools is likely to affect SRL is discussed in [8].
However, such research efforts have explored the use of LLMs in education in an unstructured way.In this paper, we present a more methodical approach to develop interventions that address the key predicament, that problem solving at the CS1 level is now trivial, and needs to be rethought.We should not be patching up difficulties as we notice them, but instead look for an organizing principle; this motivated us to design our SRL framework.
Automation and Education: Something of a parallel exists between LLMs, which may be considered "Textual Calculators", and earlier tools like Computer Algebra systems.The use of graphical calculators has been controversial [26], but a meta analysis [9] confirms that students find them motivating, and there is not much effect on their mathematical skills up to high school.However, Calculus grades in university are better if calculator use is restricted until concepts are mastered [25].
For technologies like CAS [14] and Microworlds [15] the educational promise has not been borne out.Such tools have induced new misconceptions [13,14].The calculator's journey from a general tool into a tool suitable for students [13] has been complex, and its educational effects differ from student to student.
CSEd research with LLMs has not yet made the connection to these earlier experiences.We may discover that as with calculators, LLMs should be restricted until concepts are mastered, as well as more complex situations such as misconceptions and differential effects induced by LLMs.The methodical approach to intervention design proposed here should make it easier to tackle such cases.

POSITION
Our position is the following: Generative AI technologies fully solve student-level problems just from the problem statements, so they are likely to hamper students' ability to regulate their programming problem solving process.Therefore, educators need a principled approach to develop educational interventions that use these technologies in ways that foster self-regulated learning.
LLMs like Codex and ChatGPT are able to provide correct code for most introductory programming problems [6,11].Prior to the advent of LLMs, instructors could use such problems to train students in applying key programming concepts that were taught in class.The aim was to provide students with opportunities to spend sufficient time and effort grappling with problems.It was in this process of working out the details, that students learn to plan, monitor, and reflect on their problem solving process.
However, given LLMs, instructors can no longer rely on students spending sufficient time in this self regulation process [8].Students will be tempted to directly elicit answers to programming problems from AI tools, vitiating their learning.This calls for instructors to spend sufficient time in explicitly focusing on self-regulation strategies for problem solving.
While LLMs may hamper self regulation, they also open up the possibility to design new interventions that foster self-regulation skills.In introductory programming courses, students often struggle with syntax while starting to learn a programming language [37], and it usually takes time for students to build proficiency in mastering syntax [2].Since LLMs automatically generate code, they provide the affordance to clearly separate the specification of the problem from the actual implementation during problem solving.Hence, instructors could emphasize steps other than codingproblem understanding and solution evaluation.
In the next section, we outline how conversational AI tools alter problem understanding, solution evaluation, and self-regulation strategies employed during programming problem solving.

AN SRL FRAMEWORK FOR PROGRAMMING PROBLEM SOLVING USING GENERATIVE AI
In this section, we outline an SRL framework for programming problem solving in the presence of LLMs.We view learning with LLMs as pair thinking, where an inexperienced student talks with an 'experienced but erratic' entity that aids in understanding, implementing, and evaluating programs, but does not guarantee correctness.Using LLMs thus affects problem solving (PS) stages, as well as student self regulation (SR) strategies.The effect of LLM induced changes is captured in our proposed framework, summarized in Fig. 1.
To motivate our proposed structure, let us start with a brief discussion of the PS and SRL frameworks that we draw from.Polya [31] pioneered the study of problem solving in mathematics with his four-phase mathematical problem solving process: understand, plan, execute, and look back, with elaborations such as "find an easier related problem".He notes the role of affect, calling for "an education of the will".Loksa and Ko [21] provide a CS specific six-activity PS framework: reinterpret, search for analogues, search for solutions, evaluate solution, implement, and evaluate implementation.They also propose five SR strategies that support PS: planning, process and comprehension monitoring, reflection and self-explanation.The interplay between PS and SR activities is made explicit in Pintrich [30].LLMs affect how we practice these activities.
The use of LLMs changes problem understanding and evaluation stages, and makes new demands on students' ability to maintain motivation while confronting the overconfidence of LLMs.Therefore, we combine PS activities into three broad phases: Problem Understanding, Use of Generative AI, and Solution Evaluation (top portion of Fig. 1), and include regulation strategies related to Motivation and Affect in our framework (bottom portion of Fig. 1).
The phases, their sub-categories, and constituent elements of PS are described in Sec.4.1 and Sec.4.2.The self-regulation strategies are described in detail in Sec.4.3.The sub-categories and subcategory elements of the problem solving phases and self-regulation strategies and illustrative thought examples are shown in Fig. 2.

Problem Understanding
A clear understanding of the problem is essential, since an incorrect understanding renders invalid all subsequent steps toward a solution [32].Several theories of metacognition and self-regulation highlight the importance of analysing the task, as well as understanding the surrounding context of the problem to be solved [12,29,41].According to Flavell, the problem task is perceived differently by a variety of learners, based on several parameters such as familiarity/unfamiliarity, organization (well or poorly organized), redundancy and ambiguity in the task [12].Zimmerman highlighted the importance of setting appropriate goals and strategic planning as part of the task analysis phase in the self-regulation process [41].In addition, Pintrich stressed the importance of prior content knowledge and metacognitive knowledge activation during the initial phases of problem solving [29].
What are learners doing during the problem understanding phase?They plan and reflect on the problem context and the specification, and iteratively extract key elements for the problem.These form the sub-categories of the problem understanding phase.We describe each sub-category in detail below.A summary of these sub-categories and constituent elements is shown in the left portion of Fig. 2.
Problem Context: In the proposed SRL framework, information about the programming problem task and the context of the problem must be clearly presented to the LLM.Learners need to identify the context in which the problem is situated -such as programming concepts involved (e.g.time complexity), programming constructs required to solve the problem (e.g.usage of conditionals and loops), and domain-specific knowledge (e.g.knowledge of hash tables).
Problem Specification: Using knowledge of the problem context, learners should create a mental model of the specification which needs to be provided to the LLM.The process of coming up with a clear specification can involve rephrasing parts or all of the problem statement, questioning details about the requirements (e.g. is this part of the requirement valid?), decomposing the problem into smaller parts (e.g.solving this problem requires creating 3 classes), and thinking about the corresponding output for a certain input (e.g. for the input value 1, the output should be 2).
Iterative Conversation: The student and the LLM together converge towards an understanding of the problem that results in a correct specification and then implementation.If the understanding of the LLM is incorrect, the student must rephrase the problem or solve a part of it; we are thus asking an inexperienced student to judge and guide an 'experienced' entity!Contradictory as it is, we must address the situation using interventions discussed in Sec. 5.
The rephrasing and partitioning of problem statements in a suitable way is the skill that has become known as prompt engineering [20].Prompts are defined at different levels of abstraction, based on the characteristics of the problem/sub-problem which needs to be solved.For example, learners might design a prompt at a conceptual level (e.g. a problem involving a variation of an algorithm concept), or at an implementation level (e.g.typical CS1 course problems).Implementation level prompts could be specific to the problem, or may be a generic template which serves as a starter code for learners [2].Such differences in abstractions and specificity of prompts requires learners to decompose a problem or re-purpose it, moving back and forth between understanding the problem and evaluating the solution generated by the LLM.

Solution Evaluation
Evaluation and reflection on the final as well as intermediate solutions is an important aspect of problem solving.Pintrich highlighted the evaluation of the task and context as an important aspect of self-regulated learning [29].Zimmerman notes the role of selfevaluation in calibrating one's effort on a task with a standard or goal [41].In programming, this amounts to evaluating whether the program output satisfies the problem specification.
The solution generated by the AI need not be ideal or appropriate in the first attempt.This makes evaluating the generated solutions an important activity in the problem solving process.Learners are required to monitor and reflect on the solution and strategise subsequent problem solving actions.We describe each of these below (summarized in the right portion of Fig. 2).
• Solution Explanation: Ask the LLM to explain its own solution, and also comment on specific parts of the solution • Evaluation using test cases: Learners evaluate the code by using their own test cases or even ones generated by LLMs.• Problem decomposition: In many cases, solutions generated by LLMs may be too confusing.Students must then learn to backtrack, breaking down the original problem and asking for solutions to sub-problems.In the worst case, they should forego using LLMs and work on their own.• Inspect and Adapt: Understanding code character by character, deleting extraneous code, altering incorrect code are techniques [32] that learners have used to cope with generated code.

Self-Regulation Strategies
Here we discuss how self-regulation strategies discussed at the beginning of Sec. 4 are affected by LLMs.
Planning: The student half of the pair must reflect on the AI's responses.Initial planning involves strategies that help in understanding the problem (e.g. is it possible to break this problem down into parts?).The planner needs to decide the parts that are handed to the LLM, and those that could be solved without its help (e.g.I anticipate that this part given directly to the LLM will generate fairly accurate code).Interactions with the LLM can trigger additional planning steps or deviations from the current plan (e.g.The code generated for this part of the problem is not entirely right.I will have to tweak it a bit).
Process Monitoring: This involves monitoring one's progress in the conversation, asking which problem solving phase one is in right now, which part is complete and which remains.One must guard against the conversation going off into irrelevant and eventually frustrating non-solutions.
Comprehension Monitoring: This involves monitoring one's own as well as the LLMs understanding of concepts required to solve the problem.LLMs have been trained on massive data sets, and hence they might generate artifacts based on concepts which learners might be unaware of (e.g. this part of the generated code involves constructs I've not seen before).
Reflection on Cognition of Self and AI:.Here we need not only self-reflection on one's own limitations, but also knowledge of the limitations of the AI [2], as well as the added responsibility that one must not anthropomorphize the AI into thinking it is aware or has agency [33].
Self Explanation: With LLMs in play, we need to explain our own reasoning and that of the AI.Prior work has shown that generative AI models can "hallucinate" and generate incorrect output in an authoritative and confident manner [1], making it difficult for even experts to figure out gaps and inconsistencies in its reasoning and explanations [28].In the face of uncertainty about the validity of the LLM output, we need to backtrack, decompose and iterate until the output becomes comprehensible.
Motivation and Affect: Code generation using LLMs like Codex is known to elicit positive as well as negative sentiments in learners (fear, joy, wonder, confusion etc.) [33].Learners must be taught to see themselves in the driver seat, with the LLM as a copilot.

Summary
To summarize, we believe that pair thinking with LLMs for programming involves first arriving at a joint understanding of the problem, and then the human evaluating the LLM's solution (with the LLM's help).Self-regulation strategies such as process monitoring and reflection can support learners retain control of the conversation.
In addition, learners need to be metacognitively aware of one's own capabilities as well as of the tools they use, and regulate their cognition, motivations and emotions in response to the artifacts and suggestions generated by LLM tools.

EDUCATIONAL INTERVENTION DESIGN
In the previous section, we proposed an SRL framework which describes a programming problem solving process when generative AI is used.A goal of the framework is to help methodically explore the design of learning interventions that foster self-regulation.
In this section, we outline interventions that address specific constituent elements of the SRL framework described in Sec. 4. Table 1 summarizes the features of such interventions with illustrative examples.Broadly, we classify interventions into those that (1) address problem understanding and decomposition, (2) improve program comprehension and evaluation, (3) externalize student self-regulation and metacognition, and (4) regulate motivation and affect.

Problem Understanding and Decomposition
Problem Context: LLMs can guide students in clarifying programming concepts as well as provide explanations for topics and concepts which might be needed to solve a problem.When suitably constrained, they will not solve problems completely, thereby enhancing learning environments.
Problem specification: The tools Promptly [7] and Porpoise [16] are examples of interventions which develop students' ability to specify problems well.These tools do not present learners with a textual problem description immediately usable by an LLM.Instead, they are required to interpret a visual representation of the problem, and translate this into a prompt for an LLM.The AI will automatically generate code based on the prompt, and then test the code against test cases.The essential characteristic of such interventions is to provide vague specifications that trigger behaviours which will make students think more deeply about the problem.
Problem decomposition: Interventions that focus on problem decomposition ask learners to derive the individual components of the large problem, and then prompt the AI model to generate code component by component.Learners have to combine and adapt multiple pieces to generate the final solution.Even for simpler problems, several interventions are possible, such as learners providing sub-goals [27] and comments as prompts, and using these for AI models to generate code.
Teaching 'prompt engineering': How to hold a conversation with an LLM is an important aspect of using it.Its response depends on the phrasing of requests that have become known as prompts, so instructors should equip students with prompt engineering skills.Denny et al. [6] argue that prompt engineering in itself is a useful learning activity that promotes computational thinking.Recent work has synthesized a catalog of prompt engineering techniques, known as prompt patterns [40], that have been applied to solve common problems when using conversational AI.Taking a step back, prompt engineering is about clarifying the problem statement until it 'makes sense' to both the student and the AI, reflecting a joint understanding.Interventions that assist students in learning general and domain-specific prompting techniques will improve CS students' ability to understand problems and evaluate solutions.

Program Comprehension and Evaluation
LLMs provide decent explanations of code [24,36].This ability is useful in interventions that ask students to use AI to understand large code bases.Interventions for developing program evaluation include providing activities for students to modify and adapt code generated by AI, such as altering a solution for a new use case or completing a partial implementation.
Another category of interventions are those that strengthen productive solution evaluation techniques and discourage disadvantageous ones.For example, interventions to detect and alert students who are drifting towards accepting an incorrectly generated solution would be valuable.
Interventions to improve software design are also possible.For example, we may use LLMs to detect use and misuse of design patterns, as well as address aspects of good design like well-chosen names and the quality of comments.

Externalized Metacognition and Self-Regulation
The recorded conversations between the AI and the student enables researchers to capture students' internal metacognition and self-regulation processes.Analysing the progression of student interactions with an LLM would yield a clearer picture of the underlying self-regulation strategies that learners employed in the problem solving process.Insights from such an analysis should help develop guidelines, recommendations and even more specialized interventions on enhancing specific problem solving skills.The externalization of the problem solving process suggests a new class of self-reflection activities.For example, instructors may assign essay assignments to students, asking them to reflect on their dialogue with the AI.This provides opportunities for instructors to make students aware of metacognitive and self-regulation aspects of problem solving based on students' own dialogues.

Motivation and Affect
LLMs sometimes frustrate students by generating confusing content, or lead to indifference when most programming tasks are easily solved by the tool [33,34].Motivation scaffolds [3] such as engaging in tasks of interest to students, or extending gamification methods like CTFs [5,39] to include LLMs are interventions that could be devised to address such issues; this area needs further investigation.

CONCLUSION
In this position paper, we discussed challenges and opportunities that generative AI technologies pose for novice learners' selfregulation during programming problem solving.Although LLMs might hamper students' self-regulation skills, these tools also offer transformative affordances and features to design interventions that promote self-regulation and problem solving in learners.
The key contributions of this paper are (1) A new conception of using generative AI, pair thinking, that suggests a rich role for AI tools, (2) A novel SRL framework for this new context, (3) Use of the framework to structure the design space of interventions, showing where existing ones are located and where new interventions are needed, and (4) Proposal of a new class of interventions using externalization of metacognition and self regulation.We believe this work provides guidelines for instructors and researchers to design and evaluate interventions that support effective integration of generative AI in computing education.

Figure 1 :Figure 2 :
Figure 1: The proposed self-regulation framework for programming problem solving using generative AI

Table 1 :
Features and examples of educational interventions for integrating LLMs while supporting SRL