Generative AI-Enhanced Academic Writing: A Stakeholder-Centric Approach for the Design and Development of CHAT4ISP-AI

This study examines the impact of Generative AI (GenAI) chatbots on improving students' academic writing and critical thinking skills. It addresses ethical and operational challenges, including concerns about academic integrity within AI in education (AIEd). Our study aims to analyze perspectives from a diverse array of stakeholders to inform the creation of effective GenAI chatbots. The insights gained will guide the development of comprehensive AI literacy and robust regulatory frameworks, ensuring that these advancements are both ethically sound and practically viable. The primary focus of the study is to understand stakeholders' expectations of GenAI in academic writing, leading to the development of CHAT4ISP-AI, a specialized chatbot aimed at improving the academic writing, analytical, and critical reasoning skills of first-year undergraduate social science students. This study promotes a contemporary educational approach by fostering collaboration among teachers, students, and other stakeholders, significantly advancing the integration of AI into the educational system, and thus preparing students for an AI-driven future.


INTRODUCTION
AI-powered chatbots demonstrate vast potential in diverse fields by enabling rapid information access, streamlining workflows, and boosting productivity.Hence, the notable surge of interest on a global scale regarding Generative Pre-trained Transformers (GPT) following their public introduction in late 2022 [2,10,21].This progression in AI-powered tools is thrilling, yet rapid change can be disorienting.Amidst this ongoing global disruptive digital transformation, the education sector is prominent in facing significant potential benefits and challenges [7,12,26].These challenges pose significant ethical and operational dilemmas associated with AI chatbots like ChatGPT, particularly concerning issues of plagiarism and other academic misconduct [12].
Alongside these concerns, there is a noticeable positive interest in the opportunities inherent in technology.This interest reflects the established recognition of technology's pedagogical potential within AI in Education (AIEd) literature, where stakeholders such as educators, students, academic administrators, and policymakers are eager to investigate how these tools and technologies can improve teaching and learning.Integral to this exploration is the concept of AI literacy, which entails preparing individuals with the necessary knowledge and skills to responsibly and effectively utilize AI in academic environments.This initiative integrates technology with educational objectives, underlining education's role in promoting data and AI literacy, thereby making teachers' expertise in AI technologies crucial for equal participation in these fast-changing domains [15,20,23,29].Therefore, there is a need to develop appropriate policies and regulations to facilitate the effective integration of AI chatbots into education.This should involve considering the varied viewpoints of different stakeholders to support academic excellence [19].In this regard, a comprehensive understanding of data-driven practices is essential, as it will improve their ability to utilize AI systems effectively [22].
This approach is especially notable for Humanities and Social Science students, who frequently face challenges in effectively articulating their thoughts in essay-based assessments [14].Discussions surrounding AI chatbots in assessment have revealed a disruption to the traditional assumption of equivalence between the quality of expression and critical thinking [1,4,13].Existing digital support tools for academic writing primarily focus on essay scoring and evaluative feedback, yet they generally lack significant features designed to enhance the development of writing strategies and the effective expression of arguments [20].
Our study explores the following questions: What are the concerns of teachers and the expectations of students regarding the use of GenAI technologies for academic writing, and how can their perspectives influence the development of an AI tool that upholds ethical standards and enhances writing and critical thinking skills?.We have achieved a significant milestone with the development of a custom AI tool called CHAT4ISP-AI.The tool is a functional model designed to support first-year social science students in enhancing their academic writing skills, and thereby equipping them for an AI-influenced economy.It is important to acknowledge that this study utilizes the initial version of CHAT4ISP-AI as a proof of concept.The development of CHAT4ISP-AI represents the first step in an evolving phase toward fully realizing the potential of GenAI technologies in educational environments.This study lays the groundwork for subsequent iterations and improvements that will expand CHAT4ISP-AI's capabilities and address emerging needs in academic support.The study aims to bolster academic writing skills, support scholarly arguments, and foster collaboration among academics using a modern pedagogical approach for the XXI century.It is methodically structured into six sections covering everything from literature review to research findings.

RELATED STUDIES
Disruptive innovation in education signifies a paradigm shift from traditional, teacher-centered classrooms to more flexible and diverse learning environments.This shift is notably evident in Educational Technology (EduTech), enhancing access to information, collaboration, and personalization in learning, thereby transforming the structure and delivery of education into a more adaptable and learner-centered experience [16,25,27,28].In Gabashvili's bibliometric analysis, eleven relevant literature sources on Chat-GPT applications across various fields were identified [5].While recognizing the widespread hype and transformative potential of AI across various sectors, particularly in industry and academia, the study highlights the necessity for interdisciplinary research, customized integration, and ethical considerations to ensure AI is used responsibly.ChatGPT has found responsible applications across academic domains, for example, mathematics, where it has proven effective in improving mathematical skills and has received positive feedback.However, it has proved more difficult to apply to comprehending intricate subjects and ensuring precise solutions [24].
Considerable research has been dedicated to the challenges of teaching procedural knowledge, especially in writing.[18] identifies the difficulty of expressing such knowledge as explicit propositions, noting that demonstration and practice are often more effective teaching methods.This insight has been instrumental in the development of Intelligent Tutoring Systems (ITS), particularly for subjects requiring intricate understanding.[6] explore the progress and achievements of ITS over the past two decades, with a focus on STEM education.The study highlights the emergence of a new generation of ITS that engages students in interactive dialogues, representing a notable advancement in student engagement and learning compared to traditional teaching methods.Expanding upon these developments, [17] introduced the Writing Pal, an ITS tailored for training in writing strategies.The study explores AI systems' role in analyzing essay quality and providing feedback, highlighting the significance of integrating diverse linguistic and rhetorical elements for effective prediction and meaningful feedback in essay assessment.
Another substantial issue encountered in academia when using ChatGPT as digital support for pedagogics is ethical concerns.Grassini [7] examined the transformative impact of ChatGPT.While acknowledging the remarkable capabilities of these AI models, the study raised concerns about their widespread use and opacity.It explored the mixed reactions within the educational sector, with some educators viewing it as a positive development and others expressing concerns about its potential to reduce analytical skills and promote misconduct.This is especially evident in courses that involve essay writing and the capacity for expressing arguments [20].In this regard, [11] performed a literature review on the role of ChatGPT' in academic writing and its possible linkages with plagiarism.The study findings emphasize ChatGPT's usefulness as a writing tool but stress the importance of responsible practices, such as proper citation, to maintain academic integrity.Furthermore, a pioneering study examined students' perceptions of AI plagiarism known as AI-giarism, a new type of academic dishonesty involving using AI-generated text without acknowledgment.Surveying both undergraduate and postgraduate students, the study uncovered intricate attitudes toward AI-giarism scenarios.It revealed a consensus against direct AI content generation but a mixed response to subtler applications of AI.The study introduced a novel instrument to conceptualize AI-giarism, offering a valuable tool for educators and policymakers.It emphasized the need for adapting definitions of academic misconduct in response to evolving AI technology [12].After reviewing existing related studies, we transition to the methodology section, which will describe in detail our approach using the proposed CHAT4ISP-AI.

STUDY SETTINGS AND METHOD
CHAT4ISP-AI is a proof-of-concept and an acronym for a custom AI chatbot designed to support academic writing for International Social Science undergraduate students.The project has two main parallel elements technological and pedagogical phases with complex stakeholder involvement.This study uses Soft Systems Methodology (SSM) to navigate stakeholder perspectives, aiming for iterative enhancements of CHAT4ISP-AI through feedback and evaluations, to improve its application in academic contexts.

Technological Phase
CHAT4ISP-AI is developed with Python and Javascript, focuses on improving academic writing by offering high-quality feedback, identifying errors, and personalizing learning paths.It provides human-like responses, writing assistance, and material suggestions while operating on a local server with two functional levels.The first level provides basic ChatGPT functionality for early academic writing courses, while the second is a more advanced level and will be introduced in later modules, it offers enhanced features like material recommendation, corpus analysis, readability assessment, and scholarly referencing as illustrated in Figure 1 and 2.

Pedagogical Phase
The deployment and testing of CHAT4ISP-AI occurred within the 30-credit ISP of the 30-credit ISP (International Social Science Programme), Global Issues I, an interdisciplinary English-language course that includes various student cohorts.Over a semester 100-150 students and eight teachers engaged with CHAT4ISP-AI for educational tasks across four modules.As the course progressed, teaching methods were explored and the chatbot's integration into teaching was assessed.The integration involved iterative testing for usability and privacy.This process also generated essential learning about the requirements for capacity building amongst teachers, and the effectiveness of AI-literate pedagogy, and assessment through meetings and workshops.

SSM STAKEHOLDERS ANALYSIS
This section outlines the use of SSM in stakeholder analysis for the integration and evaluation of AI chatbots in an academic setting.It highlights the diverse roles of stakeholders including developers, IT support, teachers, and students, emphasizing the importance of their collaboration.
SSM was used in this project to navigate complex issues, focusing on a search for accommodation between diverse stakeholder perspectives rather than the development of solutions to well-defined problems [2,3,8].SSM is a systematic and holistic approach, aiming for sustainable development and effective problem-solving in undefined and complex real-world situations [3].SSM also involves discussions in identifying improvement opportunities, leading to accommodations between stakeholders on actions for enhancement.Through this process, we gained deeper insights into the academic setting, revealing nuanced concerns and viewpoints of stakeholders such as teaching staff that were not initially apparent.While SSM does not aim to provide a definitive stakeholder list or a fixed system framework, its iterative ongoing cycles of action and reflection allow for continuous adaptation and refinement, making it an ideal approach for the dynamic and evolving landscape of our project.

Stakeholders Groups
The initiative entails a collaborative effort among three faculties and the academic support center, engaging a diverse group of stakeholders comprising about 100 to 150 undergraduate students, 8 teachers, and additional technical members who each bear distinct roles and responsibilities.
Developers were primarily responsible for the technological phase of the system, i.e. designing and creating the CHAT4ISP-AI.Teachers had primary responsibility for the pedagogical phase, focusing on the integration of CHAT4ISP-AI with instruction and course content.Students were the ultimate end-users of the project, seeking an efficient educational environment that could boost their learning capabilities.The IT department was responsible for the technical infrastructure such as the connectivity using the institutional server which underpins teaching at the university supported by ICT.The Digital Transformations Knowledge Environment brought together research, teaching, and public engagement perspectives, and has funded the project.Finally, there are support staff in the faculties and in the central library who provide expertise and guidance on using technology in teaching.

Implementing SSM in Practice
SSM was applied iteratively across two distinct modes, engaging various stakeholders throughout the process.Mode 1 is more structured and is based on applying the technique to qualitative data from the situation: observations and interviews.Mode 2, where analysis is co-constructed, is more adaptive and suitable for situations where stakeholder engagement and iterative learning are key.In practice, practitioners of SSM might move between these modes, combining the systematic rigor of Mode 1 with the flexibility and stakeholder engagement of Mode 2. In this study, the developer group, support staff, and a teacher engaged with Mode 2 methodology in an interactive workshop setting.During these sessions, "rich pictures" detailed illustrations capturing key elements, relationships, conflicts, and constraints were used to facilitate discussions about the project's environment and components.This collaborative approach led to the creation of verb models (this is also known as activity model, conceptual model, or root definitions) each representing different perspectives on how the project's objectives could be shaped.Additionally, further verb models were developed through Mode 1 methodology, informed by private analyses based on stakeholder meetings and observations.The primary value of these models was in aiding the understanding of various stakeholder viewpoints, rather than in their literal accuracy.This iterative process culminated in the formulation of relevant root definitions, reflecting the collaborative insights gained: • A system to develop a chatbot to support students by helping them improve their written assignments, thereby saving time for teachers.• A system for teachers to learn about the use of GenAI in a practice setting to inform and guide the use of GenAI in Higher Education (HE).• A system to provide access to staff to ChatGPT through the university, so that they can begin to learn how to apply the potential of a new technology.• A system to provide access to ChatGPT through the university's API to prototype and develop models for providing legal access to the system • An invidious system that undercuts the basis of HE in skills development and assessment by allowing students to produce text that does not reflect their actual understanding or ability.
• A system to support students' learning and development by providing an innovative context for them to reflect critically on academic writing.• A system for students to develop their capacity to productively use an emerging technology that will probably shape their future careers.• A system to push students into thinking about ICTs even if they don't particularly want to, at an early stage in their studies.
• A system to help students with tasks that they need to fulfill for their studies.• A system to demonstrate to students the value of ChatGPT for using it effectively in ways that help them to accomplish tasks and assignments.Based on this starting point, the analysis was refined using a method that has been modified from [8] to focus on a different aspect of what [2] defines as a verb model.This adapted approach still represents alternative perspectives on the project through a verb or process.In addition, it draws out the worldview that provides meaning to that particular viewpoint, an important element of verb models according to [9] analysis of root definitions across the history of SSM.This adaptation involves a reflective examination of the underlying assumptions associated with that specific perspective and is presented in Table 1

Developers
to support students to improve their writing Getting direct feedback and analysis from ChatGPT will improve students' capacity for academic writing to save time for teachers Giving detailed feedback to students is something that takes time from teachers that could better be spent on other tasks.
IT department to provide access to ChatGPT for staff Resources available to IT are limited, and a better focus for the organization is working to build the capacity of teaching and support staff to use it.
to prototype services that make the LNU API access available to users Little is yet known in the university about how to use ChatGPT in education in practical terms and therefore it makes sense to spend some time experimenting and building capacity Teachers/Instructors to undercut and devalue skills development Student use of GenAI is a shortcut, and students who use the technology will not develop their own capabilities as much as they otherwise would.
to reflect critically on academic writing Students who are confronted by chat-generated text can be drawn into analyzing it in ways that reinforce their critical capabilities.
to develop student capacity to use GenAI When these students graduate, the economy might have been reshaped by AI technologies in ways that mean their job prospects will be influenced by their AI literacy.

Students to be pushed
There is a reluctance to engage with ChatGPT that possibly comes from particular reasons not to, or is about minimizing study effort and engagement to appropriate levels.
to demonstrate the value of ChatGPT ChatGPT can be used in many ways (e.g.summarizing) that speak directly to tasks and ideas that students might struggle with.

INITIAL FINDINGS WITH DISCUSSIONS
In the first module of the course, students were given free access to the level 1 implementation of CHAT4ISP-AI, and encouraged to explore its use, reporting back what they learned in class tasks, group assignments, and the final assessment.There was a significant shift in their engagement with the technology during the class, going from a situation where a show of hands indicated that the majority of students had never used GenAI technology at the beginning of the class.Several exchange students in the class indicated that at their home universities, it was not allowed to use GenAI at all and expressed some surprise that a teacher should be encouraging them to experiment.By the end of the first week, a poll showed that over 70% of those in the class had tried GenAI outside of the exercises within lectures, and at the beginning of the second week, around 30% of the students in the class had tried the prompt engineering course recommended by the developers available here.By the final week of the course, a majority of students reported using GenAI in the exam for idea generation, summarizing sources, spelling, and grammar checking.Also in the final seminar, every group of students used GenAI in preparing a short essay on technological singularity and reported on how they had made use of it in preparing the document.Students expressed 2 kinds of negative reactions to GenAI that could limit their willingness to use it: (i) Fear that they were transgressing somehow and would be caught and punished.
(ii) a hesitancy to use it in case it took away from their authentic expression of their thoughts in their work.
In our academic settings, there has been considerable interest in GenAI amongst the broader community of social science teachers.However, the early stages of this project have highlighted that there is significant work to build teachers' understanding of the technology and its actual implications for their practice.As with many professions potentially affected by this kind of technology, there was a strong negative reaction to the idea of GenAI.This often focuses on the challenge of how to ensure the quality and reliability of assessment once students have access to it.There is a feeling that it is the weaker students who already produce sub-quality text who are most likely to be drawn into generating work that they claim as their own, without having checked it through.Yet, many teachers who have been quite willing to talk about the merits and demerits of GenAI seem not to have spent much time working with it.One effect of the project has therefore been to provide opportunities for demonstrations of the use of GenAI technologies with colleagues, and several times they then became quite enthusiastic about trying it out for themselves and with their classes.
The more sustained engagement with CHAT4ISP-AI in a classroom setting suggests that the quality of academic writing produced by it is mixed.It is much stronger in form as opposed to content and particularly struggles to integrate appropriate evidence into narrative arguments.The academic writing produced is flawed and does not accomplish everything that a good student should be able to do.GenAI's failures are in many ways more interesting than its successes in terms of being able to have a conversation with students about good academic writing.This reflects some of the discussion in the literature, where emphasis has been placed on the significance of generating authentic and credible academic output by GenAI.It underscores the need for robust scholarly dialogues, cultivation of human intelligence, and critical reasoning within the study process [21].
Many areas need to be explored to guarantee the secure and conscientious integration of GenAI or Large Language Models (LLMs) in teaching, some of these concerns can be linked to the quality or accuracy of generated outputs.However, it could be that the performance of the technology relates to the scholarly quality of pre-trained input data it receives or knowledge it was built upon.This notion is supported by the empirical investigation conducted by [30], which critically examined different LLMs with specific regard to the quality of datasets used for their pre-training.The study revealed that many existing LLMs predominantly utilize web-based, conversational, and other non-academic data sources, indicating potential limitations in their application for academic writing tasks.This observation underscores the need for further investigation into the appropriateness of LLMs tailored for educational and scholarly contexts.Moving forward, the subsequent section will delve into a focused discussion and exploration of how CHAT4ISP-AI, can be optimized or restructured to better serve the academic community, aligning with the specific aims and objectives of this study.

CONCLUSION AND FUTURE DIRECTIONS
This study is limited by its focused scope, confined to first-year Social Sciences students and potentially restricting the broader applicability of its findings.However, CHAT4ISP-AI has demonstrated versatility, extending its influence beyond the initial pedagogical objectives.Students have harnessed its capabilities for a wide range of academic tasks, providing instructors with valuable insights into both the potential and challenges associated with AI in education.As students become more proficient with GenAI technology, it is imperative to engage with them as empowered experienced users that can boost the implementation of these solutions rather than mere customers.However, the effective application of GenAIs in educational settings transcends mere technological use; it needs comprehensive strategies that align with the fundamental principles of academic inquiry and modern perspectives related to the nature of learning and knowledge in the XXI century.
Despite these initial promising results, our stakeholder analysis suggests significant divergence between various stakeholders regarding the objectives and capabilities of a GenAI-enabled platform for academic writing support.While there is substantial recognition of the notion that technology can enhance academic performance when properly integrated, questions persist about the degree of autonomy students should have when using the system without teacher feedback, as well as the level of support educators require to harness the platform's educational applications.These issues present clear avenues for further exploration.
To advance the vision of using GenAI to support academic writing and in this way increase students´critical thinking, stakeholders must prioritize the development of competencies and literacy for the responsible use of GenAI and the pre-training of LLMs on a diverse range of scholarly and data sources to optimize their educational functionality.Notable educational adaptations in LLMs include foundational models like 'EdGPT, ' fine-tuned with educational data to enhance performance in educational tasks [21].Moving forward, empirical investigations into the impact of CHAT4ISP-AI on students' academic writing remain a priority.Plans are underway to enhance the platform's user interface and features through facultywide collaboration.As a step toward this development, we are in the process of organizing a mini-conference focused on teaching and pedagogy derived from the project's framework.Also, we aim to launch our tool among humanities students to assess its adaptability and gather broader perspectives and insights.Despite practical solutions on the horizon, operational challenges, especially given the issues related to time and funding for further development, persist.To further explore the potential of CHAT4ISP-AI in educational contexts, future investigations could include empirical studies assessing its impact on student learning, comparative analyses of its use with and without teacher feedback, and investigations into how educator experience influences outcomes.Longitudinal studies on learning retention, adaptability across disciplines, and user experience surveys could provide deeper insights.Additionally, examining ethical and privacy aspects, as well as the system's integration with traditional teaching methods would offer a comprehensive understanding of CHAT4ISP-AI's role and efficacy in enhancing educational experiences.
A proposed Randomized Controlled Trial (RCT) aims to provide robust data on the tool's influence on academic writing.This entails developing or adapting stringent evaluative metrics to understand the extent of students' interaction with the system and how it reflects in their academic performance.Additionally, the data gathered from this trial can serve a dual purpose by contributing to the design and development of a Learning Analytics Dashboard alongside CHAT4ISP-AI.This dashboard could inform pedagogical strategies, facilitating continuous improvement in instructional approaches.In conclusion, our study initiative has garnered significant attention and institutional interest across various faculties at our university.This momentum indicates its readiness for additional funding and the expansion of its scope, culminating in the full development and deployment of Level 2 of the CHAT4ISP-AI model.However, as we advance to Level 2, a cautious and critical analysis is essential to evaluate its applicability and integration into teaching, moving beyond autonomous use only.Understanding CHAT4ISP-AI's effectiveness in diverse educational settings will be instrumental in its long-term success and widespread acceptance.

ACKNOWLEDGEMENT
This project is part of a broader initiative supported by the digital transformation program (link), focusing on fostering collaborative efforts within the academic community to embrace a forwardthinking educational strategy.

Figure 1 :
Figure 1: An overview of CHAT4ISP-AI's components and features

Figure 2 :
Figure 2: Phases of CHAT4ISP-AI Figure1details the unique characteristics of CHAT4ISP-AI in three categories: Category A describes its operational features and goals; Category B explains the user interaction with both frontend and backend through OpenAI API calls; and Category C showcases the interface's output layout, consisting of panels for content, generated results with highlights, and recommendations to useful scientific sources.Figure2illustrates the ongoing evolutionary phases of Chat4ISP-AI in contrast to the basic versions of ChatGPT and Quillbot, showcasing its development plan from the initial phase to the proposed final phase, where it is equipped with a comprehensive suite of academic support features.

Table 1 :
Adapted approach of Root Definition or Verb Models