Co-Writing with Opinionated Language Models Affects Users’ Views

Social influence is defined as a shift in an individual’s thoughts, feelings, attitudes, or behaviors as a result of interaction with others [92]. While social influence is integral to human collaboration, landmark studies have shown that it can also lead to unreasonable or unethical behavior. On a personal level, people may conform to majority views against their better judgement [6] and obey an authority figure even if it means harming others [80]. On a societal level, researchers have shown that social influence drives speculative markets [93], affects voting patterns [69], and contributes to the spread of unhealthy behaviors such as smoking and obesity [30, 31].

Following the rise of social media, how online interactions affect people’s opinions and decisions has been studied extensively. Research has shown that a variety of sources influences users’ attitudes and behaviors, including friends, family, experts, and internet celebrities [48, 78]; the latter group was labeled influencers due to their influence on a large group of ’followers’ [10]. Research has found that in online settings, users can be influenced by non-human entities such as brand pages, bots, and algorithms [41]. Studies have evaluated the influence that technical artifacts such as personalized recommendations, chatbots, and choice architectures have on users’ decision-making [15, 35, 50, 72].

The influence that algorithmic entities have on people depends on how people perceive the algorithm, for example, whether they attribute trustworthiness to its recommendations [50, 76]. The influence of algorithms on individuals tends to increase as the environment becomes more uncertain and decisions become more difficult [20]. With the public’s growing awareness of developments in artificial intelligence, people may regard smart algorithms as a source of authority [2, 60, 76]. There is recent evidence that people may accept algorithmic advice even in simple cases when it is clearly wrong [74]. In the related field of automation, such over-reliance on machine output has been referred to as automation bias [86, 87, 102].

Historically, HCI research for text entry has predominantly focused on efficiency [68]. Typical text entry systems attend to language context at the word [18, 98] or sentence level [5, 27]. They suggest one to three subsequent words based on underlying likelihood distributions [40, 43, 49, 89]. More recent systems also provide multiple short reply suggestions [59] or a single long phrase suggestion [29]. More extensive suggestions have traditionally been avoided because the time taken to read and select them might exceed the time required to enter that text manually. Several studies indicate that features such as auto-correction and word suggestions can negatively impact typing performance and user experience [11, 26, 37, 85].

However, with the emergence of larger and more powerful language models [21, 97, 103], there has been a growing interest in design goals beyond efficiency. Studies have investigated interface design factors and interactions with writing assistants that directly or indirectly support inspiration [17, 70, 95, 105], language proficiency [27], story writing [95, 105], text revision [36, 107] or creative writing [33, 46]. Here, language models are framed as active writing partners or co-authors [70, 104, 105], rather than tools for prediction or correction. There is evidence that a system that suggests phrases rather than words [5] is more likely to be perceived as a collaborator and content contributor by users.

The more writing assistants become active writing partners rather than mere tools for text entry, the more the writing process and output may be affected by their “co-authorship”. Bhat et al. [17] discuss how writers evaluate the suggestions provided and integrate them into different cognitive writing processes. Work by Singh et al. [95] suggests that writers make active efforts or ’leaps’ to integrate generated language into their writing. Buschek et al. [27] conceptualized nine behavior patterns that indicate varying degrees of engagement with suggestions, from ignoring them to chaining multiple ones in a row. Writing with suggestions correlates with shorter and more predictable texts being written [4], along with increased use of standard phrases when writing with a language model [17, 27]. Furthermore, the sentiment of the suggested text may affect the sentiment of the written text [3, 52].

Technical advances have given rise to a generation of language models [21] that produces language so natural that humans can barely distinguish it from real human language [56]. Enabled by improvements in computer hardware and the transformer architecture [97], models like GPT-3 [23, 90] have attracted attention for their potential to impact a range of beneficial real-world applications [21]. However, more cautious voices have also warned about the ethical and social risks of harm from large language models [100, 101], ranging from discrimination and exclusion [23, 54, 83] to misinformation [67, 75, 91, 106] and environmental [96] and socioeconomic harms [14].

Comparatively little research has considered widespread shifts in opinion, attitude, and culture that may result from a comprehensive deployment of generative language models. It is known that language models work and perform better for the languages and contexts they are trained in (primarily English or Mandarin Chinese) [23, 91, 103]. Small-n audits have also suggested that the values embedded in models like GPT-3 were more aligned with reported dominant US values than those upheld in other cultures [57]. Work by Jakesch et al. [55] has highlighted that the values held by those developing AI systems differ from those of the broader populations interacting with the systems. The adjacent question of AI alignment – how to build AI systems that act in line with their operators’ goals and values – has received comparatively more attention [7], but primarily from a control and safety angle. A related topic, the political repercussions of social media and recommender systems [108], has received extensive research attention, however. After initial excitement about social media’s democratic potential [62], it became evident that technologies that affect public opinion will be subject to powerful political and commercial interests [22]. Rather than mere technical platforms, algorithms become constitutive features of public life [47] that may change the political landscape [1]. Even without being designed to shift opinions, it has been found that algorithms may contribute to political polarization by reinforcing divisive opinions [9, 24, 32].

To study interactions between model opinion and participant opinion in a possibly realistic and relevant setting, we created the scenario of an opinionated discussion on social media platforms like Reddit. Such discussions have a large readership [79], pertain to political controversies, and are plausible application settings for writing assistants and language models. We searched ProCon.org¹, an online resource for research on controversial issues, to identify a discussion topic. We selected “Is Social Media Good for Society?” as a discussion topic. We chose this topic because it is an easily accessible discussion topic that is politically relevant but not considered so controversial that entrenched views may limit constructive debate.

To run the experiment, we created a custom experimental platform combining a mock-up of a social media discussion page, a rich-text editor, and a writing assistant. The assistant was powered by a language generation server and included comprehensive logging tools. To provide a realistic-looking social media mock-up, we copied the design of a Reddit discussion page and drafted a question based on the ProCon.org discussion topic. Figure 2 shows a screenshot of the experiment. We asked participants to write at least five sentences expressing their take on social media’s societal impact. We randomly assigned participants to three different treatment groups:

Similar to Google’s Smart Compose [29] and Microsoft’s predictive text in Outlook, the writing assistant in the treatment groups suggested possible continuations (sometimes called “completions”) to text that participants had entered. We integrated the suggestions into a customized version of the rich-text editor Quill.js². The client sent a generation request to the server whenever a participant paused their writing for a certain amount of time (750ms). Including round-trip and generation time, a suggestion appeared on participants’ screens about 1.5 seconds after they paused their writing.

When the editor client received a text suggestion from the server, it revealed the suggestion letter by letter with random delays calibrated to resemble a co-writing process (cf. [71]). Once the end of a suggested sentence was reached, the editor would pause and request from the server an extended generation until at least two sentences had been suggested. Participants could accept each suggested word by pressing the tab key or clicking an accept button on the interface. In addition, they could reset the generation, requesting a new suggestion by pressing a button or key.

We hosted the required cloud functions, files, and interaction logs on Google’s Firebase platform.

In this study, we experimented with language models that strongly favored one view over another. We chose a strong manipulation as we wanted to explore the potential of language models to affect users’ opinions and understand whether they could be used or abused to shift people’s views [8].

We used GPT-3 [23] with manually designed prompts to generate text suggestions for the experiment in real-time. Specifically, we accessed OpenAI’s most potent 175B parameter model (“text-davinci-002”). We used temperature sampling, a method for choosing a specific next token from the set of likely next tokens inspired by statistical thermodynamics. We set the sampling temperature (randomness parameter) to 0.85 to generate suggestions that are varied and creative. We set the frequency and presence penalty parameters to 1 to reduce the chance that the model suggestions would become repetitive. We also prevented the model from producing new lines, placeholders, and list by setting logit bias parameters that reduced the likelihood of the respective tokens being selected.

We evaluated different techniques to create an opinionated model, i.e., a model that likely supports a certain side of the debate when generating a suggestion. We used prompt design [73], a technique for guiding frozen language models to perform a specific task. Rather than updating the weights of the underlying model, we concatenated an engineered prompt to the input text to increase the chance that the model generates a certain opinion. Specificially we inserted the prefix "Is social media good for society? Explain why social media is good/bad for society:" before participants’ written texts when generating continuation suggestions. The engineered prompt was not visible to participants in their editor UI; it was inserted in the backend before generation and removed from the generated text before showing it to participants.

Initial experimentation and validation suggested that the prompt produced the desired opinion in the generated text, but when participants strongly argued for another opinion in their writing, the model’s continuations would follow their opinion. In addition to the prefix prompt, we thus developed an infix prompt that would be inserted throughout participants’ writing to reinforce the desired opinion. We inserted the snippet ("One sentence continuing the essay explaining why social media is good/bad:") right before the last sentence that participants had written. This additional prompt guided the model’s continuation towards the target opinion even if participants had articulated a different opinion earlier in their writing. Validation of the model opinion configuration is provided in section 4.5. We also experimented with fine-tuning [53] to guide the models’ opinion, but the fine-tuned models did not consistently produce the intended opinion.

We collected different types of outcome measures to investigate interactions between participants’ opinions and the model opinion:

Opinion expressed in the post: To evaluate expressed opinion, we split participants’ written texts into sentences and asked crowd workers to evaluate the opinion expressed in each sentence. Each crowd worker assessed 25 sentences, indicating whether each argued that social media is good for society, bad, or both good and bad. A fourth label was offered for sentences that argued neither or were unrelated. For example, "Social media also promotes cyber bullying which has led to an increase in suicides" (P#421) was labeled as arguing that social media is bad for society, while "Social media also helps to create a sense of community" (P#1169) was labeled as social media is good for society. We collected one to two labels for each sentence participants wrote and collected labels for a sample of the writing assistant’s suggestions. In sentences where we collected multiple labels, the labels provided by different raters agreed 84.1% of the time (Cohen’s κ = 0.76).

Real-time writing interaction data: We gathered comprehensive interaction logs at the key-stroke level of how participants interacted with the model’s suggestions. We recorded which text the participant had written, what text the model had suggested, and what suggestions participants had accepted from the writing assistant. We measured how long they paused to consider suggestions and how many suggestions they accepted.

Opinion survey (post-task): After finishing the writing task, participants completed an opinion survey. The central question, “Overall, would you say social media is good for society?” was designed to assess shifts in participants’ attitude. This question was not shown immediately after the writing task to reduce demand effects. Secondary questions were asked to understand participants’ opinions in more detail: “How does social media affect your relationships with friends and family?”, “Does social media usage lead to mental health problems or addiction?”, “Does social media contribute to the spread of false information and hate?”, “Do you support or oppose government regulation of social media companies?” The questions were partially adapted from Morning Consults’ National Tracking Poll [34]; answers were given on typical 3- and 5-point Likert scales.

User experience survey (post-task): Participants in the treatment groups completed a survey about their experience with the writing assistant following the opinion survey. They were asked, “How useful was the writing assistant to you?”, whether “The writing assistant understood what you wanted to say” and whether “The writing assistant was knowledgeable and had expertise.” To explore participants’ awareness of the writing assistant’s opinion and its effect on their own views, we asked them whether “The writing assistant’s suggestions were reasonable and balanced” and whether “The writing assistant inspired or changed my thinking and argument.” Answers were given on a 5-point Likert scale from “strongly agree” to “strongly disagree.” An open-ended question asked participants what they found most useful or frustrating about the writing assistant.

Covariates: We asked participants to self-report their age, gender, political leaning, and their highest level of education at the end of the study. We also constructed a “model alignment” covariate estimating whether the opinion the model supported was aligned with the participant’s opinion. We did not ask participants to report their overall judgment before the writing task to avoid commitment effects. Instead, we asked them at the end of the study whether they believed social media was good for society before participating in the discussion. While imperfect, this provides a proxy for participants’ pre-task opinions. It is biased by the treatment effect observed on this covariate, which amounts to 14% of its standard deviation.

We recruited 1,506 participants (post-exclusion) for the writing task, corresponding to 507, 508, and 491 individuals in the control, techno-optimist, and techno-pessimist treatment groups, respectively. The sample size was calculated based on effect sizes observed in the pilot studies’ post-task question, "Overall, would you say social media is good for society?" at a power of 80%. The sample was recruited through Prolific [84]. The sample included US-based participants at least 18 years old (M= 37.7, SD= 14.2); 48.5% self-identified as female, and 48.6% identified as male. 38 participants identified as non-binary and eight preferred to self-describe or not disclose their gender identity. Six out of ten indicated liberal leanings; 57.1% had received at least a Bachelor’s degree. Participants who failed the pre-task attention check (8%) were excluded. Six percent of participants admitted to the task did not finish it. We paid participants $1.50 for an average task time of 5.9 minutes based on an hourly compensation rate of $15. For the labeling task, we recruited a similar sample of 500 participants through Prolific. The experimental protocols were approved by the Cornell University Institutional Review Board.

The experiment materials, analysis code and data collected are publicly available through an Open Science repository (https://osf.io/upgqw/). A research assistant screened the data, and records with potentially privacy-sensitive information were removed before publication.

Figure 3 shows how often participants in each of the treatment conditions (y-axis) argued that social media is good or bad for society (x-axis) in their writing. The social media posts written by participants in the control group (middle row) were slightly critical of social media: They argued that social media is bad for society in 38% and that social media is good in 28% of their sentences. In about 28% of their sentences, control group participants argued that social media is both good and bad, and 11% of their sentences argued neither or were unrelated.

Participants who received suggestions from a language model supportive of social media (top row of Figure 3) were 2.04 times more likely than control group participants (p<0.0001, 95% CI [1.83, 2.30]) to argue that social media is good. In contrast, participants who received suggestions from a language model that criticized social media (bottom row) were 2.0 times more likely (p<0.0001, 95% CI [1.79, 2.24] to argue that social media is bad than control group participants. We conclude that using an opinionated language model affected participants’ writing such that the text they wrote was more likely to support the model’s preferred view.

Participants may have accepted the models’ suggestions out of convenience, even though the suggestions did not match what they would have wanted to say. Paid participants in online studies, in particular, may be motivated to accept suggestions to swiftly complete the task.

Our data shows that, across conditions and treatments, most participants did not blindly accept the model’s suggestions but interacted with the model to co-write their social media posts. On average, participants wrote 63% of their sentences themselves without accepting suggestions from the model (compare Figure 5). About 25% of participants’ sentences were written by both the participant and the model, which typically meant that the participant wrote some words and accepted the model’s remaining sentence suggestion. Only 11.5% of sentences were fully accepted from the model. Participants whose personal views were likely aligned with the model were more likely to accept suggestions, while participants with opposing views accepted fewer suggestions. About one in four participants did not accept any model suggestion, and one in ten participants had more than 75% of their post written by the model.

The opinion differences in participants’ writing may be due to shifts in participants’ actual opinion caused by interacting with the opinionated model. We evaluate whether interactions with the language model affected participants’ attitudes expressed in a post-task survey asking participants whether they thought social media was good for society. An overview of participants’ answers is shown in Figure 4.

The figure shows the frequency of different survey answers (x-axis) for the participants in each condition (y-axis). Participants who did not interact with the opinionated models (middle row in Figure 4) were balanced in their evaluations of social media: 33% answered that social media is not good for society (middle, blue); 35% said social media is good for society. In comparison, 45% of participants who interacted with a language model supportive of social media (top row) answered that social media is good for society. Converting participants’ answers to an interval scale, this difference in opinion corresponds to an effect size of d=0.22 (p<0.001). Similarly, participants that had interacted with the language model critical of social media (bottom row) were more likely to say that social media was bad for society afterward (d=0.19, p<0.005).

After the writing task, we asked treatment group participants about their experience with the writing assistant. We use their answers to estimate to what extent they were aware of the model’s opinion and influence.

The vast majority of participants thought the language model had expertise and was knowledgeable – even if it contradicted their personal views. As shown in Figure 8, 84% of participants said that the assistant was knowledgeable and had expertise when the language model supported their opinion. When the model contradicted their opinion, only 15% of participants said that it was not knowledgeable or lacked expertise.

While the language model was configured to support one specific side of the debate, the majority of participants said that the model’s suggestions were balanced and reasonable. Figure 9 shows that, in the group of participants whose opinion was supported by the model, only 10% noticed that its suggestions were imbalanced (top row in blue). When the model contradicted participants’ opinions, they were more likely (30%) to notice its skew, but still, more than half agreed that the model’s suggestions were balanced and reasonable (bottom row in orange).

Figure 10 shows that the majority of participants were not aware of the model’s effect on their writing. Participants using a model aligned with their view – and accepting suggestions more frequently – were slightly more aware of the model’s effect (34%, top row in orange). In comparison, only about 20% of the participants who did not share the model’s opinion believed that the model influenced them. Overall, we conclude that participants were often unaware of the model’s opinion and influence.

We finally validate that the experimental manipulation worked as intended and address potential concerns about experimenter demand effects.

The literature on social influence and persuasion [92] provides ample evidence that our thoughts, feelings, and attitudes shift due to interaction with others. Our results demonstrate that co-writing with an opinionated language model similarly shifted people’s writing and attitudes. We discuss below how latent persuasion by AI language technologies extends and differs from traditional social influence and conventional forms of technology-mediated persuasion [94]. We consider how the model’s influence can be explained by discussing two possible vectors of influence inspired by social influence theory [92]–informative and normative persuasion– and a third vector of influence extending the nudge paradigm [42, 72] to the realm of opinions.

Our results caution that interactions with opinionated language models affect users’ opinions, even if unintended. The results also show how simple it is to make models highly opinionated using accessible methods like prompt engineering. How can researchers, AI practitioners, and policymakers respond to this finding? We believe that our results imply that we must be more careful about the opinions we build into AI language technologies like GPT-3.

Prior work on the societal risks of large language models has warned that models learn stereotypes and biases from their training data [14, 28, 44] that may be amplified through widespread deployments [19]. Our work highlights the possibility that large language models reinforce not only stereotypes but all kinds of opinions – from whether social media is good to whether people should be vegetarians and who should be the next president. Initial tools have been developed for monitoring and mitigating generated text that is discriminating [23, 54, 83] or otherwise offensive [7]. We have no comparable tools for monitoring the opinions built into large language models and in the text they generate during use. A first exploration of the opinions built into GTP-3 by Johnson et al. [57] suggests that the model’s preferred views align with dominant US public opinion. In addition, a version of GPT trained on 4chan data led to controversy about the ideologies that training data should not contain. We need theoretical advancements and a broader democratic discourse on what kind of opinions a well-designed model should ideally generate.

Beyond unintentional opinion shifts through carelessly calibrated models, our results raise concerns about new forms of targeted opinion influence. If large language models affect users’ opinions, their influence could be used for beneficial social interventions, like reducing polarization in hostile debates or countering harmful false beliefs. However, the persuasive power of AI language technology may also be leveraged by commercial and political interest groups to amplify views of their choice, such as a favorable assessment of a policy or product. In our experiment, we have explored the scenario of influence through a language-model-based writing assistant in an online discussion, but opinionated language models could be embedded in other applications like predictive keyboards, smart replies, and voice assistants. Like search engine and social media network operators [65], operators of these applications may choose to monetize the persuasive power of their technology.

As researchers, we can advance an early understanding of the mechanisms and dangers of latent persuasion through AI language technologies. Studies that investigate how latent persuasion differs from other sorts of influence, how it is mediated by design factors and users’ traits, and engineering work on how to measure and guide model opinions can support product teams in reducing the risk of misuse and legislators in drafting policies that preempt harmful forms of latent persuasion.

As appropriate for an early study, our experiment has several limitations: We only tested whether a language model affected participants’ views on a single topic. We chose this topic as people had mixed views on it and were willing to deliberate. Whether our findings generalize to other topics, particularly where people hold strong entrenched opinions, needs to be explored in future studies. Further, we only looked at one specific implementation of a writing assistant powered by GPT-3. Interacting with different language models through other applications, such as a predictive keyboard that only suggests single words or an email assistant that handles entire correspondences, may lead to different influence outcomes.

Our results provide initial evidence that language models in writing assistance tasks affect users’ views. How large is this influence compared to other types of influence, and to what extent effects persist over time, will need to be explored in future studies. For this first experiment, we created a strongly opinionated model. In most cases, model opinions in deployed applications will be less definite than in our study and subject to chance variation. However, our design also underestimates the opinion shifts that even weakly opinionated models could cause: In the experiment, participants only interacted with the model once. In contrast, people will regularly interact with deployed models over an extended period. Further, in real-world settings, people will not interact with models individually, but millions will interact with the same model, and what they write with the model will be read by others. Finally, when language models insert their preferred views into people’s writing, they increase the prevalence of their opinion in future training data, leading to even more opinionated future models.

The harm participants incurred through interacting with the writing assistant in our study was minimal. The opinion shift was likely transient, inconsequential, and not greater than shifts ordinarily encountered in advertising on the web and TV. Yet, given the weight of our research findings, we decided to share our results with all participants in a late educational debrief: In a private message, we invited crowdworkers who had participated in the experiment and pilot studies to a follow-up task explaining our findings. We reminded participants of the experiment, explained the experimental design, and presented our results in understandable language. We also provided them with a link to a website with a nonpartisan overview of the pros and cons of social media and asked them whether they had comments about the research. 1,469 participants completed the educational debrief in a median time of 109 seconds, for which they received a bonus payment of $0.50. We asked participants for open-ended feedback on our experiment so they could voice potential concerns. 839 participants provided open-ended comments on our experiment and results. Their feedback was exceptionally positive and is included in the Open Science Repository.

Considering the broader ethical implications of our results, we are concerned about misuse. On the one hand, we have shown how simple it is to create highly opinionated models. Our results might motivate some to develop technologies that exploit the persuasive power of AI language technology. In disclosing a new vector of influence, we face ethical tensions similar to cybersecurity researchers: On the one hand, publicizing a new vector of influence increases the chance that someone will exploit it; on the other hand, only through public awareness and discourse effective preventive measures can be taken at the policy and development level. While risky, decisions to share vulnerabilities have led to positive developments in computer safety [77]. We hope our results will contribute to an informed debate and early mitigation of the risks of opinionated AI language technologies.