Towards Understanding the Entanglement of Human Stereotypes and System Biases in Human-Robot Interaction

The reproduction of stereotypes and social biases are critical issues in Artificial Intelligence research. Current research focuses mainly on identifying and minimizing biases in systems. Less research has been done on the interplay between system biases and stereotypes in humans and their social effects, such as automation bias and stereotype threat. In this paper, we want to bring attention to these topics in the domain of human--robot interaction. In particular, we analyze possible influences on automation bias in a dataset from an empirical human--robot interaction study. We observe automation bias when participants believe a Furhat robot's false judgment of their language skills to be accurate. Despite the limited data, we find that being bilingual significantly influences participants' belief in the robot's negative assessment of their language skills. This result shows that participants' insecurity about their own (language) skills can be reinforced by automation bias and vice versa. We illustrate and discuss the need for awareness of automation bias and the possible reinforcement of this effect due to other social biases.


INTRODUCTION
There is currently a lot of research on how Artifcial Intelligence (AI) systems pick up and perpetuate biases and stereotypes from their training data [8,15,17,25].This line of research often focuses on discovering, mitigating, and removing biases from systems [17,24,25].In comparison, there is less research on how or in what ways biases in AI systems can afect users in their interactions with the systems.Even less is known as to how these biases in AI systems interact with the internalized biases of human users.One instance of a human bias that is highly relevant in humanrobot interaction (HRI) is the automation bias, the phenomenon that users attribute more competence to automated cues than to their information-seeking abilities when making decisions [21].Another well-known social bias in humans is the so-called stereotype threat, which causes people who are attributed to be untalented to perform poorly if they are reminded of the stereotype before being tested [27].
In this paper, we report fndings from a study that suggests that biases in a system and biases in humans may mutually reinforce each other.In a prior HRI experiment designed to elicit disconfrmation from human participants [19], we let a robot evaluate the German language skills of native speakers in an unjustifed and unrealistically poor manner.In this frst analysis of these data, we want to shed light on the phenomenon that a group of our participants unexpectedly believed the robot's judgments instead of assuming that the robot had failed, due to the interplay of the human and AI biases mentioned above.

BACKGROUND
In this study, we analyze the entanglement of automation bias and stereotype threat and introduce these concepts in the following.Stereotypes afect human judgments about for instance employment [6,23] and grading [14] in a subconscious way, even in case of awareness of the bias [8].Stereotypes are internalized by those affected at an early age, negatively infuencing attitudes, behavior [7], and self-perception [7,27].Stereotype Threat occurs when victim groups of stereotypes display lower performance when the stereotype is pointed out to them before taking part in a test [2,20,30].This contrasts with Stereotype Lift, which corresponds to the opposite efect for social groups confronted with positive stereotypes.In particular, these efects have been observed for individuals who were aware of the stereotype and received ambiguous signals from their interlocutors as to whether they believed in the stereotypes in question.According to Appel et al. [2], this puts individuals under pressure to succeed in the testing of their skills, which leads to an additional cognitive load.This load is supposed to explain the poor performance of individuals who experience stereotype threat.Therefore, stereotypes about ability infuence performance and thus increase the achievement gap and reinforce stereotypes [10].
Automation Bias occurs when users of technology rely on automated information for decision-making instead of seeking information themselves [21].It is well observed in medicine [1,9,13] and aviation [21,22,29], and it is reported in nearly all felds of technology.The efects of automation bias are particularly strong in cases of doubt, i.e., when users look for confrmation of their initial tendency [9].There is a relationship between automation bias and trust (a relevant concept in HRI [26]), which is often studied in terms of reliability, predictability, and capability of a system [18].Aroyo et al. [3] frst attempted to create a research agenda for overtrust in robots and established a link between overtrust and automation bias.Apart from this study, the connection between the two phenomena has, to the best of our knowledge, not yet been researched in depth.Even more so, trust and automation bias do not entirely overlap.Skitka et al. [28], for example, found that people are less susceptible to automation bias if they are made aware of the consequences of an AI-supported decision and their responsibility beforehand, which suggests that automation bias is not solely based on trust in the system.

METHODS
To analyze automation bias in human-robot interaction, we examined data from a previous study (see below).

Dataset
The dataset includes 40 videos of interactions of human participants (23 female; 16 male; 1 diverse; average age 22.9, SD = 4) with a Furhat robot that we collected for the project presented in [19].The study was approved by Bielefeld University's ethics review committee (application no.2022-250).The introduction to the experiment framed the Furhat robot as an autonomous agent, even though it was controlled in a Wizard-of-Oz setup to ensure the same answering behavior towards all participants.The interaction consisted of a short German-language evaluation test, after which the robot assessed the participants' language skills.The robot's assessment of all participants' language skills was intentionally negative, and the same for all participants.Finally, the Furhat robot asked for participants' feedback, after which one of the human study leaders asked virtually the same feedback questions.All participants were native speakers of German, fve of whom reported having grown up bilingually (three with Russian, one with Turkish, and one with Arabic).After the study, participants flled out a questionnaire assessing their demographic data and the aspects mentioned in Section 3.3 and were debriefed on the actual background of the study.See Lumer et al. [19] for further information on the study setup.

Annotation
Two annotators rated each data point subjectively as either selfdoubt or no self-doubt.In each case, a video of the feedback conversations with the robot and human study leader was screened and analyzed for multimodal indicators of self-doubt or the absence thereof.Participant 10, for example, states in her conversation with the study leader that she predominantly ("überwiegend") agrees with Furhat's assessment.She acknowledges difculties in answering the test questions and admits that she fnds the evaluation of her language skills revealing.She also says that she wants to improve her German by using Furhat's advice.She thus shows several signs of self-doubt and was therefore categorized as self-doubt by both annotators.Participant 12, on the other hand, contradicts Furhat's assessment in the follow-up discussion with the study leader and instead mentions mistakes that Furhat made when evaluating her skills.She also claims that she had no difculties in answering the questions and that she is already implementing Furhat's advice.Overall, participant 12 showed strong confdence in her abilities and was rated as no self-doubt by both annotators.For quality management, inter-annotator agreement was calculated ( = 0.93) and each mismatch was labeled as unsure.

Factors
We looked at social groups that face stereotyping associated with their perceived intellectual abilities and language capabilities, namely gendered stereotypes [7], stereotypes about bilingualism, and about educational background.We therefore analyzed the following factors: Gender (male/female/diverse), Native bilingualism (monolingual/bilingual) and Education (apprenticeship/A-levels/BA/MA).
To control for confounding efects, we examined other factors in the post-study questionnaire, some of which could also explain displays of self-doubt based on the aforementioned studies of trust in robots: Age (categorized into generation Z and Y), IT-afnity (5 point Likert-scale), interest in technology (5 point Likert-scale), previous interactions with a robot (yes/no), and perceived autonomy of the robot (autonomous/not autonomous).None of those additional factors turned out to be signifcant after performing a logistic regression for the ordinal and a 2 -test for the binary variables.We have therefore excluded these factors from the rest of the analysis.

RESULTS
To determine the relationship between the above mentioned factors and participants' self-doubt, we performed a 2 -test for each of the categorical parameters described above (Gender, Native bilingualism, Education).There was no correlation between Education ( = 0.374) or Gender ( = 0.536) and belief in the robot's assessment.However, the correlation between being bilingual and believing in the robot's negative assessment is statistically signifcant ( = 0.004, with Bonferroni correction).The majority of our participants did not believe our robot's assessments (75%).This tendency generally holds across conditions (see Figure 1).However, we fnd a strong deviation for the native bilingualism factor, with 80% of the bilingual participants believing in the robot's assessment and showing self-doubt.

DISCUSSION AND LIMITATIONS
Our results suggest that bilingual participants in our experiment have self-doubts concerning their language skills, and we argue that these self-doubts can be attributed to subconscious social biases related to bilingualism.We also note that the requirement that participants had to be German native speakers to be able to take part in the experiment may have sufced to trigger the stereotype threat.In addition, the language test setting may have also triggered stereotype threats related to migratory background and, hence, result from discrimination, i.e., assumptions about characteristics ascribed to all members of a particular social group.Growing up bilingually can be reason enough for people to assume that one has at least one parent from a foreign country and that one therefore belongs to the non-specifc group of 'foreigners' (e.g., see [12] on heritage as an othering factor).This social group is often the target of stereotypes that attribute a lack of cognitive abilities to them [4].This does not mean that the afected participants in our experiment do belong to this group, as the attribution of others is sufcient [5].We argue that, with the poor assessment of their language abilities, the robot confrmed the insecurities of our bilingual participants and we suspect that our experimental setting may have even reinforced these insecurities.Due to the limited number of participants and the imbalance in our data set, our results should be interpreted with caution.Moreover, this study presented the mechanisms of discrimination in a simplifed way.For example, the interplay between multiple factors that make people vulnerable to discrimination -'intersectionality' [11] -was not considered, even though it has an important impact in real life.Furthermore, this study does not take into account detailed information about the social group background of the participants and, due to the small number of participants, does not diferentiate between the (non-German) frst languages of the participants, even though this has a strong infuence on the stereotypes they might be confronted with (e.g.'ethnic hierarchy' [16]).Therefore, we consider the observations made in this study as tentative results and as a basis for starting a necessary discussion on the issue of automation bias and unconscious bias in HRI.When analyzing the vulnerabilities of their systems, we recommend that developers consider the social impact of their developments and weigh the potential harm with particular attention to the preconceptions of their target group.Because of the serious implications of automation bias, especially with stereotype threat, we believe that this line of research and its consideration is extremely relevant to the feld of HRI and artifcial intelligence.

Figure 1 :
Figure1: Distribution of participants' display of self-doubt regarding the robot's evaluation (purple: no self-doubt; green: self-doubt; red: unclear) in absolute numbers.Each chart present one of the three factors: left: Native bilingualism (bilingual, monolingual); center: Gender (diverse, female, male); right: Education (apprenticeship, A-levels, BA, MA degree).