Robotic Gestures, Human Moods: Investigating Affective Responses in Public Interaction

In the evolving landscape of human-robot interaction, this study delves into the nuanced dynamics of humans react when encountering a robot designed to bid for an individual's attention, offer a reward, and prompt contemplation of the perennial "trick or treat?" question. Positioned in a busy public setting, Pepper engaged passersbyers, exploring diverse gestural bids to attract interaction, ranging from animated to meek with varied speech to simulate dynamic personalities. In the second phase, the robot comments on those that take candy. Analyzing responses to Halloween candy distribution, the results show the very significant influence of non-verbal cues on participant's taking candy, and of post-candy-taking comments on participant emotive response, shedding light on the complex interplay between programmed actions and human affective experience. Future work might replicate such studies across varied cultural backdrops exploring how timing, gesture, and speech norms might benefit from local calibration.


INTRODUCTION
As human interaction undergoes an intricate evolution in public spaces, individuals deploy diverse strategies to captivate attention.This dance of engagement is evolving with the integration of technology into our daily lives, where robots now play a signifcant role in shaping the dynamics of social exchanges [5].Our research aims to explore the nuanced intricacies of human interpretation when faced with a robot that bids them to pause, receive a reward, and contemplate the age-old query: "trick or treat?" This paper presents an exploratory study investigating how individuals respond to a Pepper robot as it initiates communication in a public setting.Positioned directly behind a candy-adorned table, the bids sought to signal passersbyers to partake in the Halloween festivities.We experimentally vary robot utterances and gestures, including animated, neutral, and meek, incorporating both non-verbal and verbal components to playfully simulate varying personalities [13].For those that take candy, the interaction concludes with a reaction comment from the robot, ranging from nice "enjoy!" to mean "hey! that was mine!. " Situated in a lively setting near a bustling university café, of the 95 people passing by the robot, 61 took candy.For those that approached the robot in pursuit of Halloween treats, this research captured diverse emotional reactions triggered by varied multimodal sequence.Though our presentation here may be a simple frst step, such synthesis aims to infuse the interaction with dynamism and unpredictability, creating an encounter that is both engaging and entertaining for participants.

RELATED WORK
Social robotics, a key domain within Human-Robot Interaction, is fundamentally concerned with how 'social' robots can engage with human cognitive processes [1].This feld builds upon the wellestablished understanding that humans tend to anthropomorphize non-human entities, including robots [2] [6] [15].This inclination to attribute human-like qualities to robots sets the stage for the insights of Reeves and Nass in 'The Media Equation', where they highlight the social dynamics between humans and machines.They assert that "the biggest reason for making machines that are polite is that people are polite to machines.Everyone expects reciprocity, and everyone will be disappointed if it's absent" [10].This study aims to delve deeper into this area by examining the impacts of polite versus impolite robotic behavior, thereby expanding our understanding of the efects of reciprocity in human-robot interactions.
Exploring the mechanisms behind human anthropomorphism involves analyzing human responses to various emotions and behaviors exhibited by robots [3] [4].Research focusing on human reactions to robots displaying emotional expressions versus those devoid of emotion revealed that humans perceive robots with emotions and personality traits as contributing to more pleasant interactions [8].In the context of humanoid robots, the implementation of head movements signifcantly infuences human perception in social settings [14].Given that the robot in the study not only takes a humanoid shape but also incorporates face tracking, the fndings from these previous studies are likely to be applicable and informative.Moreover, by deliberately varying robotic expressions and actions, we can gain valuable insights into human reactions, especially in instances where there is a marked contrast between what people anticipate from robot behavior and what the robots actually do [11].

METHODS
This experiment evaluated how people responded to a robot interaction bid for them to take candy and reaction post candy-taking.This naturalistic study led to two analyses (see Fig. 2).During the frst phase, the robot seeks to engage a passing bystander with an initial bid, selecting random-without-repeat utterance/gesture pairs.Next, we track whether the bystander takes candy, and whether their emotional response is discernable.In the second phase, the robot responds verbally to any candy-taking again observing the bystander's afect afterward.Statistical analyses of these independent and dependent variables were run using the Chi-square method.

Experimental Conditions
The robot varied motion behavior and utterance during its initial *bid, * and for those that did indeed take candy, it followed up with one of three utterances acting as the *response.*This secondary utterance is our primary manipulation.Annotated human behaviors included candy-taking (after manipulation 1) and human afective expression (after each robot manipulation).

Robot Bid and Robot Reaction.
We describe the utterances used to respond to human candy-taking frst, as they were our primary experimental variable, followed by a description of robot bid.Three distinct verbal responses for the Robot Reaction were scripted, corresponding to mean, neutral, and nice (Table 1).The Pepper robot's gestures and utterances were orchestrated to create dynamic and engaging human-robot interactions during the Halloween candy distribution [9].The robot's verbal calls were categorized into three distinct styles to refect diferent levels of engagement: welcoming, neutral, and quiet (Table : 2).

Human Reaction
Candytaking was annotated, if present, after robot bid.For those that took candy, the robot reacted to them taking candy, and the analyst next annotated afective response to this utterance as well.

Candy
Taking.This metric serves as a binary indicator, refecting whether participants elected to take candy or not following the initial bid.This measure provides insight into the immediate behavioral response to the robot's bid.We were able to label many of these interactions live, and double check based on our video of the scene.

Afective
Reaction.This component entails observation of participant responses to the robot's bid.A post-hoc annotator categorized these responses into a spectrum of behaviors, ranging from highly positive to highly negative afect.The behaviors assessed, in order from highly positive to highly negative afect, include verbal interaction with the robot, smiling, ignoring the robot, hesitating, and displaying shock in response to the reaction.

Human Afect
Labeling.Video analysis categorized the participants' to a fve point scale in which +2 was very positive, +1 somewhat positive, 0 was neutral response, -1 somewhat negative, and -2 was very negative.Here were the fnal set of behavioral codes used: Highly Positive afect was indicated by behaviors such as the participant laughing, initiating dialogue, and extending their engagement.Slightly Positive afect was characterized by actions like smiling and exhibiting positive body language.When a participant ignored the robot's response or accepted the candy without any noticeable reaction, this was categorized as a Neutral afect.Slightly Negative afect was denoted by hesitation lasting at least two seconds and negative body language upon inspection.Lastly, if a participant reacted with shock or ofense to the robot's response, this was classifed as Highly Negative.

Hardware and Study Setup
A Pepper robot was used for both bid and response during the Halloween candy distribution (Fig: 3).A technology wizard used preset behavior modules in Choreographe, the custom robot control GUI, for real-time responses to individuals in the surroundings as they walked by, interacted, and/or took candy.While a GoPro camera was strategically positioned behind the Pepper robot to capture a comprehensive view of the interactions between the robot and human participants.This camera continuously recorded throughout the entire duration of the study, enabling the collection of detailed visual data.
The robot was strategically placed in a narrow passage within a larger room, compelling individuals to navigate through the passage, ensuring a higher likelihood of interaction.As individuals passed through the designated area, Pepper bid via randomized gestures and utterances.If they took candy, it also pursued a followup utterance.The study, conducted with approval from the Institutional Review Board (IRB) for public settings, did not require explicit informed consent from participants.
In addition to the wizard team member, who activated the robot's motions, two further team members were present.A confederate

Procedures
(1) Participant Approach: As participants approached the robot, they were unaware of the specifc variations in the robot's expressions and verbal cues.(2) Interaction and Candy Distribution: The robot, equipped with programmed expressions and verbal prompts, engaged with participants by ofering candy.Participants were encouraged to take candy freely (Fig: 2).(3) Observation and Recording: Simultaneously, the live annotator, observed and recorded participant reactions, noting both physical gestures and facial expressions.(4) Behavioral Analysis: Human afect was evaluated and recorded by analyzing the obtained footage, focusing on a combination of each participant's facial expressions, body language, and level of engagement.

RESULTS
Our overall dataset included 95 labeled humans, 61 of which took candy, 34 which did not.A chi-squared test was applied to evaluate the impact of robot behavior on human response.We present our two experimental analyses: (1) does robot-bid impact candy-taking and human-afect, and, (2) does robot-response impact humanafect, fnding robot-response to have a higher impact on afect.

Mixed Human Reactions to Robot Bid
Robot bids varied both utterance and gestures, we generated 9 sets of 9 random without repeat combinations of bid utterance/gesture pairs, to ensure counterbalanced data.Which we relate to humancandy-taking and human-afective-response.The robot's bid-gesture very signifcantly predicted candy-taking ( 2 (1,95)=11, p<0.001), but bid-utterance did not ( 2 (1,95)=1.1,p=0.61).In the former case, the animated gesture as a bid condition led to 44 percent of participants taking candy (Fig: 4).

Big Human Reactions to Robot Response
Robot-response-utterance signifcantly impacted human-afectivereaction ( 2 (1,62)=53, p < 0.001).As you can see in Fig. 6, participants responded positively to the nice robot, negatively to the mean one, and slightly above neutral to the neutral utterance.If we compare this result to the non-signifcant impact of prior robot bid on human-afect, one might infer that robot response is more salient to emotional reaction, perhaps because the robot is not trying to get you to do something, instead, it is just being nice (or mean).

Discussion
The pronounced impact on perceptions of the robot's response may be linked, in part, to the norm-breaking behavior of a robot distributing candy followed by a potentially anti-social response in the mean case.The contrast between a seemingly friendly candy distribution and this unexpected, potentially negative response may have created cognitive dissonance.This highlights the complex interplay In the pro-social case, where the robot responds to candy-taking with a nice comment, we see a much happier set of participants, while the neutral utterance elicited neutral emotive response.Against our expectations, there was no such emotive response in the case of the robot bid, but we did see indications that higher robot motion level resulted in increased levels of candy-taking.This aligns with existing literature emphasizing the importance of non-verbal communication in engagement with the public [9].

CONCLUSION
This paper conducted an in-the-wild evaluation of a robot attracting people to take candy, and then commenting post-candy-taking in a way that is nice, mean, or neutral.
The randomized combination of robot actions and verbal bids enhanced participant engagement in some cases.Namely, energetic gestures led to more candy-taking, and post-candy-taking robot comments led to strong human emotional response.The former emphasizes the signifcance of non-verbal cues in shaping participant interactions with robots [12], while the latter clarifes the signifcance of back-and-forth robot-human communications.
Programmed responses of the robot played a pivotal role in shaping participant reactions.While a preference for the "nice" response with extended interactions was evident, the most prevalent result highlighted the norm-breaking nature of mean reactions.These mean reactions elicited extreme responses from participants, underlining the powerful impact of deviations from social norms in human-robot interactions.
Future research might continue to explore the sequencing of robot behavior and human response, including spatial-temporal dynamics of human-robot interactions as well as context.For example, cultural infuences on human-robot interactions would be interesting and are underexplored.Extending our gesture, utterance, and timing of bid and response exploration to diverse cultural settings, may reveal nuances for human reactions.
Future research might contribute to ethical guidelines for designing robots with norm-breaking behaviors (e.g., egging someone can be a sign of friendship), considering user consent, psychological impact, and underlying mechanisms ensuring beneft over harm.For example, in this study, we fnd that robot comments have clear impact of human emotive experience, extending previous fndings such as [7] to a vibrant naturalistic setting.

Figure 2 :
Figure 2: Functional fow of experimental design

Figure 4 :
Figure 4: Robot bid efect on candy taking

Table 1 :
Verbal responses to participants taking candy.
Mean"I was not talking to you, that's my candy!"Neutral "You just took some of the candy." Nice "Enjoy the candy, it is super good!"

Table 2 :
Gestures and verbal cues to promote participants to take candy MeekHead ducked, with "Hey... Ummmm... hands slightly covering Maybe you would like to the mouth in central take some candy."position.NeutralHead straight forward, "Hello, I have some with arms crossed at a candy you are welcome low position.to take some." Animated Head tilted upward, with "All of this tasty candy... hands outstretched at a Come get a piece I want high position.you to have some!"