A First Look into Targeted Clickbait and its Countermeasures: The Power of Storytelling

Clickbait headlines work through superlatives and intensifiers, creating information gaps to increase the relevance of their associated links that direct users to time-wasting and sometimes even malicious websites. This approach can be amplified using targeted clickbait that takes publicly available information from social media to align clickbait to users’ preferences and beliefs. In this work, we first conducted preliminary studies to understand the influence of targeted clickbait on users’ clicking behavior. Based on our findings, we involved 24 users in the participatory design of story-based warnings against targeted clickbait. Our analysis of user-created warnings led to four design variations, which we evaluated through an online survey over Amazon Mechanical Turk. Our findings show the significance of integrating information with persuasive narratives to create effective warnings against targeted clickbait. Overall, our studies provide valuable insights into understanding users’ perceptions and behaviors towards targeted clickbait, and the efficacy of story-based interventions.


INTRODUCTION
Clickbait is a text or a thumbnail link designed to attract attention and entice users to follow that link to visit the linked piece of online content, which is typically deceptive, sensationalized, or otherwise misleading [127]. 1 Even worse, clickbait is often used in social engineering attacks, tricking users to click on posts or links that direct them to malicious websites [2,10,38,107,116,118,139,165].Since clickbait threatens online security, there have been many attempts to limit it both from industry eforts [20,67,90,143] and academic research [18,72,129,147].Clickbait, however, is still efective [11,33,98].This can be explained through relevance theory [140,141], which explains that humans are driven to take in the most relevant information.Clickbait uses defnite referring expressions together with superlatives and intensifers to create an information gap, which drives the reader to click on the associated link with the expectation to fnd relevant information [127].
Relevance theory further states that a stimulus (clickbait) will be optimally relevant if it aligns with the reader's preferences [140,141].That explains why clickbait on topics that a user is not inherently interested in may not be efective.Unfortunately, with the wide use of social networking sites, public information about users, including their afliated institutions, interests, and even their friends, are readily available [66,79].Users can be targeted on social media with posts inherently relevant to them using their public information [7,79,85].Relevance theory would suggest that this kind of targeted clickbait should be even more efective than non-targeted clickbait.
Therefore, it is important to frst understand the public information that could be used to create targeted clickbait (targeting factors) and the countermeasures that could protect against it (RQ1).
To address these initial needs, we conducted a focus group discussion (FGD) with six participants and an online survey with 30 participants in North America (see Figure 1a), which focused on the following: i) brainstorming targeting factors, ii) brainstorming ideas for (countermeasures), and iii) selecting the most efective targeting factors and countermeasures.
Based on the fndings from these studies, we developed storybased interstitial interventions against targeted clickbait by involving 24 end-users in a participatory design study (see Figure 1b).In this study, we focused on interstitial warnings that interrupt users' primary tasks, considering their proven efcacy to protect against phishing and malware [72,117].Our focus on storytelling is due to the efectiveness of this approach in persuading users [37,91,137]. 2eople often engage in risky behavior despite knowing the potential harms [28,50], but storytelling could be a powerful tool in changing such behavior [54,71,137].Within the framework of story-based interstitial interventions, participants refected on a key remaining question: what stories should we tell?In other words, our study answers, "How can we leverage user-informed stories in designing interstitial interventions to protect users from targeted clickbait?"(RQ2).The existing literature [17,101] suggests that involving end users from an early stage results in improved designs.To this end, we leveraged participatory design [45,59,155], where participants were facilitated with graphics to create their version of the warning.
The fndings from our participatory design study provide us with concepts that users believe should guide the story in a warning against targeted clickbait.These concepts then inform our fnal warnings that we evaluated through an online survey with 114 participants (see Figure 1c).The study answers, "How do social media users perceive and behave towards user-informed storybased interventions designed to help them make informed decisions about targeted clickbait?"(RQ3).The results from this evaluation survey indicate that user-informed stories can efectively support social media users to understand and counter clickbait.
In summary, our fndings frst contribute to the identifcation and taxonomy of targeting factors in social media and the countermeasures against targeted clickbait.Based on these targeting factors, our studies reveal the efectiveness of targeted clickbait and lead to the creation of story-based interventions based on user participation through ideation and design.Finally, our work evaluates the designs that emerged from these eforts, and fnds them to be efective against targeted clickbait.To the best of our knowledge, this is the frst systematic exploration of targeted clickbait, its efcacy in tricking users, and the design and evaluation of potential countermeasures through involving end users.Taken together, our studies provide valuable insights into users' perceptions, needs, and behavior around targeted clickbait, and the user-informed stories designed to protect them from clicking on a clickbait.We provide a set of recommendations based on our fndings, which include using stories interchangeably to avoid habituation, and adapting warnings based the level of threat.

BACKGROUND AND RELATED WORK
Users have been found to be vulnerable to the misleading and sensationalized information provided in a clickbait [53,65,[111][112][113]145].Relevance theory suggests that targeted clickbait would be even more efective [21,126,127].We thus provide background on the working and potential of targeted clickbait in tricking users at the beginning of §2.1, followed by a discussion on our motivation to curate targeting factors (RQ1).In §2.2, we discuss prior studies on clickbait that encourage us to explore warning-based interventions (RQ2 and RQ3).Lastly, §2.3 highlights our respective rationales behind using interstitial, story-driven, and user-informed warnings (RQ2 and RQ3).

Targeted Clickbait and Targeting Factors
Relevance theory is based on two main principles: the cognitive principle and the communicative principle [32,140,141,153,154].According to cognitive principle, humans are geared to maximizing relevance [32,140,141].Clickbait uses defnite referring expressions (e.g."These were Chávez' last words" use of "These" in place of the words themselves) [21,126], and superlatives (e.g.terrifying, coolest, genius, unreal) or intensifers (e.g.ridiculously, crazy, "THIS") to create relevance for its readers [126,127].Prior work [89] revealed that these elements contribute to an information gap by encouraging readers to construct new conceptual fles based on the terms used in a headline while providing little or no content for those fles [127].The instance of an information gap can be seen in a headline such as, "The worst [superlative] day of the week [information gap by hiding simple information] to eat at a restaurant".The information gap then drives a reader to click on the associated link, expecting the article to contain relevant information [127].
In contrast, regular clickbait does not align with the communicative principle, which states that input will be optimally relevant if it is (a) worth the reader's efort to process and (b) the most relevant input allowing for the reader's abilities and preferences [32,140,141].Online communication often occurs in a so-called collapsed context [92,151].When ofine, we speak to one group in one context, but when online, we communicate across groups of people and contexts [92,127,151].That is true for clickbait creators, as they cannot know who will see and read their work, or when it might be read [127].This is becoming less true all the time, however.Users often make large amounts of information about themselves available on social media, either publicly or to only loosely defned friend groups, including interests, location, job type, relationship status, and associated institutions.This wealth of information makes it possible to align clickbait to users' preferences and beliefs and thus ensure it is optimally relevant [7,66,79,85,140,141]. Advancement in artifcial intelligence has made it easy and inexpensive to access high-quality text generation technology to automatically select and produce even more relevant headlines and material based on this information.Furthermore, faceswap and other deepfake image generation technologies are also becoming easier and cheaper to access.A malicious actor can use these tools to weaponize publicly available information about friends to create even more compelling targeted clickbait with pictures and video [13,30,49,52,76,77,129,144].As targeted clickbait more strongly aligns with users' beliefs and interests, they may feel that it is worth their efort and time to click on the post and discover the answer [7,85,127,140,141].
There is a gap in existing literature to understand targeted clickbait and countermeasures against it.We found a few studies [105,121] in the realm of targeted attacks in general, which mentioned location, friends, relatives, and afliated institutions as the examples of targeting factors.However, we found a little work that has listed and ranked the targeting factors for targeted clickbait.To this end, we identifed and ranked targeting factors and countermeasures against targeted clickbait (RQ1).

Learning from the Studies on Clickbait
Since there are no studies on targeted clickbait, we look into the studies about clickbait in general to begin the process of creating countermeasures against targeted clickbait.Because clickbait is efective [55,119,120], prior works aimed at mitigating the problem, where most of them focused on the moderation of clickbait in social media [3,27,29,75,81,115,134,160,162,163].Moderation often sufers from issues, however, including reliability [35] and false positives/negatives [75] that lead to blocking a substantial amount of non-clickbait posts or vice versa.It is also difcult to estimate how efective the detection and moderation would be for targeted clickbait.While clear cases of clickbait and known malicious sites should be blocked, we argue that less obvious cases can be combated by supporting users to understand the risks in real time.
Only a few works have aimed at supporting end users through interventions to inform and persuade them to avoid clickbait [18,19,27,53,61,72,147]. Bhuiyan et al. [18,19] developed browser extensions to nudge users to refect on the credibility of news in social media.A few studies used interventions through identifying clickbait and misinformation, and letting users to block such posts [27,43,85,125].These studies show some promise in countering clickbait, highlighting the importance of informing and warning users about targeted clickbait.Therefore, we designed and evaluated warning-based interventions (RQ2 and RQ3).

Refection, Stories and Users
Warnings can generally be classifed into interstitial (blocking) and contextual (non-blocking).Prior work [44,72,109,117,158] found that users routinely ignore contextual warnings such as banners or pop-ups.They instead notice interstitial interventions that interrupt the user's primary task, allow them to refect on their actions and respond by seeking information from alternative sources [16,44,72,117,158].Here, the refective design promotes conscious thought and decision-making, and helps users consider their actions [48,[101][102][103].The Psychology and Marketing literature [12,62,87,114] support that refective designs help increase engagement and thoughtful decision-making.The study of Kaiser et al. [72] further informs us that interstitial warnings can efectively inform users about the risk of harm.Therefore, we designed and evaluated interstitial warnings (RQ2 and RQ3).
We combine interstitial warnings with storytelling, leveraging its power to persuade users to avoid targeted clickbait [37,51,64,78,86,91,137].The prior work [28,50] referred to cognitive dissonance, a phenomenon representing that even when users know something is bad, they tend to do the action.For instance, people often do not quit smoking despite knowing it can cause lung cancer.In such cases, strategic story, also called narrative persuasion, is a powerful tool that combines relevant information with emotion [37,54,63,71,84,99,137].It changes the persuasive message from a dry listing of information to something that is embedded in a larger narrative, where we get the message by seeing people's stories unfold [37,54,63,78,91,99,137].Therefore, we leveraged the power of stories to persuade users to avoid targeted clickbait (RQ2 and RQ3).
We focus on a participatory design approach to create stories for the warnings [14,45,59,122,155].Designers often overlook the intricate difculties and challenges end users may face [17,42,70,100,101,152].Prior studies [100,101,152] suggest that users are not just the target audience for a design but a collaborative party holding knowledge about its development.These studies also highlight the importance of understanding the perspectives and needs of users from an early stage to facilitate concept generation, as well as increase the adoption of a design [17,31,70,100,101,152].Therefore, we involved end users from the early stages of our design ideation.
While prior studies primarily focused on detecting and moderating regular clickbait, targeted clickbait is yet to be studied.To the best of our knowledge, our work is the frst to understand user perspectives towards targeted clickbait, involve users from selecting targeting factors to designing story-based warnings, and evaluate user-informed story-based warnings.

TARGETING FACTORS AND COUNTERMEASURES (RQ1)
In §3- §5, we present a series of studies and corresponding fndings.A summary of the fow of these studies along with its goal and outputs are presented in Figure 1.For consistency, we use these terms based on the frequency of participants' comments in each study: a few (0-20%), some (21-40%), about half (41-60%), most (61-80%), and almost all (81-100%).
Due to the novelty of targeted clickbait and a lack of prior work in this area, we frst need to understand the factors that can be used to create targeted clickbait.Considering possible cognitive load and confusion of participants caused by having too many study variables [34,74,93,96], we broke this process into two studies (see Figure 1a) to curate targeting factors and countermeasures that align with participants' beliefs and interests [60,104,136,138,149].

Study I: Brainstorm and Select FGD
We conducted a Focus Group Discussion (FGD) [59,80,97] with six participants (FGD1 -FGD6) -including three User Experience (UX) experts -to brainstorm and rank ideas about both targeting factors and countermeasures.The group setting provides a platform to generate ideas and refne them in a single session through collaboration.Table 6 in Appendix B provides the demographic information of our participants.
3.1.1Methods.We recruited participants through snowball and convenience sampling, where we contacted them via email.The FGD was conducted in person (audio-recorded), and lasted around 75 minutes.The session started with brainstorming targeting factors in social media.Then the participants rated each idea from 1 (not likely to click) to 5 (very likely to click) while providing their rationales.Finally, they were asked to discuss potential countermeasures against targeted clickbait.
We ranked the targeting factors based on participants' ratings, and listed the countermeasures.We transcribed the audio recording and extracted the rationale behind participants' ratings using thematic analysis [15,22,40].Two researchers independently coded the transcript of the focus group, where they read through it and developed codes.Then the codes were compared, and the coders discussed and resolved any discrepancies in the codes.

Findings.
Selection of Targeting Factors.Our participants came up with 12 targeting factors through brainstorming, which we ranked using their ratings.We then selected the following fve factors with an average rating greater than the median.
(  2a and 2b).Since targeted clickbait increases relevance and feeds on users' emotional reactions [29,159], we aimed to examine if the change in emotional valence impacts the efectiveness of a targeted clickbait.For the countermeasures, we presented the information in text on top of the post with a red overlay and a generic header (see Figure 2c) aimed at selecting the best idea for countering targeted clickbait.

Study II: Curation Survey
We selected the two most efective targeted clickbaits that align with users' preferences, and in turn, increases relevance [140,141].Similarly, persuasion, which is the basis for behavior change, starts from the values, beliefs, and motives of users [60,104,136,138,149]. Therefore, we selected the two most efective countermeasures to align with users' values and motives.The Institutional Review Board at our university approved the study.

Methods.
Based on the designed targeted clickbait and countermeasures, we conducted a survey with 30 participants.A power analysis indicated that 30 participants would provide a large effect size, which can only detect efects that apply to at least 80% of the population [58,59].This efect size is appropriate for this study, given that the goal is to fnd directions for the next steps: participatory design and evaluation.
Participant Recruitment.We recruited participants over Amazon Mechanical Turk (MTurk).Following the guidelines from prior work [82,110], we recruited participants with a 99% HIT approval rate 3 to increase the quality of responses in our study.As each design was presented on a separate page with only a single question, i.e., participants did not need to go through multiple questions in a single page, we felt it was reasonable to not include any attention check questions.Participants had to be 18 years or older and live in the United States or Canada to participate in our study.We set the location of target participants as United States or Canada on MTurk, and we confrmed this by geolocating their IP address as recorded by the Qualtrics survey platform.The study took between 10 and 15 minutes to complete.Each participant was compensated with USD $2.00.Table 7 in Appendix B shows the demographic information of our participants.
Procedure.To start, participants were presented with an Informed Consent Document (ICD).After agreeing to the ICD, participants were shown a scenario where they encountered a targeted clickbait.An example scenario for face-swap clickbait was: "Imagine you are a friend of the person shown below [a picture of person is shown].While browsing through social media, you encounter the following post [a face-swapped post using the friend is shown]." We used scenarios in our study, as they are a powerful tool to help participants in imagining their interaction with targeted clickbait [26].
Each participant was shown fve scenarios (one for each of the fve targeting factors) and ten targeted clickbait (positive and negative for each factor).Participants were asked to rate the likelihood of clicking on each clickbait on a fve-point Likert scale (1: Extremely unlikely, 5: Extremely likely).Afterwards, they provided rationale for their choice of the most efective targeted clickbait.A similar process was followed for the countermeasures, where participants rated their usefulness and provided their rationale.At the end, they responded to a demographic questionnaire.
Analysis.We used statistical tests to analyze our quantitative results.We consider the result to be signifcant when we fnd p<.05.While comparing two conditions, we used a Wilcoxon signed rank test for the matched pairs of subjects.Wilcoxon tests are similar to t-tests but do not assume the distributions of the compared samples, which is appropriate for our collected data.For the qualitative results from the open-ended questions, we performed a thematic analysis, where two independent researchers coded the responses and later discussed and resolved the discrepancies in the codes.The inter-coder reliability was 91.6%.
Efectiveness of Targeted Clickbait.Our fndings reveal that users were most likely to click the targeted clickbait using the face-swap of a friend, followed by the one using afliated institution (see Table 1).Thus, we selected these for the next phase of our research.We found that users were signifcantly more likely to click on faceswapped posts than those using location or feld of study/work (see Table 3).However, there were no signifcant diferences between  face-swapped posts and the ones using afliated institutions (W=58, p=.21) or niche activities (see Table 3) .
From the open-ended responses, we see that about half of the participants believed that the personal nature of the face-swap post would pique their interest.One of them noted, "Most are familiar clickbait formats that I would ignore regardless of how well tailored they are to my interests.Seeing a friend's face in clickbait is something I have never experienced before.If it were a real story, I would be fascinated.If it turned out to be fake, I would be very upset and add that to the list of dirty tricks I try not to fall for."Similarly, some participants believed that the afliated institution post would grab their attention, as one explained, "As an avid Utah State University football fan, I would be very concerned if one of the athletes' health were threatened, and I would like further information about it." A summary of the taxonomy of targeting factors is provided in Table 8 in the Appendix.

Usefulness of Countermeasures.
For the countermeasures, we found that conveying consequence was perceived as the most useful way to prevent users from clicking on a targeted clickbait, followed by reporting the post as misleading (see Table 2).Both of these methods were rated signifcantly better than the approaches of revealing the mystery and providing alternate sources (see Table 4).Therefore, we selected conveying consequences and reporting posts as misleading as the countermeasures for participatory design study.
Based on the open-ended responses, we observed that about half of the participants found conveying harm efective due to the fear appeal.One participant mentioned, "Because I do not want to get a virus.I also do not want to support some scammy website that does such a thing.Finally, I do not want to waste my time on clickbait."Some participants highlighted the efectiveness of reporting posts as misleading due to the social factor associated with it.One of them mentioned, "This one is giving real-time crowd-sourced information that others have reported this [the post] as misinformation.That makes me think that I want to fnd another source to fnd out about this local news event."

PARTICIPATORY DESIGN (RQ2)
We conducted the participatory design study with 24 participants (P1-P24) with the goal of generating story-based warnings for evaluation (see Figure 1b).Here, we used the curated countermeasures (see §3) to create design tasks in our participatory design study.Through these design tasks, participants created story-based warnings that we leveraged to generate concepts for designing the warning.We later address how we evaluate the warnings through an online survey in §5.
We created two variations of the design task, each representing a goal for story-based warning: conveying consequences (Harm Design) and reporting posts as misleading (Report Design) -the countermeasures selected from the curation survey.These design goals align with behavior change persuasion theories (see Appendix A for further details on theoretical framework used to develop our design tasks).In each design task, participants were given fve categories of components: overlays, headers, navigation (buttons), messages, and components for expression (see Figure 3).Overlays, headers, messages, and navigation are based on prior studies that point to essential components in interstitial warnings [44,72,117,158].

Methods
We recruited participants via email and by sharing the study information with various university departments.Participants had to be at least 18 years old to participate in this study.We had 13 women and 11 men as participants, aged between 18 and 44.Table 9 in the Appendix shows the demographic information of our participants.
We conducted the study over Zoom (audio-recorded).When a participant showed interest, we emailed them the Informed Consent Document (ICD), which they agreed to before we scheduled a time for a Zoom session.The Institutional Review Board at our university approved the study.
Procedure.First, participants were given an overview of the study.Then they were randomly assigned one of four targeted clickbait designs (2 emotional valences × 2 targeting factors), for which they reported their perceptions and whether they would click on it and why or why not.Next, participants were randomly assigned one of the two design tasks, where they frst selected an overlay and a header, followed by providing the rationale behind their selection.Then, participants selected the message they would want to convey.They were asked to express their message through a story using components for expression.They were allowed to ask for additional components for expression (e.g., one participant asked for an elderly character).Then the participants provided their rationale behind the selection of message.Next, they were asked to select the navigation buttons and explain their selection.
Once the design was completed, participants explained their perceptions of why the story depicted in their design would be efective to prevent users from clicking on a targeted clickbait.Thereafter, participants were shown the remaining three targeted clickbaits one by one, and asked to elicit what changes they would make in their warning design considering the variations in targeted clickbaits; they also explained why the changes, if any, were necessary.They were also asked to select the most efective targeted clickbait and explain their choice.Finally, the participants were asked to complete a demographic survey hosted in Qualtrics 4 .They were compensated with a $15 Amazon.comgift card for their participation.
Analysis.The audio recordings from the study were transcribed and combined with the warnings designed by our participants.We performed thematic analysis on our transcriptions and the stories created by our participants [15,22,23,131].Two independent researchers coded each transcript and the story in the warning.The researchers read through the transcripts and stories of the frst few interviews, developed codes, compared them, and then iterated until we had developed a consistent codebook.After the codebook was fnalized, two researchers independently coded the remaining interviews.88.9% of the codes matched between the two reviewers, resulting in Cohen's Kappa score of 0.83.The two coders discussed and agreed on the discrepancies in the remaining codes.Finally, we organized and taxonomized our codes into higher-level categories.

Findings
4.2.1 Perceptions of Targeted Clickbait.The participants reported their perceptions and rationale behind clicking or avoiding the post (targeted clickbait), where our analysis revealed four prominent themes as presented below.
Relevance.About half of the participants agreed they would click on the post just because it was relevant.Some of them mentioned they would be surprised to see their friends in a social media post, and click any such post about someone they recognize.One of them said, "Yeah, I probably would [click on the targeted clickbait], especially if I knew the person I would want to see what happened."Curiosity.Most participants reported that curiosity about the post was enhanced by its relevance.They further agreed that curiosity depended on the headline and the photo presented in the post.Some participants particularly pointed to words like "collapsed" and "you won't believe" in highlighting the role of headlines to grab their attention.One participant said, "Yeah [I am interested in the post].It is probably the use of words like collapsed for the player's situation, and then, when it says what happened, I expect to learn what actually happened if I click on it." (P15).
Suspicion.Some participants raised suspicion when presented with the post due to their prior negative experiences with similar posts on social media.A few of them also identifed the post as clickbait.One participant said, "No [I will not click on it], because the frst time I got this kind of post, and I clicked on it, it was fake, so I am afraid to try it." (P12).A few participants felt the post was sketchy due to attention-grabbing headlines or fake-looking thumbnails, one of them commented, "Yeah, it defnitely catches my eye.Just because any time I see you won't believe what happened, I know it is clickbait.... I probably wouldn't click on it" (P13).The responses show that clickbait features like the attention-grabbing headline can be an identifying factor that help users to detect and avoid a clickbait.A few participants also refected on their experience with posts using image manipulation in social media, similar to a targeted clickbait.One participant said, "Lately there's tons of post on social media that's like someone you know died.And then all my friends are tagged in it.And I'm like, obviously that's fake." (P7) Habituation.Some participants reported their inclination to click on the post despite facing similar posts and identifying it as clickbait due to their non-consequential past experiences.One of them commented, "Sometimes I do realize, like, okay, this is probably just like some sort of clickbait like trying to get me to click.However, if that is a topic that I am interested in, then I usually will click on it as I don't think it is that bad." (P3).Our results indicate that participants are unaware of the hidden consequences of clickbait, like the common use of clickbait by malicious sites, and collecting information through cookies.While some participants are aware of consequences such as lots of ads and time wasting, they rarely think these consequences are harmful.These issues habituate users to interact with clickbait.

Perceptions of Warning Components.
In this section, we report our participants' perceptions of the warning components.
Overlays.For Harm Design, about half of our participants selected the red overlay, citing three advantages: attention, immediate conveyance of danger, and efcacy in warning users.One participant said, "Red is just a big warning color.... I mean, since we learn that red means stop, red is the color that makes you pause and, like, have caution.So if you're going to have an eye-catching warning, then it should be red." (P13).Some participants selected yellow.Only a few selected gray as they did not consider targeted clickbait to be malicious.For Report Design as well, about half of our participants selected the red overlay and about half selected the gray overlay.Those selecting the gray overlay argued that red is too strong of a color to convey that a post is misleading.
Headers.In Harm Design, about half of our participants selected the goal-oriented header that reads "Clicking on this post may harm you," as it conveyed the risk of clicking a post.One participant commented that the goal-oriented one was better than the other two: "I probably wouldn't even think about [selecting the frst one].
[As for the second one that says] You should not click on this post.Yeah, I probably shouldn't, but I probably would still do it.But having this may harm you would get my attention more."(P4).About half Messages.In Harm Design, about half of our participants preferred to convey the stealing of information due to relevancy, where cookies from sketchy sites often collect user information.Some participants shared their experiences, such as when they clicked on a link and started receiving promotional emails.Some participants related the message to losing their sensitive information as a result of hacking.Only a few participants selected any of the other messages conveying the harm of targeted clickbait.For Report Design, about half of the participants preferred fact-checking tools as the source of report, where they believe there would be no bias from artifcial intelligence.One participant said, "[I like] This bottom one: fact-checking tools report this post as misleading.That seems more reliable than the other ones." (P3).Only a few participants preferred any of the remaining messages.
Navigation.In Harm Design, about half of the participants selected "Keep Scrolling" button, as they found the suggestive language appropriate.Most participants selected "Proceed Anyway?(unsafe)" due to the button acting as a second reminder to the users.One participant commented, "[I like proceed anyway (unsafe) because] They have the unsafe in red just to give this person an extra opportunity not to click on it."(P1).In Report Design, most participants selected "Keep Me Safe!" button as they felt that clicking on it could protect them from misleading posts.One participant mentioned, "I expect, keep you safe to work in a way that if someone clicks on keep me safe, then this content would be hidden in future."(P20).About half of the participants preferred including the "Proceed Anyway (unsafe)" button, again due to it acting as a second reminder.Here, about half of the participants preferred the "I accept the risks" button as it would remind users that the link is risky and put liability on their action.

Themes in User Stories.
In this section, we report on the stories from our participants, which are categorized into four themes.
Harm: Emotional Story.About half of the participants who did Harm Design wanted to convey an emotional story, where a character faced consequences from targeted clickbait.Almost all of these stories were motivated by negative past experiences of participants or someone they know.One participant (see Figure 4a) mentioned, "Well, I actually did have a friend before provide information, and she lost several thousand dollars for making that mistake.Okay, I think what I would like to do is have one of her where she's just like neutral.And then another picture of her like down here, of her very sad after what had happened." (P1).Upon further decomposition of such stories, we identifed three key elements in the design: a character who clicked on clickbait, a consequence that the character faced, and a negative emotion depicted on the character's face.
Harm: Competent Peer.About half of the participants depicted a story where a character interested in the targeted clickbait was stopped by another character, conveying its consequences.One participant (see Figure 4b) said, "So my brother came across a post on social media with a picture and a headline that said, you won't believe what celebrity was arrested for this crazy crime.... I warned him to let him know that there are lots of websites that create posts like this, and then they will steal information from your account if you click on it."(P5).Upon further analysis, we found three key elements common across almost all such stories: a character interested in the post, a knowledgeable character, and a consequence explained by the knowledgeable character.
Report: Realistic Conversation.About half of the participants depicted a conversation between two characters in their story.Almost

Clicking on this post may harm you
Interesting!! Let me click on this...

Your brother Your brother
My info just got stolen!Oh no!!Somebody help! 10 mins

Keep Scrolling
Proceed Anyway?(unsafe) This post is reported for misleading users.You don't want to be one of them.

I accept the risks
This post is reported for misleading users.You don't want to be one of them.
Hey! Doesn't this look interesting.Maybe I should click on it.
You should not.Fact checking tools report the post as misleading.

Friend Competent Peer
(a) A realistic conversation as a story (b) Story with a competent peer reporting from a credible source all of them reported that they usually work or study with a friend, and when they encounter something like this, they would fgure out together what to do.One participant (see Figure 5a) said, "I think it's pretty common for people to be either in a work setting or working on homework or just friends getting together and scrolling on the Internet where one person is looking like, oh, this seems weird.And the other person reassures like, yes, that is weird.It looks like a scam."(P3).Our decomposition of these stories revealed two key elements: two characters talking with each other, and the source from which they discover that the post is misleading.
Report: Competent Peer with Credible Source.About half of the participants created a story where a character interested in the post was informed about a credible source reporting the post as misleading by another character.One participant (see Figure 5b) said, "[The story would go like] Oh, shoot!I just clicked on this thing, and this warning came up.I don't know what to do.Another person would say, Whoa!What did you click?And they'd be like, this is reported as misleading from fact-checking tools." (P8).These stories contained three key components: a character interested in the post, a knowledgeable character, and a credible source of report.

4.2.4
Warning Changes for Targeted Clickbait Variations.Most participants agreed that only a change in emotion did not change the targeted clickbait enough to warrant a change in the warning.Therefore, we would not use emotional valence as a variable in the evaluation survey.About half of the participants agreed that a warning might need to change with the change in targeting factor.We note that these changes are related to increasing the threat level of warnings.One participant mentioned, "I would probably put more of a stronger warning on this one [lottery with face swap].Just because I feel like anything with money is just very scammy."

I accept the risks
This post is reported for misleading users.
You don't want to be one of them.

Translation to Final Designs
Since the participatory design study is qualitative [15,22,23], it is not reasonable to select the most used components, especially where the diference is small (e.g., fve participants select red overlays, while six participants selected yellow overlays).Therefore, we conducted a focus group discussion (FGD) with four participants (PD1-PD4), including UX experts, graphic designers, and psychology majors.The participants were recruited via email to the respective departments of our university.The study was conducted over Zoom and lasted around an hour.The Institutional Review Board at our university approved the study.

Methods.
At the beginning of the FGD, participants were provided with the qualitative and quantitative measures from our fndings in the participatory design study.Then they discussed the fndings that guided their fnal selection of components.Our participants frst selected the overlays, headers, and navigation buttons.The expression of the message through storytelling varied among our participants in the participatory design study, and we used thematic analysis to identify themes and key components within the stories (see §4.2.3).During the FGD, participants discussed and selected combinations of components, resulting in the stories we fnally used.Here, four themes were translated into four story-based warnings, which we evaluated in the next study (see §5).At the end of the FGD, participants completed a demographic survey and were compensated with a $15 Amazon.comgift card.

Story-based Warnings for Evaluation.
Harm: Emotional Story (shortened to Harm: Emotional).For the story elements, participants had the two characters be siblings.One participant expressed, "Siblings are the characters that users would endear and care about in the warnings".(PD3).Similarly, stealing information was selected as the consequence, and anxiety as the negative emotion (see Figure 6a).The participants selected the red overlay, the goal-oriented header, and the "Keep Scrolling" and "Proceed Anyway?(unsafe)" buttons for the remaining components.Our participants agreed with the responses from participatory design in selecting these components.
Harm: Competent Peer (shortened to Harm: Peer).Our participants selected the same warning components as for the emotional story for similar reasons.For the story elements, participants had the knowledgeable character and the character interested in the post be friends.One of them mentioned, "I can imagine two friends talking with each other where one is competent and informs the other".(PD1).The participants selected infection of devices through viruses as the consequence since it seemed realistic and convincing to them in a conversation between two friends (see Figure 6b).

Report: Realistic Conversation (shortened as Report: Conversation).
The participants discussed and agreed upon two friends as the characters having a conversation and fact-checking tools as the source from which they discover that the post is misleading (see Figure 7a).One participant mentioned, "Realistically speaking, I think the conversation between two friends is the only option I can think happening in reality." (PD1).For warning components, participants selected the red overlay and goal-oriented headers, agreeing with the responses from our participatory design.Participants selected the "Keep Me Safe!" button, as it felt like a safeguard against misleading posts, and the "I accept the risks" button, as it conveyed that users would be liable for consequences.
Report: Competent Peer with Credible Source (shortened as Report: Source).Our participants selected the same warning components as with the realistic conversation.For the story, participants selected friends as both the knowledgeable character and the character interested in the post.They agreed that fact-checking tools are the most credible source (see Figure 7b).

EVALUATION (RQ3)
Along with the four story-based warnings (see §4.3.2),we created two control conditions (one each for harm and report) that include all but the story in their design (see Figure 8).The comparison between controls and story-based warnings thus shows the impact of stories in warning design.We evaluated six warning variations through a Qualtrics survey with 114 participants (medium efect size based on power analysis) over MTurk (see Figure 1c).The Institutional Review Board at our University approved the study.

Methods
Participant Recruitment.Participants had to be 18 years or older and live in the United States or Canada to participate in our study.We followed the guidelines from prior work [82,88,110] to increase the quality of responses, where we recruited participants with a 99% HIT approval rate, 5 and used masters qualifcation considering the nature and length of the survey.We compensated the participants with USD 2.5.The study took between 12 and 25 minutes to complete.Table 5 shows the demographics of our participants.
Procedure.At the beginning, participants were presented with an Informed Consent Document (ICD).After agreeing to the ICD, they   Thereafter, each participant was shown the six warning variations, in random order to avoid order efects.After each warning was shown, participants rated it in seven diferent survey questions on a 7-point Likert scale.The questions were presented in random order, with some questions reversed using antonyms to  avoid bias [36,150].The questions asked participants to evaluate the warning based on its Perspicuity, and Usefulness [123] using UEQ+, a validated scale of user experience [124].We also added custom questions -similar to prior studies [39,161,164] -where we asked participants about their Interest and Likelihood to click on the targeted clickbait with the warning; they also rated the warning in terms of personal Connection, Credibility, and Adoption.
We included six attention checks in this survey [69,83].Three participants failed at least one attention check question, and their responses were removed from the analysis.Participants were also asked two open-ended questions about their feedback on each warning and their rationale behind adopting (or not adopting) it.At the end, participants answered a set of demographic questions.
Analysis.We used statistical tests to analyze our quantitative results.We consider results to be signifcant when we fnd p<.05 using a Wilcoxon signed rank test.Figure 9 highlights the comparisons of our study conditions and corresponding outcomes.We performed thematic analysis for the qualitative results from our open-ended questions, where two independent researchers coded the responses and later discussed and resolved the discrepancies in their codes.The coding included a total of 1332 responses (111 participants × 6 warning variations × 2 open-ended questions), where the inter-coder reliability was 87.3%.

Findings
In this section, the reported means of the measures are on a -3 to 3 scale.Based on the UEQ handbook, 6 values between -0.8 and 0.8 represent a neutral evaluation, values > 0.8 represent a positive evaluation, and values < -0.8 represent a negative evaluation in a non-benchmarked scale.Since Interest and Likelihood are undesirable, we reverse the colors for these measures in Figures 11a,  10a, and 13a.Further, UEQ points out that due to the calculation of means over a range of diverse participants and answer tendencies (for example the avoidance of extreme answer categories), values close to +2 or -2 are considered extremities.We will use these classifcations for our measures except for Perspicuity, which is benchmarked and has defned levels for positive, negative, and neutral evaluations.

Is Targeted Clickbait Efective?
For the targeted clickbait, participants rated their Interest and Likelihood to click.We observed high scores for Interest (M=1.91,SD=1.39), which implies that based on the scenario, participants found the post relevant.Our fndings from the curation survey and participatory design highlight the importance of relevance in participants' decision to click . Further, these two measures (Interest and Likelihood) are strongly correlated (r=0.94,p<0.001).These fndings are in line with the communicative principle of relevance [140,141], showing that the relevance of targeted clickbait is a key factor in its efcacy.

Do Control Warnings without Stories Work?
In this section, we report on the efcacy of control warnings (i.e., warnings without stories).We observe that even without stories, warnings substantially decrease Interest in the targeted clickbait (see Figure 10a).As expected, given that Interest is an important aspect of users' decision-making process, we also see a considerable decline in the Likelihood measure.
Signifcance tests between the targeted clickbait post (without warnings) and the control conditions revealed that both of our control warnings signifcantly reduced Interest and Likelihood (see Figure 10b).Open-ended responses support these results, where participants highlight the power of refection due to the interruptions caused interstitial warnings.One participant noted, "I probably would adopt the warning in real life because it does not stop me from clicking the article if I really wanted to.It makes me stop and think why this warning would be there and discourages me from reading an unsafe article".Even without stories or details in the warnings, some participants refected on their actions and decided to fnd an alternate source to their answer.One of them mentioned, "The warning ofers to let me continue to the story, but it reiterates that it is unsafe.I'm going to keep scrolling, as the message suggests.I can always fnd out if the story is true by other means."Further, some participants pointed to the other benefts of the warnings, including saving time and avoiding fake information.

Do Stories about Harm Improve Warning Eficacy?
We observe that the control warning for harm is efective.The control was rated above average in all of the measures except Connection and Adoption (see Figure 11a).Now we aim to understand if adding user-informed stories further improves the control warning.From the survey results, we observe that story-based warnings conveying harm were rated higher than the control in all measures (see Figure 11a).Notably, Likelihood for the story-based warnings is rated below -2.1, indicating their efectiveness.
In open-ended responses, about half of our participants highlighted the lack of specifcity of harm in the control harm warning.Such specifcity is provided in story-based warnings, which helps explain their higher ratings.One participant commented, "The [control] warning doesn't do a good job of explaining exactly how the article may harm me.Are they going to attempt to phish me?Is it harmful to me in some kind of emotional sense (like it may cause me a negative reaction based on false information) or something?I don't really like this warning, and it isn't very clear, and I now have more questions than answers about what the site might be about.".Thus, most participants reported that they might not adopt the control warning, supporting the below-average rating for its Adoption.One participant mentioned, "I am not sure whether or not I would adopt this warning in real life, if I am being honest.I fnd myself slightly torn.The general idea is that the user should not click this link, as it may cause harm, but it's unclear exactly what type of harm.The more I think of it, the more I lean towards not adopting it." Signifcance tests pointed to the efcacy of the "Harm: Emotional" warning, which signifcantly reduced the Interest and Likelihood as compared to the control (see Figure 11b).About half of our participants appreciated the communication of harm through a story.One of them noted, "I can tell that it is trying to protect me and prevent me  The "Harm: Peer" warning performed signifcantly better than the control in all the measures (see Figure 11b).Openended responses support the fndings from our signifcant tests in Perspicuity, Credibility, Connection, and Adoption, where most participants reported positively about the warning's clarity, use of a competent peer, trust in their friends, and conveyance of specifc harm, respectively.One of them noted, "I would adopt the warning in real life because it is coming from a competent peer that I know.I would feel a personal connection to them, and I would heed their warning.I would want my friend to have a good opinion of me, and I would trust their judgment.".Some participants also highlighted the relevance of a story presented in the warning.One of them mentioned, "It's pretty easy to understand.It's like talking to your friend.It makes sense to me to speak to my friend in this matter so that I can protect her from harm." When comparing the two-story based warnings, we observe that "Harm: Peer" rates higher than "Harm: Emotional" in all measures except Interest and Likelihood (see Figure 11a).Signifcance tests between these two warnings revealed that "Harm: Peer" performed signifcantly better in terms of Credibility and Perspicuity (see Figure 12).Open-ended responses suggest that some participants found the message of harm coming from a friend more credible than the story depicting a single character facing harm.They also mentioned that conversations are easier to understand, which could explain the ratings for Perspicuity.One participant commented, "I feel that the conversation gives a few more details that makes it easier for me to understand what could happen if I click on the link."In light of the scores for Interest and Likelihood, however, we are unable to declare a champion story-based warning conveying harm.Most critically, our fndings show that adding user-informed stories can enhance the efcacy of warnings conveying harm.

Do Stories about Reporting Improve Warning Eficacy?
As with control warnings conveying harm, we observe that control warnings conveying report -i.e., the post is reported as misleading -are efective.The control was again rated above average in all measures except Connection and Adoption.Now we compare these with story-based warnings about reporting.First, we see that the "Report: Source" warning is rated higher than the control in all measures (see Figure 13a).As with the warnings conveying harm, about half of our participants conveyed the need for additional information in the control.One participant said about control, "The warning is very basic, and doesn't give a lot of context.While the warning is clear, it doesn't do a good job of explaining why the post is misleading.".The story element of the "Report: Source" warning addressed this need.
Signifcance tests revealed that the "Report: Source" warning signifcantly reduced the Likelihood to click compared to the control (see Figure 13b).In open-ended responses, about half of our participants appreciated providing the source of a report in the warning.One participant noted, "It gives some information about peer reviews, so I like that it explains why there is a warning."The "Report: Source" warning also performed signifcantly better than the control in terms of Connection (see Figure 13b).Similar to the harm warnings, some participants found the information coming from a friend personal and relatable.One of them mentioned, "The situation seems more realistic than the others, and the cartoon is done in a more positive light in that the character is trying to help the other one and giving a good reason of why they shouldn't click on it.".
The "Report: Conversation" warning, however, rates lower than the control in Perspicuity, Usefulness, Connection, and Adoption (see Figure 13a).Further, we fnd that the control was signifcantly better than the "Report: Conversation" warning in terms of Perspicuity (see Figure 13b).This warning attempts to depict via a comic-like presentation how two characters are conversing to discover the report.Even though open-ended responses for the control warning highlight the need for specifc information and more details, participants found this presentation to be complicated and difcult to comprehend.One participant mentioned, "It's a good way to warn people about viruses but it is also very complicated and not straightforward.I would rather use a diferent approach and one that doesn't have many steps to it." According to some of our participants' comments, the difculty in comprehension negatively impacted their perceptions of warning's Usefulness and their response for its Adoption.
When comparing the two story-based report warnings, we observe that "Report: Source" is rated higher than "Report: Conversation" in all measures (see Figure 13a).Signifcance tests between these two warnings revealed that "Report: Source" performed signifcantly better in terms of Perspicuity, Usefulness, and Adoption (see Figure 14).As described above (for "Report: Conversation"), these results can be explained through the difculty of participants in understanding the concept of two characters discovering that a post is reported.
On the Connection measure, we perhaps surprisingly found that all six warnings performed poorly (see Figure 11a and 13a); the reason is unclear from the open-ended responses.We speculate that participants did not feel personally connected to warnings, as they would stop them from performing their primary task.Further studies are needed to have more in-depth understanding of participants' perceptions and ratings on personal connection.We note that story-based warnings performed signifcantly better than the control warnings for Connection (see Figures 11b and 13b).

Is
There a Champion Story?Finally, we compare all four story-based warnings (two each for harm and report).We observe that the "Harm: Peer" warning is rated the highest in all measures except Interest and Likelihood (see Figures 11a and 13a).In these two measures, "Harm: Emotional" is rated the highest."Report: Conversation" warning is rated the lowest in all of the measures except Credibility, in which measure "Harm: Emotional" is rated the lowest (see Figure 11a and 13a).
In terms of signifcance tests, we fnd that both harm warnings signifcantly outperform "Report: Conversation" in nearly every measure (see Figure 15)."Harm: Emotional" and "Report: Source" are roughly even, with only on signifcant result for "Harm: Emotional" in the Interest measure."Harm: Peer" is at least slightly better across the board versus "Report: Source", but with only two signifcant diferences: Credibility and Usefulness.Given that "Harm: Peer", "Harm: Emotional", and "Report: Source" are all efective with few signifcant diferences, we cannot select a clear champion.

DISCUSSION AND IMPLICATIONS
Our studies report on the understanding and behavior of users towards targeted clickbait, the design of user-informed story-based warnings, and their efectiveness against targeted clickbait.In this section, we discuss the implications of our fndings and provide guideline for future research in these directions.

Moving Towards a User-Informed Design Process
Prior studies [17,70,100,101,152] highlight the difculties and challenges faced by end users that are often overlooked by designers.According to Norman [101], information that the designers want to convey through warnings may difer from the information perceived by end users.Therefore, our studies include end users, starting from the ideation of targeting factors and countermeasures and continuing throughout the design process.
During our user-informed activities, we faced challenges, particularly in the participatory design study.We had to conduct multiple pilot sessions with non-technical users and internal feedback sessions to align our design process, design activities, and interview questions with the understanding of the users.Based on our observations, users fnd it difcult to design artifacts without signifcant structure and guidelines and may struggle to express their ideas through designs.These observations led us to create a guided stepby-step design process, where the design is divided into multiple steps (e.g., select an overlay for the warning) and are guided by the researchers (e.g., think of the size for the overlay).
Our fndings indicate the efcacy of this process, where the evaluation of our six user-informed warnings (including control warnings, where the components outside of stories were also userinformed) point towards their efectiveness against targeted clickbait (see §5.2.2, §5.2.3, and §5.2.4).Based on the positive reception of our designs, we encourage the HCI community to adopt more user-informed design processes and involve users from the early stages.We believe that this approach will beneft the community in aligning designs to users' needs and expectations.

A Challenge: The High Relevance of Targeted Clickbait
Scott [127] explains the working of clickbait through the cognitive principle of relevance: it makes the information behind the link suddenly seem relevant to answer a question created in the user's mind by the clickbait title [140,141].Untargeted clickbait does not, however, align with the communicative principle of relevance: it should be the most relevant point for the user.The user has other reasons to be on social media that may hold more sway than the clickbait's curiosity-inducing title.Targeted clickbait aims to satisfy the conditions for both cognitive and communicative principles of relevance at the same time.
Our fndings support the working of targeted clickbait based on relevance theory.Our results from the curation survey represent high scores for users' Likelihood to click on targeted clickbait (see §3.2.2).Similar results are echoed in the participatory design study, where most participants' decision to click on the post was infuenced by its relevance (see §4.2.1).In the evaluation study, we see high scores for Interest in the post.Moreover, Interest has a strong correlation with Likelihood of users to click on the post (see §5.2.1).These fndings support the working of targeted clickbait through relevance theory, and suggest that targeted clickbait can be a big threat in the near future with the increasing ease of access to artifcial intelligence (text generation and image manipulation) and public information in social media.We encourage industry and academia to adopt preventive measures against the threat of targeted clickbait by developing and deploying appropriate tools and technologies.In doing that, we believe the user-informed warnings from our studies will function as an initial reference for future research and development.

Efcacy of Refection through Interruption
We observe that users are primarily motivated to click on targeted clickbait due to its relevance and the manufactured information gap.Interstitial warnings shift users' focus from their desire to click on the post to a refection of their action through interruption [72].
The efect of interruption is apparent from the efcacy of control warnings and our participants' open-ended responses (see §5.2.2).Although control warnings do not contain any stories, thanks to their interstitial nature, they induced participants to refect on their actions.These fndings support the prior literature in phishing and malware, demonstrating the efcacy of interstitial warnings [44,72,117,132,158].We found that a few participants had already experienced targeted clickbait through the manipulated pictures of their friends in social media (see §4.2.1).As access to image-manipulation and text-generation technologies increase the viability of targeted clickbait, story-based warning can be a reliable way to support users' in making informed decisions.We suggest leveraging the efective variations of stories interchangeably to avoid habituation.Vance et al. [146] point to habituation as a primary inhibitor to the efcacy of security warnings, and suggests using variations of the warning to address this issue.

Threat Personalization in Warnings
Our fndings lead us to recommend the personalization of warnings against targeted clickbait.We observe that users have varying perceptions of the threat level of a post (see §4.2.4).Our participants also perceived diferent levels of threats from two targeting factors, even though both of them could be equally harmful.Therefore, we emphasize that a warning against clickbait should convey the correct threat level.Our fndings suggest using colors in the overlays of warnings based on color theory to portray threat levels.The correct conveyance of a threat level ofers multiple benefts.For instance, users immediately understand the threat they will face if they click on the post, persuading them to avoid malicious websites -otherwise, some users might choose to ignore the warning, thinking the post is not harmful.If all warnings use a red overlay and convey a high level of threat, then a user may lose trust in the warning if they keep encountering them for posts that they fnd non-malicious but only waste their time.Such experiences can lead them to ignore similar warnings in the future [146].
We acknowledge the need to develop artifcial intelligence to scalably identify the threat levels of clickbait with accuracy, and thus recommend future research in this direction.

Limitations and Future Work
The participant pool in our studies is limited to users from the U.S. and Canada.We note that the societal and cultural background, literacy rate, public policy, economic condition, and infrastructural support could impact users' perceptions and behavior towards targeted clickbait and the story-based warnings designed to protect against it.Recent studies [5,6,41,108,129,133,147] point towards the importance of looking beyond Western contexts.To this end, we encourage future research to validate and extend our work, and include participants from diverse backgrounds and geographic regions, including developing countries.
Twenty four participants took part in our participatory design study, where we followed widely-used methods for qualitative research [9,15,22,23,130], focusing in depth on a small number of participants.We acknowledge the limitations of such study that a diferent set of samples might yield varying results.Thus, we do not draw any quantitative, generalizable conclusions from this study.Rather, we leveraged the fndings from participatory design to conduct an evaluation survey, where we targeted a medium efect size based on our power analysis.
Since users' security and privacy perceptions are positively infuenced by their knowledge and technical efcacy [68,94,128], and the majority of our participants are educated, we speculate that the perceptions and behavior of users reported in this paper represent an upper bound in the context of protecting against targeted clickbait.We recommend future work to focus on less-educated population in understanding their behavior towards targeted clickbait and identify the scope of enhancing our warning designs to address their needs and expectations.

CONCLUSION
Our fndings contribute to the taxonomy of targeting factors in targeted clickbait and countermeasures against it (RQ1), which provides directions and framework for future exploration on this problem.Using the taxonomy, we create story-based warnings against targeted clickbait through user participation (RQ2).Our evaluation then shows the efcacy of these story-based warnings (RQ3).Our study reports on the efcacy of targeted clickbait in tricking users (see §5.2.1) and the motivations behind users' decisions to interact with targeted clickbait, including relevance of the post, the curiosity gap created by the post, and habituation due to non-consequential past experiences (see §4.2.1).It also shows that at least three variations of story-based warnings designed through user participation can be efective against targeted clickbait (see §5.2.3, and §5.2.4).Finally, we recommend threat-level personalization and interchangeable use of user-stories to resist habituation.

APPENDIX A DESIGN TASK AND ITS THEORETICAL FRAMEWORK
The theory of planned behavior [4,8] is the foremost theory of behavior change, which has brought about multiple studies developing Behavior Change Techniques (BCTs).The BCTs are clustered and taxonomized by the studies of Abraham et al. [1] and Michie et al. [95].Based on these studies [1,95], the conveyance of harm, and infuence through others (reporting) are two of the most efective BCTs.These theories are in line with our fndings from the curation survey (see §3), and motivate our design tasks (see §4).
Overlays.Based on color theory [25,47], we provided three variations of overlays: red, yellow, and gray to convey the threat level (from high to neutral).Prior studies [57,135] showed that colors in warnings could immediately convey the threat to users, helping them make informed decisions.
Headers.A header represents the frst message users see in a warning [56,73,156,157].We provided two generic headers and a goal-oriented header.Generic header has two variations conveying that the post is baiting users, and users should not click on the post (see Figure 3).The goal-oriented header conveys that the post can cause harm (for Harm Design), and the post is misleading (for Report Design).
Navigation.Since users make their decisions upon refecting on the content of an interstitial warning [24,117], we provided two types of navigation buttons (see Figure 3): blue button (Scroll Buttons) that users can click to avoid the post, and a text button (Ignore Buttons) that would remove the warning and let users click through the post.In our study, we provided three variations of scroll buttons: "Keep Scrolling" (conveying suggestion), "Scroll Away" (conveying action), and "Keep Me Safe!" (conveying desirable outcome).We also provided three variations for the ignore buttons: "Proceed Anyway?" (conveying action), "Proceed Anyway?(unsafe)" (conveying a secondary reminder), and "I accept the risks" (conveying liability to users).
Messages.The messages depend on the goal of design tasks.For Harm Design, the messages varied based on diferent possible consequences of clicking on a clickbait (see Figure 3).For Report Design, messages varied based on the source, which reported the post as misleading.These sources include social media users, anonymous crowd of users, fact-checking tools, and professional fact-checkers.
Components for expression.The selected message is expressed through a story in the warning.We presented the story through the portrayal of characters, emotions, and other graphical components, including thought bubbles, speech bubbles, and arrows that are commonly used in comics (see Figure 3).We leveraged the taxonomy of basic emotions [46,142] to choose a negative emotion surrounding anxiety/fear, sadness, or anger that is depicted on a victim's face.

Figure 1 :
Figure 1: Summary of the studies reported in the paper

Figure 2 :
Figure 2: Designs used in the Curation Survey.(a), (b) show targeted clickbait using face-swaps with friends, and (c) shows the text-only warning conveying harm.

Figure 3 :
Figure 3: Harm Design (conveying harm) with all the design components

Figure 4 :
Figure 4: Story-based warnings conveying harm created by users (P3).Some participants reported they would click on posts about their university as they would be enticed to know.These responses indicate that the relevancy of targeted clickbait would infuence a user's decision to click on it.Participants also mentioned they would click on the post despite a warning identifying it as a clickbait, where one of them said, "It [posts about university football] is something I usually talk about with friends and family.So I usually click on it even if it is clickbait so I can talk to them about it and know what's happening." (P5).

Figure 5 :
Figure 5: Story-based warnings conveying post is reported as misleading created by usersof the participants selected the header about baiting them, since it explicitly denoted the post as a clickbait.Only a few participants selected the header about not clicking the post, as they did not like being told what to do.For Report Design, about half of our participants selected the goal-oriented header, as they liked being informed about the specifc reasons why they should avoid the post.One participant said, "This one gives a little bit more context than the other headings.Instead of just saying this is clickbait or don't click on it, it says, well, this post is reported as misleading.So, I like that one a little bit more." (P3).

Figure 6 :
Figure 6: Story-based warnings conveying harm, as used for evaluation

Figure 7 :
Figure 7: Story-based warnings conveying reported posts, as used for evaluation (a) Control condition for warnings conveying harm (b) Control condition for warnings conveying post is reported

Figure 8 :
Figure 8: Control condition warnings with the same components except the story

Figure 9 :
Figure 9: Comparison of the study conditions and its outcomes

Figure 10 :
Figure 10: Comparisons between the targeted clickbait post (without warning) and control warnings

Figure 13 :
Figure 13: Comparisons for report warnings: control vs. story-based

Figure 14 :
Figure 14: Signifcance test results comparing the two story-based report warnings

Figure 15 :
Figure 15: Signifcance test results comparing the story-based warnings conveying harm and report

Table 5 :
Demographic Information of the Participants in the Evaluation Survey ( =Number of Participants)

Report Story I: Realistic Setting Report Story II: Credible Source Control Report
5https://www.mturk.com/worker/helpShrestha et al.

Comparison of the Study Conditions and its Outcomes
[37,54,63,71,84,99,137]hat leveraged persuasive narratives (stories) to change user behavior[37,54,63,71,84,99,137], our fndings show that users can be persuaded to avoid targeted clickbait using storytelling, where persuasive narratives in warnings helps them to refect and make an informed decision about their online safety.The efcacy of story-based warnings is clear from the significant reduction in the participants' Interest and Likelihood to click on targeted clickbait (see §5.2.3, and §5.2.4).Further, our fndings highlight the efcacy of story-based warnings in measures such as Perspicuity, Usefulness, and Credibility, highlighting the importance of combining information with stories (see §5.2.3, and §5.2.4).