Understanding Users' Interaction with Login Notifications

Login notifications intend to inform users about sign-ins and help them protect their accounts from unauthorized access. Notifications are usually sent if a login deviates from previous ones, potentially indicating malicious activity. They contain information like the location, date, time, and device used to sign in. Users are challenged to verify whether they recognize the login (because it was them or someone they know) or to protect their account from unwanted access. In a user study, we explore users’ comprehension, reactions, and expectations of login notifications. We utilize two treatments to measure users’ behavior in response to notifications sent for a login they initiated or based on a malicious actor relying on statistical sign-in information. We find that users identify legitimate logins but need more support to halt malicious sign-ins. We discuss the identified problems and give recommendations for service providers to ensure usable and secure logins for everyone.


INTRODUCTION
Login notifications intend to inform users about recent sign-ins, to protect accounts from unauthorized access.Depending on the service, notifications are sent if the login occurred from an unknown location or new device, which may indicate malicious activity.
Notifications are often delivered via email and include details about the device (browser and OS), approximate location, date, and time of the sign-in.Users need to decide whether the reported login is legitimate or malicious and are recommended to change the password in case the login is unfamiliar.Logins can be confused to be malicious when users share accounts, and friends or family log in unknowingly.While the notification is intended to protect users and provide a feeling of security, it can also be perceived as burdening and overwhelming by requiring a decision based on technical jargon and highlighting negative consequences.
Previous work [41] focused on challenge-based notifications and studied incident-response information-seeking and mental models about attackers.In contrast, we focus on granted access notifications informing users about a recent sign-in and analyze users' comprehension, expectations, and reaction to the notification.
In this work, we collected and analyzed 72 login notifications sent by real-world services and developed a baseline notification that we employed in a user study.The structure of the study is shown in Figure 1.We disguised the study ( = 229) as a psychological test, and let users create an account they had to sign into during different stages of the study.Participants then either received a legitimate notification to their email upon signing in themselves (Legitimate) or unexpectedly received a notification prefilled with sign-in information a non-targeted statistical attacker would use after around one week (Malicious).
We sought to answer the following questions: RQ1 [Reaction & Comprehension] Which actions do users take in response to receiving notifications, and is resolving the situation a priority?Do users understand why they received the notification and which factors may have caused receiving it?
We found that participants correctly understood that "a login" caused receiving the notification.However, they are unaware of or misinterpret the trigger and are thus unsure how to react appropriately, especially in the malicious case.
RQ2 [Decision-Making & Execution] Do the state of the art notifications help users distinguish malicious and legitimate logins?Which information helps account owners with their decision, and do current notifications appropriately guide users in resolving the situation?
Based on device and location, participants can correctly attribute notifications caused by their own logins, but they are confused when the notification is unexpected (Malicious) and struggle to identify the correct reaction even if (as it was the case in our study) all necessary information is provided by the notification.

RQ3 [Perception & Expectation]
How do login notifications make users feel?When do they expect notifications to be sent, and how does prior experience affect their decision?
Notifications about malicious logins evoke (more) negative emotions, but participants who changed their password also felt empowered by taking action to protect their account.Interestingly, more than 90% of the participants expect services to send login notifications because it makes them feel protected.
Analyzing 72 real-world notifications revealed malformed login notifications and problematic anti-phishing advice.Our user study shows that login notifications contribute to account security, yet our results suggest room for improvement.We find that only 22% of the participants who should have changed their password to protect their account did.While participants appreciate when companies decide to monitor their accounts for incidents, services that send notifications for every, or almost every login in a "better safe than sorry" manner contribute to warning fatigue.We give clear recommendations for service providers to improve their notifications.While login notifications can help reinforce account security, protecting their accounts by identifying malicious logins should not be solely the user's responsibility.

RELATED WORK
Next, we outline how our research extends related work.

Risk-Based Authentication, Login Notifications, and Account Sharing
Only few have studied login notifications in the context of riskbased authentication so far.A qualitative interview study ( = 67) by Redmiles [41] explores the account security incident response at Facebook by interviewing users who experienced a login incident.Unlike our work, Redmiles focused on "secondary authentication" notifications that prompt users to enter a code to regain access to their accounts after the account has been temporarily disabled due to suspicious account activity.Redmiles interviewed participants from 5 countries and reported on incident-response informationseeking and mental models about attackers.Regarding the notifications' effectiveness, Redmiles identified a lack of key information as problematic, e.g., the likelihood that the notification is about a legitimate threat.In contrast, our work studies a different type of login notification (see Section 3).It focuses on users' comprehension, expectations, and reaction to the notification, not on regaining access or mental models about attackers.Markert et al. [30] studied administrators' risk-based authentication (RBA) configuration.Administrators are responsible for the content of the login notifications users receive.The researchers found that the predefined notifications were often barely customized, and only a few administrators opted to disable them entirely.Also, participants lacked consensus about which information to include, indicating a knowledge gap.The administrators also wished for more context and explanation to prevent phishing attacks and pointed out the inaccuracy of IP-based location estimation.Our research helps identify key features of notifications that yield correct user comprehension.These results can help administrators align RBA configurations with users' expectations.
A study by Doerfler et al. [14] evaluated the efficacy of login challenges in preventing account takeovers, finding that up to 94% of phishing-rooted hijacking attempts and 100% of automated hijacking attempts can be prevented.This highlights the efficacy of login notifications in account protection and motivates the design of usable and understandably designed notifications.Still, Gavazzi et al. [17] found that only about 20% of popular websites employ risk-based measures.Wiefling et al. [62] showed that verification codes sent via email are the de facto standard for login challenges enforced by RBA.In a subsequent study, they demonstrated that providing this code in the subject of the notification can reduce the login time [63].Using account login notifications, Wardle [61] measured the time it takes for leaked credentials to be abused by creating accounts on web services and intentionally leaking the credentials online.Adding to this literature, our research contributes insights on concrete user behavior, identifying key success features to deepen the understanding of the notifications' efficacy.
Shared passwords and accounts are of particular concern when it comes to login notifications.When multiple individuals access the same account, the intended account owner might find it challenging to maintain control and recognize logins.In this context, Obada-Obieh et al. [36] investigated online account sharing and found that users struggle to remember which accounts they share and with  whom.Similarly, Song et al. [49] studied account-sharing practices in the workplace and observed conflicts over simultaneous access and difficulties controlling access.While account sharing is out of scope in our research, it can have influence on users' understanding of notifications which we also address in our discussion.

Security Warning & Notification Design
There is a large body of literature on security warning design [11,44,59].The most prominent applications are notifications in the context of TLS [3,16], phishing [39], malware [4], and cookie banners under GDPR [13,24,35,57], as well as warnings for developers [20] or for countering misinformation [23].For user authentication, there is work on breach notifications [22,64], password-reuse notifications [19,53], notifications to promote the use of 2FA [18,42], or FIDO2 [25], or protect users from using common PINs [29].While considering the best practices for notification design in other domains is important, this is not the main focus of our study.However, in our notification analysis (see Section 3), we try to identify common design patterns.

LOGIN NOTIFICATIONS IN THE WILD
Login notifications intend to inform users about recent sign-ins and often include technical details such as the login time, used device, or approximate sign-in location.However, depending on the service, they are not sent for every login.While theoretically significant location or device changes trigger notifications, the probabilistic nature involving factors like sign-in history and user behavior makes it difficult to predict when notifications are sent.Some services sent notifications for every login; others only sent notifications in case of significant location and device changes, causing a higher risk level.For example, we noticed receiving fewer sign-in notifications if the affected account had two-factor authentication enabled.Interestingly, the cause for receiving a login notification is not always transparent to the user.We encountered multiple instances where notifications were not triggered by the account owner logging into their account.Most commonly, the phenomenon of unexpectedly receiving a login notification is related to shared accounts (i.e., Netflix or Amazon) [8] but is also known from third-party apps or services automatically signing into an account on behalf of the user [1,40].

Notification Types
Based on the type of information they convey, notifications can be divided into three different types [30].Examples of each of them are shown in Figure 2.
(1) Granted Access: The notification informs about granted access.
Some services send such notifications for every sign-in, while others follow a risk-based approach.(2) Additional Challenge: These notifications inform about a new sign-in attempt for which an additional challenge needs to be solved (i.e., insert a code or click a link).(3) Blocked Access: The notification informs users about blocked access, which can happen because the risk-based authentication system ranks the sign-in as too risky.For the remainder of this work, we focus on the first type, i.e., notifications informing the user about granted access.This popular notification type is deployed by well-known companies such as Alphabet [5], Amazon [6], Apple [10], Meta [31], and Microsoft [32].Every organization can send this type, as it does not require an advanced risk assessment (i.e., basic logic and the ability to display login details are enough).Moreover, we limited our dataset to emailbased notifications.While notifications can also be sent via other channels, e.g., SMS or push notifications, establishing them requires additional effort.

Analysis Method
To familiarize ourselves with the state of the art of granted access notifications, we collected over 90 login-related emails from real-world services by enumerating over 500 existing accounts.To trigger the notification, we signed in using the Tor browser, which is often classified as suspicious activity, and monitored our inbox.We also searched through account remediation pages [34] and community support forums [9] and learned about them via friends and colleagues.In both cases, we created an account on the service to obtain a notification.Our collection is limited to the top Tranco list [26] websites (as of June 2023), with about 1  3 being in the top 100/1, 000/50, 000 respectively.The dataset includes popular websites from social media, streaming, shopping, finance, travel, email, and gaming services.Most of them are US-based (44) and the rest are from Europe (18) and Asia (10).The dataset is biased towards English notifications; few non-English notifications have been translated.The full list can be found in Appendix B and C.  For the analysis, two authors categorized 72 emails as granted access notifications.The authors then independently analyzed the notifications based on a set of features derived following an iterative coding approach until no new codes and themes emerged [12,55].In particular, the authors checked which sign-in information the notification includes (i.e., login time, location, device), what the main components are (i.e., headline, malicious instructions), salient design and wording decisions (i.e., logo, highlighting of sign-in details, neutral language), and metadata such as sender and subject.Conflicts were resolved when they emerged by consensus discussion with a third member of the team (resulting in a hypothetical final agreement of 100%).

Findings of Notification Analysis
We summarize our findings in Figure 3. Please refer to Appendix B and C for the full details.
Sign-In Information As depicted in Figure 3, the majority of notifications included the login 7 Time (78%), 5 Account Name (74%), 7 Country (68%), 7 Browser (63%), and 7 Operating System (62%).Less frequently, the notifications included the 7 Time Zone (59%) or a login IP Address (43%).The small number of notifications, including the login City (37%) or State (29%), is explained by geographical differences between the U.S. and Europe.For our dataset, we collected notifications from different sources and observed that notifications for logins in the U.S. mostly reported the state.Notifications for logins from Europe often also included a city.
Components Most notifications made use of a 4 Headline (73%) that was often (76%) different from the email subject.Another critical component were the instructions describing how users should respond to the notification.While only 64% provided instructions in the 8 Legitimate case, more than 97% explained how to react in the 9 Malicious case if the user does not recognize the login.The large majority (66%) recommended changing the password.Fewer (high-ranked) web services included a button to report the login as malicious or legitimate on a separate web page (9%) displaying account remediation steps.Similarly, a small number (9%) suggested to visit the account activity page.Prominent among financial services was the option to contact support (4%).A dedicated Why Notification component was included in 21% of the notifications.It primarily creates context and explains to users why they received the notification.It often gives examples of legitimate (i.e., new device) and malicious causes ("someone unauthorized gained access") that might have triggered the notification.33% included a link to a dedicated Help Page (note: regular support links in the email footer were not counted).About 24% of the emails tried to use the Opportune Moment to tell the user about other options to secure their account (i.e., enabling 2FA).The dangers of Phishing and methods to double-check the legitimacy of the notification were mentioned in 17% of the emails, with the most prominent suggestion being not to click the "change password" link and instead sign in to the website by manually pasting or typing in the URL.About half of the notifications included a 10 Closing (50%) text that often thanked the user and included the name of a "{service} account team."A footer with 11 Legal information was included in 68% of the emails, and an Unsubscribe link was present in 8% of the notifications.
Wording & Design Using affinity diagramming, we identified the wording of most email subjects as Neutral (65%), with a strong focus on "New login to {service}." In some cases, it is alarming (23%), like "Security alert" or a prompt (9%), like "Please review this sign in!. " In two cases, it was a question (3%), e.g., "Did you recently sign into {service}?. " Almost all emails (92%) referred to 6 "your account" to emphasize the importance of the notification.A few services tried to address the inaccuracies of IP-based location estimation by describing it as Approximate (26%).Most notifications (89%) were sent as HTML emails; the rest were sent in plaintext.For a "corporate look-and-feel, " 80% of all notifications included a 3 Logo, with an even split between a centered or left alignment.Interestingly, 23% of the emails displayed the sign-in information in a visually detached box, most likely to draw the user's attention to the login details.
Subject & Sender We identified three different types of email subjects: (a) The majority of email subjects (79%) did not include specific details and were relatively generic, e.g., "Your account has been logged into" (Tumblr).(b) A small fraction (15%) made use of login metadata, e.g., "New login to Twitter from {browser} on {OS}" (X, formerly Twitter).(c) Only a few (6%) included the account name, e.g., "{Name}, did you recently sign into Etsy?" (Etsy).Five services did not use an email sender name (display name).
Technical Details Throughout our analysis, we found numerous areas for improvement regarding the parsing and displaying of technical details such as location data, browser, and OS.We found incorrectly escaped HTML "&Icirc;le-de-France" instead of "Îlede-France," empty placeholders, e.g., "Browser: N/A," and cryptic smartphone model numbers "SM-S908B/DS" instead of accessible names like "Samsung Galaxy S22." Operating systems were often reported with their full version number, e.g., "iOS 17.1.2." We even found four notifications that included the raw User-Agent string.
Questionable Advice Some notifications included information about phishing, which does not always align with state of the art recommendations on those topics.Questionable advice is given by X (and three other major services), which suggests that the presence of a padlock icon will "let you know a site is secure" and that users should check for the presence of "https://" and "{domain}" in the hyperlink.Similarly, Amazon suggests better copying and pasting the "It wasn't me"-link into a browser "just to be safe." Spotify advises users to verify that the email was sent from "@spotify.com," which is only expedient if the email server and DNS are configured correctly.In line with the latest research, PayPal's advice [38] explicitly mentions to "not rely on the padlock symbol and the 's' in HTTPS".Interestingly, LinkedIn added a security footer message [28] to their login notification that includes the affected account name and corresponding profession to authenticate official emails.

Selecting a Representative Notification
For our user study (see Section 4), we aimed to use a notification that closely resembles the state of the art of real-world login notifications.Our data-driven baseline (see Figure 4) includes all components used by at least 50% of the analyzed notifications, leading to 11 components comprising the notification: It uses a neutral subject and a slightly modified headline.We adjusted the email sender, opted for an HTML email, and included a logo.We also mentioned the affected account name and referred to "your account." We listed the most popular sign-in details and legitimate and malicious instructions with actions for users to take after receiving the notification.Our study sample was U.S.-based, so we included the 7 State in the sign-in details.The email also had a closing and footer with fictional legal information.
By deriving a representative notification, we could test users' general understanding , their reaction, and their perceptions.While the majority of the 72 collected notifications included slightly fewer components (57% include at least 9 components) we still decided to include all 11 components in the baseline to be able to make a statement about each components' usefulness in an idealized scenario.Components omitted in real-world notifications most often were the 5 Account Name, e.g., "Hi {Account Name}, " and the 10 Closing, e.g., "Thanks, The AcmeCo account team."

METHOD
The following section outlines the protocol, treatments, recruitment, ethics, and limitations of our user study.

Study Protocol
Participants in this study should receive a notification for a concrete account.To resemble a real-world setting, the protocol had to fulfill four criteria: (1) A real account gets created, (2) participants are unaware that the study is about login notifications, (3) participants receive the notification in their personal email account, (4) and reactions to login notifications are measurable.
For this, we invited participants to take part in a multi-stage study about changes in the cognitive ability of mental rotation over time [48,58].This framing allowed us to inform people about the length of the commitment without revealing our interest and justified the account creation on our website.The task was also a strong cognitive distractor that prevented participants from paying too much attention to the notification and authentication task.
We used two treatments, and the baseline notification (see Figure 4) was adapted to the branding of our study's website: The legitimate group ( = 110) received a notification only after they logged in.The location, date, and device in the notification were derived from the metadata of their login.The malicious group ( = 119) received a notification unexpectedly at a time when they had not interacted with the account for multiple days.This resembled a login attempt by a malicious actor.The location ("California, USA") and device ("Chrome on Windows") were selected to have the highest statistical chance of matching any user in our U.S.based sample [50,51,56].Trawling (untargeted) attackers would likely use a similar configuration when signing into the account, to minimize the risk of being detected by RBA systems [14].Because the details are intentionally chosen to closely align with common user configurations -just like in real life -some participants in our study (10 of 119) also matched these details, making it more challenging for them to recognize that the notification is a result of a malicious login.We did not allow mobile devices and recorded if the email was opened via a small self-hosted image, which itself was invisible in the email.Stage 1: The first page on the study's website explained the mental rotation test.To ensure participants regularly check their email and understand the value of the account, they saw a privacy notice after giving their consent, which highlighted the importance of the account as it would be used to store the study data and email address.It also explained that the email would be used to send invitations to subsequent stages, and the compensation would be in the form of Amazon gift cards.After the account creation, participants solved 5 mental rotation tests and provided demographic information (MD1-MD4).At the end, participants in the legitimate treatment were informed that invitations to Stage 2 would be sent in approx.7 days; in the malicious group, the note said 14 days.Stage 2: After 7 days, participants in the legitimate group received an email inviting them to return to our website for another mental rotation test.To do so, they had to log into their account, which triggered a notification.Participants in the malicious group expected their next email after 14 days.However, to imitate a malicious login, we sent them an (unexpected) login notification filled with our statistical sign-in data 7 days after they completed the first stage.Stage 3: For the legitimate group, invitations to the final Stage 3 were sent 48 hours after they completed Stage 2; in the malicious group, 48 hours after they received a notification for a login they did not initiate.We chose this time frame to give participants enough time to react.After logging into Stage 3, participants were debriefed and told about the purpose of the study.This was followed by our questionnaire (see Appendix A).From then on, the notification we sent was shown on the left side of their screen for reference.
(1) Email: First, we asked if participants remember receiving the notification (MQ0); if not, they were forwarded to a different section (see Appendix A).Participants for whom we received a read receipt or who changed their password skipped this question.ticipants understood why they received the notification.MQ12 and MQ13 asked participants when they expect real companies to send notifications.(7) Prior Experience: We concluded with three questions covering negative experiences with security incidents (MQ14), as well as their opinion on regular (MQ15) and event-driven password changes (MQ16).

Recruitment & Demographics
We used the panel provider Cint for the recruitment of the study.They are a comparable platform to larger providers such as Respondi and Kantar, operating numerous sub-panels for different locations across the globe.Criteria for participation were being 18 or older, being willing to participate in deception studies, and being US-based.For Stage 1, we recruited 625 participants, about 3 times more than the desired number of completions as recommended by the panel provider.Our a priori power analysis determined the minimum sample size to be  = 100 per group -in order to achieve 80% power for detecting a medium effect (Cohen's  = .4),at a  1 shows the participants' demographics.Regarding the demographics, we observe a shift towards male-identifying participants (65%).The age distribution is diverse, ranging from 14% to 21% for all age groups between 25 and 74.Most participants had a high school (33%), Bachelor's (26%), or trade degree (23%) and did not have a technical background (82%).

Ethical Considerations
At the time we conducted the study, none of the authors worked at an institution with an IRB.However, we carefully followed the guidelines provided in the Menlo Report, including a risk-benefit evaluation, developing the protocol with peers familiar with conducting user studies and following the legal requirements.The study included deception and sent a login notification to participants' personal email accounts, which could have caused more anxiety than just imagining to have received a login notification.
To protect participants from unnecessary risks, we implemented several safeguards: i) Our panel provider offered the study only to participants who agreed to studies that might involve deception.
ii) The affected spatial reasoning account had no subjective value to the participants and only allowed to access the email address.Participants might have been concerned about their answers to the questionnaire, which, in their impression, were also tied to the same account.However, at the time the malicious login notification was sent, participants had not been asked any sensitive or personal questions yet.iii) All participants (including those who decided to withdraw early or drop out) have been debriefed.In particular, we told them about the true purpose of the study, and in case they belonged to the malicious treatment that "This sign-in did not take place; at no time was your account at risk, " and asked them whether they prefer to leave the study early (while being fully compensated), which nobody did.iv) We provided an optional contact address and feedback form that we closely monitored (we have not received any complaints).v) We shared a website (also accessible from outside the study) that participants could visit and share to learn more about login notifications and related account security measures.vi) We created a distinct email account for sending notifications that applied all state of the art email security features, which can prevent email spoofing attacks.We also allowed participants to reply to the notification and ask for assistance.Finally, all email addresses were only stored encrypted, separated from the study responses, and were deleted after the study in accordance with progressive data protection laws like GDPR and CCPA.

Limitations
For this study, we relied on a controllable artificial account setting, which might lack ecological validity.However, only 7 participants mentioned the non-real-world setting as a reason for not reacting to the notification.We expect more participants to change their password if the notification was sent for an account with a higher subjective value.We did not control for VPN usage, which might also slightly influence results in real-world settings.Like many human-subject studies, there is the potential for a bias in questionwording.To circumvent this, we piloted the study and tried to keep the questions short and clear.The full survey instrument can be found in Appendix A. Lastly, we only recruited US-based participants, which can have culture-based influences on the results.

RESULTS
Next, we present the results of the study.The qualitative coding was done by two of the researchers, who started by separately coding 10% of the answers.Afterward, they agreed on a joint codebook (see Tables 4-6 in Appendix D) and used it to code the remaining 90%.The agreement between the two coders was high ( = 0.77).When quoting individual participants, e.g., L61-N, one can derive their treatment (Legitimate or Malicious) and password change behavior (No Change or Change).Similarly, we use color codes like A Was Me corresponding to Figure 5 within the text when referring to participants' explanations.

RQ1: Reaction & Comprehension
Reaction General Out of the total 229 participants, 48 participants, 23 in the legitimate and 25 in the malicious treatment, F cannot remember the notification.Still, for 26 of them, we received a read receipt, so they must have at least opened the notification.Among  the large majority of participants who saw the notification (181; 79%), it was very rare that they completely ignored its content.In response to MQ1, just 6% said that they only read the subject.About 90% read the notification completely or at least skimmed the body.

Legitimate
Reaction Legitimate No participant in the legitimate treatment changed their password.As shown in Figure 5, the majority of participants (60; 55%) explained their reaction in response to MQ3b by saying A it was their own login.Another 12, i.e., 11%, described it as a E spontaneous reaction, e.g., M42-N: "I just didn't think much of it." We also see that some participants do not understand what the notification is saying, which was the driving reason for H 6% (6) to ignore it.Finally, we recorded themes of participants who were I suspicious about the legitimacy of the notification (4; 4%), felt C fatigued (3; 3%), or J protected (3; 3%).
Reaction Malicious In the malicious group, only 26 of the 119 participants, i.e., 22%, changed their password; all of them correctly saying B it was not them logging in.The reasons given by the other 78% (93) in response to MQ3b for not changing their password mostly overlapped with responses given by participants in the legitimate treatment: E spontaneous reaction (15; 13%), notification looked I suspicious (14; 12%), or was H not understood (8; 7%), C being fatigued (6; 5%) or D unsure how to react (6; 5%).Finally, there are two justifications that are owed to the study design: participants describing they A logged in themselves although they did not (11; 9%), likely an example of social desirability, and those who assigned a G low value to the account (7; 6%): "This account has no value, it was not a streaming or banking account or amazon account" (M74-N) This justification can be reasonable, but users need to keep in mind that an attacker can also target other accounts that verbatim or partially reused the compromised password [37].
Comprehension When asked why they have received the notification (MQ11), 85% (93) in the legitimate and 79% (94) of the participants in the malicious treatment realized that a new login happened.Very few who gave a different explanation believed it was a phishing attempt (3; 1%), most simply did not understand what has happened at all (39; 17%): "I had no idea, which is why I deleted it."(M93-N) Those in the legitimate treatment who mapped the notification to a new login usually perceived it as a simple info email (42; 38%), followed by those who saw it as a prompt to review the login (28; 26%).Fewer responses (15; 13%) explicitly mention that the login must have been abnormal.In the malicious treatment, most participants who understood that a new login happened described that they were (potentially) compromised (46; 36%).Another 19% (22) perceived it as an informative but non-critical email.The remainder (13; 11% each) either mentioned that the system rated the login as unusual or wants them to review the login.
We observed a low comprehension of what might have caused the notification, especially in the malicious group.One explanation might be the temporal connection between logging in and receiving the notification.From MQ6, we know that about two-thirds read it immediately, and most of the others within a few hours.Hence, participants in the legitimate treatment had indeed a connection, and their understanding was substantially better.This influence of contextual factors was already observed by prior work on warning design [20,44] and could be achieved by including a Why Notification section.Some websites already do (see Section 3), and we will elaborate on this in the discussion.
Summary About 80% saw the notification.Participants in the legitimate treatment who triggered it themselves understood what it was telling them and reacted accordingly.In the malicious treatment where participants did not have this context, only 22% changed their password, and they had more difficulties explaining the circumstances.Hence, the number of password changes in the malicious treatment is substantially lower than expected.

RQ2: Decision-Making & Execution
We now focus on the decision-making process to understand if participants struggle with determining whether it was them or not, especially for malicious logins.Helpfulness of Login Information Foremost, we wanted to get insights into the helpfulness of the displayed login information (MQ5).In Figure 6, we can see that for those in the Legitimate and Malicious (Change) group, all information is about equally helpful: 22-35% find the different types moderately and 38-62% even extremely helpful.Participants in the Malicious (No Change) group, in contrast, appear to have a less distinct opinion as ratings are more equally distributed, ranging from 8-30%.A Kruskal-Wallis test also showed significant differences for all types of information when comparing Malicious (No Change) to Legitimate and Malicious (Change), respectively.This uncertainty of participants in the Malicious (No Change) group regarding the displayed information aligns with the previous section, where we found that those participants misattributed or did not understand the cause of the notification.
Effect of Other Factors In addition to the already-known influence of the login information, we were also interested in the effect of other exogenous and endogenous factors (MQ4).Figure 7 gives an overview.Generally speaking, the content (e.g., provided information, instructions, wording) and prior experience in dealing with such notifications had the highest effect on participants' reactions, with 42% expressing a moderate or major effect on average.Followed by that is the metadata (e.g., sender, subject, time of arrival) with 29%.All other factors seemed to have a weaker influence, with only 18% (appeared to be phishing) to 23% (was expected) of the participants reporting a moderate or major effect.
When comparing groups, Legitimate is the one where most participants reported a factor having no effect.The Malicious (Change) group, on the other hand, is the one where participants describe the strongest influences of the factors.Using a Kruskal-Wallis test with Bonferroni-correction for pairwise comparisons, we found that metadata had a significantly higher effect for Malicious (Change) participants compared to Malicious (No Change) participants ( 2 (2) = 6.65,  < 0.05).The same is true for the email content ( 2 (2) = 7.73,  < 0.05).Thus, to nudge more users to change their password upon receiving potentially malicious login notifications, focusing on designing the content and metadata is vital.
Influence of Negative Experiences Overall, 30% of participants described falling victim to a security breach within the last two years (MQ14).In the malicious treatment, 42% of those who changed their password reported prior negative experiences.Only 32% of those who did not change their password said so.The difference is not statistically significant,  2 (2) = 2.61,  = 0.271, but suggests that prior breach experience increases the likelihood of users changing their password upon receiving a notification.
Summary When comparing login information side-by-side, we can conclude that all factors are equally essential.We also observed that the helpfulness of the information for the Malicious (No Change) participants is significantly lower, which further explains the issues of this group when determining what happened.Regarding other factors, the content of the notification, its metadata, and prior experience in dealing with it had the highest effect across all treatments.Negative experience tends to influence the reaction as well; other aspects appeared to be less crucial.
Expectation So far, the study showed that there is a substantial number of participants who have not changed their password although they should, some of them mentioning that it was a spontaneous reaction, which this fatigue may also explain.Hence, we used MQ12 to learn when users expect to receive notifications.A majority of participants (151; 66%) expressed they want to receive notifications after suspicious account activity.On average, 60% want to be notified if a login takes place from a new device, 47% for logins from a new location, 31% if they have not logged in for a while, and 22% for logins that take place at an unusual time of the day.Only 9% want to receive a login for every login, and even fewer (9; 3%) do not want to receive login notifications at all.

Summary
We can conclude that participants who changed their password felt both more positively and negatively, probably because they assumed some form of compromise but also had a sense of achievement after preventing it by changing the password.The other groups had lower scores, aligning with them not expecting any harm.We showed that participants expect services to send login notifications and can further specify this by saying that participants want to be notified after suspicious logins, logins from new devices, and logins from new locations.Fewer participants expect to receive notifications based on temporal deviations.

DISCUSSION
Next, we discuss the takeaways and give recommendations.

Effectiveness of Login Notifications
We wondered if login notifications that many services use daily, achieve their goal of increasing the security of online accounts.
Effectiveness Depends on Trigger According to our findings, the notifications achieve what they intend-at least to a certain extent.In the malicious login case, we saw that only about 20% of the users in the study reacted to our baseline notification and changed their password.Hence, we conclude that login notifications can improve account security partially, as every password change can help to stop a malicious actor.However, at the same time, 80% of the participants in the malicious group who should have changed their password did not.While some participants might have decided not to change their password due to the study accounts' low value, it is still a high number, questioning the overall cost-benefit trade-off of the notifications.Arguments against Notifications On the cost side of things, we found several participants being annoyed, which is in line with research on fatigue in the context of security warnings and notifications [3,7].Another argument against the notifications is that they shift the responsibility for account security away from the service provider onto the user.In a sense, such notifications can be perceived as burdening and blaming [21,46].If service providers which hold exhaustive records about a user's login history are uncertain, why should the user be able to determine the legitimacy of a login?It is fair to say that in some cases users may know better whether i.e., their location has changed.However, in real life with shared accounts [8], third-party apps and services that automatically sign in to an account [1,40], or simply on busy days only few of us can realistically remember which accounts they used (note that we did not explicitly test these factors in our study).Thus expecting users to determine the legitimacy of a login better than a service provider is unfair.From a service provider's perspective, allowing logins and hedging them with a notification rather than blocking them makes sense; for them, it is the easiest "solution" to the problem.On a conceptual level, it all boils down to whether users should be made responsible for damages or if it should rather be the service provider's duty to implement robust security measures [27].
Arguments for Notifications Contrary to concerns about burdening users, it can be argued that users took the appropriate action in the legitimate case-namely, ignoring the notification.Our study showed that most participants correctly followed the instructions when prompted by a legitimate login notification and our qualitative results proved that they even correctly understand its meaning and cause.Additionally, some users felt more protected and satisfied when receiving such notifications.According to their qualitative feedback, such notifications' reassurance contributes to a positive user experience and reinforces trust in the service.

Refining Notifications
As indicated before, research and development should focus on refining notification systems to ensure their maximum effectiveness and usability.Past examples in the warning design space, i.e., TLS warnings, have demonstrated how improved warning designs can increase comprehension and adherence and decrease click-through rates [16].One approach to facilitate appropriate reactions may be to align notifications' implementations with users' understanding.As shown in Section 5, participants appear to expect and need contextual factors to determine what caused a notification.Especially, we saw significant differences in the helpfulness ratings of information between those in the malicious group, who changed their password, and those who did not.
While future research needs to investigate the exact root cause for this difference, we can certainly say that the information provided and users' ability to understand it correlates with their behavior in terms of password change.Malicious (No Change) participants, in particular, often misattributed or did not understand the cause of the notification-indicating that this information needs to be refined.Services could address this and adhere to users' expectations by including a distinct Why Notification component, e.g., by explicitly saying that a login happened from a previously unseen device.Our initial analysis of real-world notifications only found this contextualizing section in about 20% of the real-world notifications.Moreover, from a security standpoint, services need to provide more help than just suggesting to change the password.While password change is a first line of defense upon account compromise, it is by far not sufficient to ensure that the compromised account and other accounts are safe.Service providers should thus initiate a thorough remediation process, including expiring all sessions, reviewing third-party access, enabling 2FA, suggesting using a password manager, and checking related accounts [33,60].

Expectation vs. Fatigue
Besides the correct reaction and understanding of notificationsuser satisfaction with notifications is equally important.Fortunately, we found that more than 95% of the participants expect services to send login notifications, and only 3% do not want notifications to be sent at all, underlining an overall positive assessment.However, it is crucial to find a balance between sending them too often and too rarely.For service providers, sending notifications following a "better safe than sorry"-mentality may be tempting.Yet, for users, this leads to security warning fatigue [3,15,52].This fatigue is most likely caused by unnecessary login notifications, i.e., those that do not convey a real risk teach users that all notifications are unimportant.The situation is aggravated by services like Etsy, Git-Lab, Mozilla, Tumblr, and others that send notifications for each and every login.We saw that over 90% did not want to receive a notification on every login, and 15% even explicitly expressed "fatigue." Thus, we strongly dissuade sending notifications on every login.Instead, they should only be sent if the service suspects malicious account activity.Concretely, the majority the participants wants to be notified when a login takes place from a new device or location, and especially if a login appears "suspicious."Service providers can accomplish this with advanced logic provided by riskbased algorithms [62,63].Time-related notifications (i.e., a login after a long or at an unusual time) are less demanded.In favor of sparsity, time-related notifications should be omitted unless there is a concrete reason for suspecting malicious account activity.

Good Intentions & Questionable Advice
In the study, about 8% of the participants questioned the legitimacy of the notification or referred to it as phishing.Prior work explains how to best advise users on this topic [47], yet most of the 10 real-world notifications that include information about phishing do not follow the recommendations.While contradicting security advice, as well as no consensus among security experts about its prioritization, is nothing new in the community [43,45], for us, it was surprising how potentially dangerous and obsolete some of the given advice is (see Section 3.3).For example, many large services like X (Twitter), Spotify, and Amazon portrayed the padlock icon of the browser as a type of trust and legitimacy indicator.While this is not only false, in early September 2023, Google removed the padlock icon with Chrome 117, as HTTPS should be considered the default state [2].

Recommendations
Based on our findings, we give some recommendations for service providers below.
Notify about Devices, Locations, and Suspicious Logins We advise against sending notifications after every login, mainly because some of our participants reported being annoyed by the frequency of real-world services sending notifications (see Section 5.2).Instead, we recommend that services send login notifications when a login takes place from a new device or location, and especially if a login appears "suspicious." Describe What Happened What triggers a notification, e.g., an "unusual login, " is often unclear to participants (see Section 5.1).Services could easily address this issue by explaining what triggered the notification, yet only 21% of the evaluated emails currently provide examples of common triggers (see Section 3.3).Explaining the circumstances would also help to create context, which is especially important when users receive unexpected notifications and struggle to assess the situation correctly.
Include Information in Metadata We found that the metadata is an influential factor, and 75% of the participants paid attention to the email subject (see Section 5.2).Hence, in addition to the most important information, the email subject should already provide context for deciding how to react.Currently, only 15% of our analyzed notifications make use of subjects like "New login to Instagram from {browser} on {OS}" (see Section 3.3).Similarly, websites should make use of the email sender's name so that recipients can quickly parse the information about the sender.
Specify Instructions Based on the findings of our email analysis (see Section 3.3), notifications should include instructions for both outcomes, i.e., legitimate and malicious logins.For the legitimate case, most services suggest to ignore the message.For malicious logins, the recommendation needs to prompt users to visit the website and change or reset the password or, even better, initiate a thorough remediation process.Most services facilitate this by including a link, which is a controversial practice.However, 8% of the participants were suspicious, some of them due to the presence of a link, and did not change their password.
Provide Comprehensible Details We found that all types of information (account name, location, time, and device) have a positive influence (see Section 5.2).Still, services need to be careful when it comes to parsing and displaying technical details such as location data, browser, and OS (see Section 3.3).Here, special care and testing are required, as a badly parsed or displayed detail could impact the overall perceived legitimacy of the notification.
By addressing the identified areas, service providers can continue to strengthen account security and foster user trust.

CONCLUSION
We explored users' comprehension, reactions, and expectations of login notifications that are sent by services to help users protect their accounts from unauthorized access.
In a three-stages user study ( = 229), we evaluated a baseline notification that was created by collecting and analyzing 72 notifications sent by real-world services.To prevent participants from spending most of their attention on the notification and authentication task, we introduced a strong cognitive distractor by implementing a mental rotation test.We split participants into two treatments: a) The legitimate group received a notification only after they logged in, using metadata derived from their login information.b) The malicious group received a notification unexpectedly at a time when they had not interacted with the study website for multiple days using a generic location ("California, USA") and device ("Chrome on Windows") with the highest statistical chance of matching any user in our U.S.-based sample.
Overall, we find that login notifications achieve their goal of increasing account security.However, the tested notification failed to convince the majority of participants to change their password.Participants expressed the need for more contextual factors to help determine what caused the notification.Thus, instead of talking about an "unusual login," services need to explain what triggered the notification to assist users who received the notification unexpectedly.Even though some participants expressed feeling more protected and satisfied after receiving a notification, we argue that the service and not the user must be held accountable for implementing robust account protection measures.Interestingly, we find that more than 90% of the participants expect services to send login notifications when a login takes place from a new device or location, and especially if a login appears "suspicious." However, participants also expressed that they do not want to be notified for every login, highlighting the importance of finding the right balance between sending notifications too often or too rarely.

C REAL-WORLD NOTIFICATIONS: FEATURES
(a) Granted Access from Twitch.(b) Additional Challenge from Amazon.(c) Blocked Access from Google.

Figure 2 :
Figure 2: Real-world examples of the three sign-in notification types (logos removed due to copyright, cropped, as of Dec. 2023).

Figure 3 :
Figure 3: The information included in login notifications for granted access notifications ( = 72) sent by real-world services.

Figure 4 :
Figure 4: The baseline login notification, which we derived from ( = 72) real-world notifications.For our user study, we rebranded the text and the look to match the study website.

( 2 )
I-PANAS-SF:To learn about the feelings and emotions in reaction to the notification, in MP we utilized the Positive and Negative Affect Schedule (I-PANAS-SF)[54].(3)Reaction: Next, we asked how thoroughly participants read the notification (MQ1) and how and why they chose to react to it (MQ2a-MQ3a).Participants who changed their password were specifically asked about any other actions (MQ2b-MQ3b).(4) Content & Design: To better understand the reactions, MQ4 asked about influencing factors like metadata, content, and design.MQ5 specifically asked about the helpfulness of the account name, location, date, and device.(5) Time & Location: MQ6-MQ10 investigated the time when and location where the notification was read.With MQ7, we verified if the location, which had been derived automatically, was actually accurate or could have led to confusion, and MAC2 was an attention check.(6) Comprehension & Expectation: With MQ11, we captured if par-

Figure 5 :
Figure 5: Breakdown of treatments into participants who have or have not changed their password and their reasoning.
significance criterion of  = .05.After filtering 12 participants who failed the attention check (MAC1), 613 participants remained.At the end of Stage 3, we had 252 completions.This high number of dropouts is almost exclusively attributed to participants who did not return after the first stage of the study.Lastly, we removed 23 participants who provided unrelated answers or failed the second attention check (MAC2) for a final number of  = 229.Stage 1 took, on average, 2.5 minutes and was compensated with $3.00 USD.Stage 3 took, on average, 6 minutes and was compensated with $4.00 USD.Participants in the legitimate group received an additional $1.00 USD for the completion of Stage 2, which took 2 minutes on average.Table

Table 3 :
Information contained in notifications sent by real-world services.