"Make Them Change it Every Week!": A Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication

Usable and secure authentication on the web and beyond is mission-critical. While password-based authentication is still widespread, users have trouble dealing with potentially hundreds of online accounts and their passwords. Alternatives or extensions such as multi-factor authentication have their own challenges and find only limited adoption. Finding the right balance between security and usability is challenging for developers. Previous work found that developers use online resources to inform security decisions when writing code. Similar to other areas, lots of authentication advice for developers is available online, including blog posts, discussions on Stack Overflow, research papers, or guidelines by institutions like OWASP or NIST. We are the first to explore developer advice on authentication that affects usable security for end-users. Based on a survey with 18 professional web developers, we obtained 406 documents and qualitatively analyzed 272 contained pieces of advice in depth. We aim to understand the accessibility and quality of online advice and provide insights into how online advice might contribute to (in)secure and (un)usable authentication. We find that advice is scattered and that finding recommendable, consistent advice is a challenge for developers, among others. The most common advice is for password-based authentication, but little for more modern alternatives. Unfortunately, many pieces of advice are debatable (e.g., complex password policies), outdated (e.g., enforcing regular password changes), or contradicting and might lead to unusable or insecure authentication. Based on our findings, we make recommendations for developers, advice providers, official institutions, and academia on how to improve online advice for developers.


INTRODUCTION
Password-based authentication is the status quo on the web and beyond [6,13,14]-despite its many known problems.This includes challenges around memorizing passwords, password reuse [35,86,107], easily guessable and insecurely stored passwords [55,107], being forced to follow complicated password policies and change passwords regularly [55,60,102,103], or phishing attacks [57].Eventually, passwords are a prime example of the importance of usable security: The underlying problem is not the technical security of passwords but the overwhelming challenges they present to users that negatively impact authentication security.Replacing passwords with potentially more usable and secure alternatives has been a popular idea for many years [13].However, alternatives such as two-factor authentication (2FA) and FIDO2 both lack adoption [8,18,24,32,64,87], and passwords still remain the de facto standard.
Overall, companies, development teams, individual developers, and other involved stakeholders significantly impact the usable security of software systems in general and authentication in particular [41].For example, they make decisions on password composition policies or the deployment of 2FA.To inform security-related software development decisions, including decisions on authentication deployments, developers draw heavily on online security advice, e.g., from Stack Overflow (SO) [2,3,5,16,27,28,29,73,120].In the ACM CCS 2022 keynote, Michelle Mazurek concluded that security advice is a "disaster" and affects developers and other stakeholders [68].As developers draw on security online advice and online advice on authentication is common, the question of what developers can learn from such advice to make authentication deployments usable and secure was the main driver of this work.Based on a qualitative exploration of the status quo of online security advice on authentication deployments for developers, we aimed to develop further ideas for improving future advice.Previous work on general security advice for end-users [15,23,46,52,89,91,93] and for developers [2,3,5,10,11,16,27,28,29,31,39,76,92,120] does not address online security advice for developers on implementing usable security for end-users.To the best of our knowledge, we are the first to explore this research gap.
Goals.In this paper, we aim to qualitatively explore developer security advice on implementing authentication that affects enduser usable security.We aim to understand the accessibility and quality of online advice and to provide insights into how online advice might contribute to (un)usable and (in)secure authentication.More specifically, we seek to explore which topics are covered and identify potential challenges developers might face when following advice on implementing authentication online.Based on our findings, we make recommendations for future advice to better support developers in implementing usable and secure authentication.
Approach.In our qualitative exploratory in-depth analysis, we applied the following multi-step process: First, we collected online documents that contained advice on authentication.Therefore, we recruited 18 professional web developers to collect a set of 264 online resources (Sect.3) and their Google search queries.We used these queries to enrich the document corpus further.The overall document corpus consists of 406 documents on web authentication.Second, three researchers extracted all advice on web authentication from each document through qualitative open coding (Sect.4.2).Finally, in an in-depth qualitative analysis, we explored advice distribution (Sect.4.3) and topics (Sect.4.4) and investigated characteristics like recommendable (Sect.4.5), debatable (Sect.4.6), and contradicting advice (Sect.4.7).
Key Findings.The advice we found roughly reflected current (usable) security authentication challenges on the web, e.g., passwords are still ubiquitous, and password advice was most prevalent (51.6% of documents).For example, the advice covered password composition policies or password recovery.Moreover, only about 22.5% of advice was relevant to usable security of authentication.We classified 31.6% to be recommendable.However, we also found 18.0% to be debatable, e.g., being outdated, factually wrong, or counteracting usable security (e.g., enforcing regular password changes).We identified 15 groups of two or more pieces of advice that contradict each other.Overall, we found that advice from research results and official institutions was rarely integrated into the advice we analyzed.Furthermore, the advice was scattered across many online resources, making it difficult for developers to find.
Contributions.In this paper, we make the following contributions: (i) To the best of our knowledge, this is the first study investigating online advice for developers on authentication that affects security and usability for end-users.(ii) We present an overview of authentication advice topics and further explore recommendable, debatable, and contradicting advice.(iii) Based on our findings, we make recommendations for future usable security advice for developers.Furthermore, we highlight open challenges and outline future research.(iv) Finally, we provide our document corpus, all pieces of advice, and additional supplementary material in a replication package (Appendix A).

RELATED WORK
We discuss related work in three key areas: (i) general security advice for end-users, (ii) (usable) security advice for developers, and (iii) usable and secure web authentication and its challenges.We highlight our novel contributions and put it into context.General End-User Security Advice.A large portion of online advice on security addresses end-users [15,23,47,52,72,89,91], e.g., concerning topics like how to handle passwords or spam.Nonexperts, like most end-users, heavily request such advice [46].As lots of advice exists, especially online, the selection of advice sources has to be considered [89].For example, factors like the source's perceived trustworthiness [89] and convenience [23] influence advice adoption.One explanation for low advice adoption is that users rationally reject advice as it often offers a poor cost-benefit tradeoff [47].Overall, security advice can be challenging to adopt, as a mismatch between advice and behavior was perceived among both experts and non-experts [15,52].Even security experts were found to lack consensus for advice [93].Similarly, users and experts (like administrators) struggle with advice prioritization [91].
While advice can be used to educate end-users, we argue that it is (at least equally) important to help software professionals design systems that people can use securely-ideally without users needing any advice.Therefore, we focus on advice for software developers.(Usable) Security Advice for Developers.In recent years, researching human factors and security with a focus on developers has gained popularity [4,40,88,110].This is also true for research on security advice for developers.Several studies found that advice impacts security and that sources containing insecure code find their way into software [2,3,5].For example, Acar et al. found that developers who used SO produced significantly less secure code and that only 17% of SO posts contained secure code snippets.Insecure code from SO is also prevalent in many Android apps [27] and contained in top Google search results [28].However, nudging [29] and re-ranking search results [28] can create a positive security impact of SO.Overall, SO is a place for software professionals to discuss security-relevant topics [120], while secure and insecure answers were almost balanced [16].Apart from SO, researchers also found challenges in other advice sources, e.g., not actionable IoT advice [10,11].
Advice can also be given through other channels.This includes IDE plug-ins that aid developers in preventing security pitfalls.For example, Nguyen et al. presented and evaluated a plug-in that supports Android developers [76].Similar tools exist for other use cases, e.g., secure password storage [31].Besides such tools, Gorski et al. found that integrating security advice into application programming interfaces (APIs) can significantly improve code security without affecting API usability [39].Other advice forms can be more general by outlining principles (e.g., Microsoft's NEAT design principles for security warnings [92]) and abstract guidance (e.g., heuristics for accessibility and usable security on the web [74]).
Most similar to our work, Iacono, Gorski, et al. systematized security principles, guidelines, and patterns based on scientific literature [38,62], but only "concentrated on principles and patterns for end-users" [38].In contrast, we focus especially on advice for developers that affects end-users' usable security.Moreover, we focus specifically on (non-scientific) online resources.To the best of our knowledge, no paper examined advice for developers that covers end-user usable security.We argue that end-user (usable) security, however, highly depends on decisions and implementation during software development.Our analysis addresses this gap.
Usable and Secure Web Authentication.Passwords, probably the oldest authentication approach [14] and a starting point for early usable security research [6], are the most used authentication approach despite many shortcomings.This includes but is not limited to insecure storage [55,107], password reuse [35,86,107], and usability and (resulting) security problems through password policies [55,60,101,102,103].To aid in creating secure passwords, password strength meters are a common tool and visualization technique [112].However, their construction is difficult.For example, researchers found inconsistencies, weaknesses [21], and accuracy issues [33,115].Similarly, password managers (PWMs) should help users deal with their passwords, e.g., by creating secure (random) passwords [121] that fit websites' password policies [30].However, even PWMs have interaction problems with websites [48] or overwhelm their users [79], both limiting their usability.Due to passwords' numerous problems, the idea of replacing them became popular.However, many approaches failed [13], and replacing passwords remains an unsolved challenge.
For example, replacing passwords could be achieved with the FIDO2 standard [26].However, FIDO2 lacks adoption, which might be caused by overlooked benefits [24], concerns, and not trusting the new technology despite good usability [32].Despite generally good usability, users struggle with hardware token setup [19,96] or rate the usability of smartphones for FIDO2 as low [83].As tokens can get lost and are associated with costs, electronic IDs can be used for FIDO2 [100]-albeit being hard to set up [53].
2FA, or generally multi-factor authentication (MFA), is an important concept for strengthening security-especially when using passwords-but could also negatively affect usability [1,94,95].Users might have problems with the initial setup [94], and the 2FA user experience (UX) is inconsistent [64].Overall, 2FA adoption is still relatively low and needs to be increased [8,18,87,90].Widely adopted approaches like SMS-based 2FA are known to be vulnerable to phishing [66].
To conclude, authentication has pitfalls that software developers need to consider.Interestingly, most outlined challenges are no technical security problems and boil down to a usability problem.For example, passwords would be more secure if used correctly.Therefore, we investigate how developer advice supports usable and secure authentication.

COLLECTING ONLINE DEVELOPER ADVICE
In this section, we describe our data collection of 406 online resources on authentication.While we aimed to create an ecologically valid dataset, our main goal was diversity due to the exploratory nature of our work.In a developer survey between August and November 2021, 18 professional web developers searched and reported online resources on web authentication they considered for an imaginary web authentication development task.We extend the document corpus further by including search results for our participants' search queries.As we had already reached saturation with this document corpus, we stopped recruiting more participants.

Developer Survey
Our goal was to create a comprehensive corpus of documents containing online advice regarding usable and secure web authentication for developers that impacts usable security for end-users.We asked professional software developers to search for online resources they consider relevant for implementing usable and secure web authentication.We gave them a scenario and task to search for relevant advice on the web.
3.1.1Scenario.We created a task containing a web development scenario and framed it towards authentication and usable security.We aimed to collect document URLs, search queries, and other information from software developers. 1 Our scenario consisted of a hypothetical new web-based health platform KeepYourHealth, that should become the central pivot point for personal health, e.g., storing and sharing medical data or communicating with doctors.We chose the healthcare framing for three reasons: First, medical and personal health information is considered highly sensitive, hopefully shifting the participants' attention to security.Second, to highlight the need for good usability, medical applications have a diverse user base that likely contains many non-tech-savvy users.Third, medical applications must follow certain compliance or data protection rules (e.g., HIPAA in the US, PIPEDA in Canada, GDPR in Europe).While this might skew the results slightly towards official documents, we also hoped to receive a more diverse and complete document sample.
As part of the KeepYourHealth project, we wanted participants to imagine that they are responsible for designing and implementing the authentication mechanism for both a mobile and a desktop version.To create the concept for KeepYourHealth's authentication system (including registration and login), the participants had to perform online research and look for helpful resources.We designed the scenario to be very open to ensure diverse results, as participants can explore various topics in their search.Moreover, we instructed participants to freely consider any resource, authentication approach, technology, etc.Our only requirement was that the authentication system had to be usable and secure.We used words like easy-to-use, user-friendly, secure, usability, and security to emphasize the requirement for usable security.

Search Task.
To collect online advice, we gave software professionals the task of searching for online resources that provide advice that potentially helps to build a usable and secure web authentication system for KeepYourHealth.We explained our research project's purpose and that we wanted to collect and analyze advice from various online sources.We additionally explained that we, as researchers, are biased and probably would search differently than web development experts.We also wanted the software developers to report how they found a specific online source, i.e., the search query.We provided the participants with a table to report documents and search queries.The table consisted of the following columns (cf. Figure 4): URL, how the document was found (e.g., bookmark, link within another site, search engine), and (if applicable) the search query that yielded the document.Each column's purpose was also explained.Our participants were free to choose any search approach they wanted, i.e., we did not constrain them to a specific search engine.
3.1.3Questionnaire.We embedded the search task in an online questionnaire.After an initial consent form, the participants were introduced to the scenario and their tasks.We gave them 30 minutes for the task.For the search task, we displayed the above-mentioned table for the users to enter URLs and search queries.After completing the task, we asked the participants for their most used search engine and to rank different information resource types by their helpfulness.Next, we followed up with standard demographic questions, including age, gender, country of residence, highest level of education, and employment status.Additionally, we asked some questions specific to this study's context regarding participants' experience, skills, and security and usability education.We estimated the maximum time for the whole survey at 45 minutes.The replication package (Appendix A) includes the full questionnaire.

Instrument Development & Piloting.
We developed and piloted the questionnaire, scenario, and task in multiple iterations.We asked nine persons to pilot the study and questionnaire, including researchers with experience conducting usable security studies and experiments.We made smaller adjustments regarding wording and description texts based on the feedback that we collected via thinkaloud and feedback text boxes on each survey page.Finally, we tested the whole setup with three professional developers until no further adjustments were necessary.During piloting, we collected 48 documents and 38 search queries.We included those in our final document corpus to enhance its diversity further.

3.1.5
Ethics & Data Protection.We received ethical review board (ERB) approval for this study from one of the involved institutions, concluding that there were no ethical concerns.The remaining institutions had neither an institutional review board (IRB) nor an ERB that applied to our study.This study was designed to adhere to the ethical research principles of the Menlo Report [54].We further adhered to the strict German and EU privacy laws.All participants consented to collecting, storing, and analyzing their data for this research.We clarified that we would only evaluate and publish de-identified and aggregated data.Furthermore, we informed all participants that they could terminate the study at any time.
3.1.6Recruiting.We aimed to recruit a diverse sample of software developers with web development experience.Therefore, we used the freelancer platform Upwork.We created an Upwork job post describing our study and communicated the idea of our research.Participants were required to have sufficient proficiency in English.Besides that, we checked the Upworkers' profiles and job history to ensure they have web development experience.We also directly invited Upworkers who had indicated experience in web development in their profiles.The job was rewarded with $45 (hourly wage of $60), above Upwork's $15-$30 median hourly pay for software developers [111] to offer competitive compensation.
3.1.7Demographics.We recruited 18 web developers via Upwork whose demographics are comparable with those of professional developers from SO's latest developer survey [105].An overview of participants' demographics can be found in Table 1.The participants were from 11 different countries, with the UK and the U.S. being the most common, including others such as Canada, Nepal, Romania, Germany, Nigeria, France, and Turkey.The majority had at least a bachelor's degree.Their professional software development experience varied between 1.0 and 22.0 years.Participants covered various roles, such as software engineers, web developers, architects, team leads, and CTOs.Most participants reported some experience with usable security.Some participants also received education in this area in their training, studies, or professional career.We collected 145 distinct search queries from our participants.All participants reported having used Google for their web searches.The most frequent terms and term bigrams are shown in Tables 2b  and 2c.While authentication is the most frequent term, participants searched specifically for security aspects, best practices, and usability aspects.This also indicates that our task worked as intended, as the terms reflect the goal of usable and secure authentication.
Summary: Initial Document Collection.The search task with 18 professional web developers resulted in 264 documents and 145 search queries.Both document domains and search queries show high diversity.

Search Expansion
We extended our document corpus with search results for the collected search queries that participants reported in addition to the directly reported documents.We performed this search expansion because participants might have missed relevant documents to counterbalance participants' personalized search results.Moreover, this increases overall diversity and helps to reach saturation in the best case, while adding no additional insights in the worst case.For each query, we obtained the corresponding search results to expand our document corpus.Therefore, we leveraged Google's Custom Search API [37] to execute a search for each query and extracted its top 10 results, i.e., its URL and some metadata.We chose this Google-based engine, as Google is the most used search engine [75,106] and all participants reported that they use Google as one of their main search engines.We chose 10 results per query, as this corresponds to the first page of search results.Moreover, the clickthrough rate (CTR) drops below 10% after the fourth and quickly decreases further to almost 0% for lower-ranked results [84].That said, almost no user looks at the second page of results [20].A top-10 expansion, therefore, covers search results a user potentially clicks while also including a safety margin.

Final Document Collection
The search task with professional software developers resulted in 264 distinct documents and 145 distinct search queries.As our goal for this qualitative study is diversity, we decided also to include all 48 documents and 38 search queries from piloting.As described above, we extend our corpus with documents found using Google.We randomly sampled 100 search results from the search expansion and included those in the final document corpus.Combining all these documents, we obtained a final dataset of 406 documents overall.For the documents that indicate a publishing date, the median year for the last update was 2021 (oldest: 2002, newest: 2021).We downloaded each as a PDF for our analysis.
We stopped after these 406 documents, as we had already reached saturation, which means that we would not gain new insights from analyzing more documents and advice, i.e., no new themes emerged in Sect. 4 (e.g., advice topics).Repeated advice in multiple documents also indicated saturation.Consequently, we did not survey further participants.
In the survey, we asked our participants about the helpfulness of different information sources (cf. Figure 1).Overall, participants reported a high affinity towards online resources.The top-three sources were official documentation, blog/news articles, and online communities.Similarly, this is also reflected in the different types of documents (Table 3).We found blog posts (some including news) to be the most common type, followed by documentation and wikis and project websites (including source code repositories).Surprisingly, only 15 documents (3.7%) were from Q&A forums, 13 of them from SO (cf.discussion in Sect.5.4).Minor document types included educational material and courses as well as video and audio resources.6 documents had no clear type.4 (1.0%) were not available online anymore when we downloaded the documents.
Summary: Document Corpus.Our final document corpus consisted of 406 distinct documents which we downloaded for analysis.Blog posts, documentation, and wiki pages were the most common document types.

Limitations
This work has some limitations that are typical for this type of study.Generally, the developer survey is subject to self-reporting, social desirability, under-and over-reporting, and sampling biases.We note that our document corpus cannot contain all online resources and that it is influenced by the participants' experience and especially the personalization of their search results.We addressed this by recruiting a diverse set of participants and extending the document corpus with search results until we reached saturation.Furthermore, we received only English documents as it was the study's language.We decided that English as the only language is an acceptable trade-off as it is the main language used in software development (e.g., documentation or SO).The search task and scenario (Sect.3.1) itself could bias the document corpus.However, we favored a realistic scenario without many requirements for external validity over a general, undirected search task.

ADVICE ANALYSIS
In this section, we provide details on the advice analysis and its results.Our analysis is a multi-stage process that starts with identifying and extracting advice (Sect.4.2) before analyzing it in detail.
Throughout the paper, we report counts, e.g., counts of advice or documents.As this is qualitative work, those counts are used to convey weight and should not be interpreted as quantitative statistical results.While we try to give an overview by discussing advice examples, we cannot describe and discuss every single piece of advice due to their high number.

Positionality Statement
Before giving detailed insights, we want to highlight that this is qualitative research.While we always involved multiple researchers and based decisions on external references where possible, this study's design and results might be influenced by the involved researchers.That said, the results are not guaranteed to be entirely objective and should be interpreted in context.Therefore, we aim to clarify our backgrounds with this positionality statement.All authors are usable security researchers with varying levels of experience, ranging from third-year Ph.D. student, to tenured faculty and full professors with more than 30 years usable security research experience.Our study backgrounds cover computer science, IT security, mathematics, and psychology.The authors are based in Germany.

Identifying & Extracting Advice
Below, we describe our advice identification and extraction approach in detail.It consists of two steps: (1) First, we extracted and systematized each piece of advice from all documents through qualitative coding.
(2) Second, we applied inclusion and exclusion criteria to reduce the set of advice to address usable and secure web authentication relevant to our study.

Advice Extraction.
We extracted each advice for developers by reading all documents and manually assigning codes that summarize each piece of advice.We used the following definitions to detect advice.The definitions' goal was to make the extraction process transparent, objective, and reproducible.
• (Concrete) Advice: We considered explicit prompts to do something in a specific way or specific recommendations concrete advice.Phrases like "we recommend, " "a developer should, " or imperatives like "ensure" often indicate advice, which the researchers therefore used as proxies to detect advice.An example of such a concrete recommendation is: "Allow users to toggle password visibility when typing it."• Descriptive Information: We deliberately refrained from extracting information as advice that was not explicitly directed at the reader or any other form of specific recommended action.Some document passages just inform and describe, leave room for interpretation, and require reading between the lines.It might be unclear whether the resource's author wanted to give advice, whether a reader perceives it to be advice, and how one might interpret it.
Three researchers each annotated a third of the final dataset in an iterative open coding process.As advice in the documents can be long and wordy, we summarized it by creating codes that briefly paraphrase the advice [56,67].Table 4 shows examples of original document excerpts and our assigned advice codes.As shown in the table, different text passages can have the same meaning and were therefore assigned the same advice code (examples 3 and 4).We used these advice codes for our analyses throughout the paper.Each researcher grouped similar or related paraphrases in a code system.The leaf nodes in the code tree contain the advice.The inner nodes are used to build a hierarchy (e.g., group all advice on passwords).After coding sets of 50 documents, at least two coders merged, refined, and regrouped the code system if necessary; all three reviewed the resulting code system.The coders repeated the described process eight times until advice from all 406 documents was extracted.In accordance with McDonald et al. [70], we refrain from reporting inter-rater reliability (IRR) because extracting only concrete advice leaves little room for the researchers' own interpretations and because the following analysis of the advice (following subsections) is our main contribution-not the code system itself.
As we focus on non-scientific online resources, we excluded 31 (7.6%) documents being scientific papers; 4 (1.0%) documents because they contained only a video or audio guide; another 29 (7.1%) documents were not related to our research topic (i.e., not about authentication and usable security); and two documents because they were not available anymore or a duplicate of an existing document.119 documents contained no advice at all.For the remaining 221 documents, we identified 1,208 distinct pieces of advice.

Inclusion and Exclusion
Criteria.In the previous step, we extracted all advice for developers and obtained 1,208 paraphrased pieces of advice.We defined inclusion and exclusion criteria to focus only on advice relevant to this paper's scope.We excluded advice if any of the following criteria were met: (1) if the advice  was not related to security, (2) if the advice was not related to usability aspects and end-users, or (3) if the advice did not cover authentication.The remaining advice was included as it is relevant for usable security of web authentication.Three researchers did this assessment independently and later resolved conflicts until they reached consensus.
Advice Inclusion.272 distinct pieces of advice out of the initial 1,208 covered usable security for web authentication and were therefore included for further analysis.For example, the advice to "Use a minimum of 8 characters" for passwords matched our inclusion criteria because passwords are a security and authentication feature, and their length affects usability [55,101,102,103].Overall, the 272 pieces of advice matching the inclusion criteria occurred 655 times across 124 different documents.Hence, only a small fraction of the online advice considers or affects usable security.Instead, most advice is only about security-not considering usability and human factors of security at all.Advice Exclusion.According to the above-mentioned criteria, we excluded 936 pieces of advice from further analysis because they did not cover usable security of web authentication, thus not being relevant for our research goals for one or multiple of the following reasons.We excluded 257 distinct pieces that did not cover security, e.g., "Provide live form validation" or "Only use singlecolumn forms".Similarly, we excluded 669 distinct pieces of advice that were not relevant for end-users or usability.For example, "Use object-relational mapping (ORM) to prevent SQL injections" or "Use bearer tokens," are technical security details that users do not notice.We excluded another 73 pieces of advice that covered usability and security but were not relevant to web authentication, e.g., "Limit upload size for users".We note that we focussed our analysis on advice in prose text and excluded any code snippets (occurring in 102 documents).
Summary: Identifying & Extracting Advice.We found and extracted 1,208 pieces of advice in 406 documents.Focusing on web authentication advice with relation to usable security, we came up with 272 pieces of advice from 124 documents for further analysis.

Advice Distribution & Density
As already stated, the advice was widely distributed across 124 documents.A document contained 4.5 different pieces of advice on   average (median: 2.0).However, the distribution was overall skewed (cf. Figure 2a): while the document with most advice contained 35 different pieces of advice, the median was only 2.0 different pieces of advice per document.That said, most documents provided only little advice, while a few were much denser.This was similar for the domains on which advice is provided.While individual domains (that contain advice at all) provided on average and median only 6.6 and 3.0 different pieces of advice, respectively, the OWASP Cheat Sheets contained most, having 55 different advice imperatives (see Table 5).Considering that mobile-security.gitbook.ioby OWASP was also among the top domains, OWASP resources contained by far the most advice.However, authentication provider swoopnow.com'sblog and the developer portals web.dev and dzone.comalso contained a considerable amount of advice.While most documents contained little advice, developers are not limited to considering only a single document.In line with that, our participants found 26.3 different pieces of advice on average among multiple documents (median: 13.0).81 different pieces of advice was the highest number of different advice imperatives found by a participant.Looking into multiple information sources is also likely to yield new advice because 176 of 272 pieces of advice were only included in one document (cf. Figure 2b).On average, a piece of advice was only contained in 2.1 documents (median: 1.0).However, a few pieces of advice were covered in a higher number of documents, as shown in Table 6.Offer/Use 2FA/MFA was the most frequent one, being contained in 35 documents.
All in all, advice is scattered, and most documents contained little advice, with a few exceptions.Hence, developers must consider multiple documents if they want to get a comprehensive set of advice.Looking at a single document would yield only a small subset of advice if not looking at one of the rare dense documents.
Summary: Advice Distribution & Density.While documents contained 4.5 pieces of advice on average, most only contained two or less.However, some documents like OWASP Cheat Sheets were denser.176 of 272 pieces of advice were only covered by a single document.

Topics of Web Authentication Advice
Early on, we noticed that the advice covered various topics.For further investigation, we took a closer look and identified 12 different topics in our dataset's advice.

Identifying
Topics.Three researchers independently used an inductive coding strategy to identify advice topics within the 272 pieces of advice and assign each advice paraphrase to its respective topic(s).Afterward, the researchers discussed each individual topic in a consensus session to create the final list of topics.It consists of 12 different topics (cf.Table 7).We assigned each topic a definition to achieve a reproducible assignment for each piece of advice and describe the topic.With the final list of topics, each coder independently coded all 272 pieces of advice.After the independent coding, the coders finally compared and resolved existing conflicts until they reached consensus.We allowed assigning multiple topics if a piece of advice fits several topics at the same time.For example, the advice to "Use password strength meters" covers passwords and also implies user interface (UI)/UX requirements.

Results
. Overall, we found 12 different advice topics, as depicted in Table 7 (including each topic's definition, example advice, and prevalence counts).Within the 12 topics, we found five topics corresponding to major authentication types: passwords, 2FA/MFA, single sign-on (SSO)/3rd-party login, token-based authentication, and biometrics.Besides these five, some minor authentication types emerged that we grouped into an other category (e.g., TLS client certificates).Advice on passwords forms the most common authentication type, with 101 distinct pieces of advice across 64 documents.
The topic covers several aspects, like password policies, reset, entry, managers, or strength meters, and includes usability aspects, e.g., toggling visibility in password fields.Advice on 2FA and MFA is also popular (54 documents), followed by advice on SSO, third-party, and social logins.Token-based and biometric authentication were only rarely discussed.The remaining six topics were more general and not about a specific authentication type.Usage recommendations, i.e., what authentication mechanism to use, were very common (83 documents).Another topic spanning all authentication types was session and account management (e.g., account recovery).The remaining minor topics were about user and usability aspects.They covered general authentication UI and UX, how to interact with the user through notifications and messages, or how to educate users.7 within the same document.
The co-occurrence heatmap of topics within a document in Figure 3 reveals that authentication types are often not jointly covered within the same document.For example, while password advice is in 64 documents, only in 28 documents both passwords and 2FA/MFA are discussed together.This is similar for passwords and SSO/3rd-party login.On average, a document touches 3.2 different topics (median: 3.0).However, when just focusing on the authentication type topic subset, only 1.7 different authentication types are discussed.61 documents cover only one authentication type.
Summary: Advice Topics.Overall, we found a wide range of 12 different advice topics.Advice on passwords was by far the most frequent, followed by 2FA/MFA.The topics are often discussed in isolation, e.g., not discussing security benefits of 2FA when talking about passwords.

Recommendable Advice
Below, we provide details on advice we deem recommendable for developers.We consider authentication advice recommendable if it contributes to usable and secure authentication on the web.
Based on these literature references, two authors assessed for each advice whether it could be considered recommendable.We used discussions to obtain consensus decisions.For transparency, we provide supporting references with their publication years to

Passwords
Includes all advice, which directly mentioned a policy, a feature, or an implementation detail, that is directly or indirectly related to the creation, change, or processing of passwords.
• Enforce a minimum password length of 8 characters.
• Reject passwords from breaches.

101 64
2FA/MFA Advice related to two-factor or in general multi-factor authentication and the different approaches but also backup and recovery and general considerations.
• Require adding a 2nd factor for features with higher security.

SSO/3rd-Party Login
Recommendations related to the implementation or use of SSO, 3rd-party authentication services, or social logins, e.g., via Facebook or Google.
• Use OAuth 2.0 • Stick to a few common social/third-party providers.

Token-Based Authentication
Authentication mechanisms based on tokens, their design, and usage.This also includes token formats such as Bearer or JWT.Cookie-and session-based are authentication also included.
• Give tokens an expiration.
• Limit number of token usages.

Biometrics
Includes all advice which directly is related to any kind of biometric authentication method.
• Prefer fingerprint scanning above facial recognition.
Other Advice related to other authentication types, not belonging to one of the other authentication types.
• Do not use TLS Client Authentication for public websites.

Usage Recommendations
Advice on how to choose or which authentication method to use.
• Use risk-based auth controls.

Session/Account Management
Advice on management of sessions and accounts, including login and logout process, account creation/registration, recovery, blocking, user identifiers, etc.
• Require re-authentication for sensitive actions (step-up auth).
• Invalidate all sessions after password change.

103 63
UI/UX Concrete request to change, integrate or implement features and elements in the UI in a specific way, e.g., how to design a password field.Also includes advice that directly mentions some aspect of user experience.
• Use generic response code for auth/reg/recovery errors • Don't prohibit pasting passwords.

Notifications, Error-, & Warning Messages
Advice on informing the user through notifications and messages in general, their design and usage, including push notifications, email, or SMS/text message, but also warnings, error messages.
• Notify users via mail when password was reset.

User Education & Engagement
Advice on engaging or educating users on different topics of authentication.
• Educate users to create strong passwords.
• Encourage users to accept passwords from password managers.

7 8
Miscellaneous Includes all advice that does not fit into any other category.
• Use captchas after failed attempts.
• Involve stakeholders early on for security requirements.indicate their age.Additionally, we provide written rationales that explain our reasoning within our replication package (Appendix A).We considered 40 pieces of advice recommendable without having a reference or scientific evidence.Therefore, we excluded this advice in the results below and call it potentially recommendable.Potentially recommendable advice includes advice such as "Use social login if building apps on top of Google or Facebook" or "Avoid login completely if not necessary".For completeness, we provide the full list of potentially recommendable in the appendix, Table 10.

Results
. We considered 86 of 272 pieces of advice (31.6%) recommendable.Overall, 92 documents (74.2%) contained recommendable advice.Table 8 provides an overview of the most frequent recommendable advice.Comparing this to the overall most frequent advice (Table 6) shows some minor differences.For example, we did not consider the advice to "Require strong passwords" recommendable because it does not provide concrete guidance.However, we consider the related advice to "Enforce a password policy" recommendable since official institutions generally recommend password policies [77 (2017)], and there is scientific support for the use of password policies (while the concrete policy should be adapted to the context) [55 (2011), 60 (2022)].
Among the advice that is only contained in one or two documents, we found highly recommendable advice for developers.For example, "Don't prohibit pasting passwords" (to not interfere with PWMs [48 (2021)]) or to "Provide a password change feature."The latter is crucial to change compromised passwords, but to our surprise only mentioned in two documents-maybe because it is too obvious.Advice like "Don't force users to regularly change passwords" Summary: Recommendable Advice.We found 31.6% of the advice to be generally recommendable to developers, as it adheres to state-of-theart research results or official guidelines.Additionally, we consider 14.7% of advice potentially recommendable that lacked references for evidence.

Debatable Advice
Besides recommendable advice, we considered other advice to be debatable whether developers should follow it.This includes if it contributes to insecure or unusable authentication on the web, is inconsistent with literature references, is only applicable in certain use cases, or different opinions on the advice exist.
4.6.1 Identifying Debatable Advice.We identified debatable advice as we did for recommendable advice (cf.Sect.4.5.1).Using the same references, two researchers assessed the advice and discussed debatable advice in a consensus session.In cases without a supporting reference, we only tagged advice as debatable if a clear rationale existed.For traceability, we summarized the rationals of why advice is debatable in Table 11.All 49 debatable pieces of advice are available in the replication package (Appendix A).

Results
. We considered 49 of 272 pieces of advice debatable.Overall, we found debatable advice in 39 documents (31.5%).For a convenient overview, we grouped similar debatable advice in 14 groups (referred to as Dx) and reported it along with the rationale why it is debatable and examples in Table 11.
Most debatable advice covered password policies (D1) and was discussed in 22 documents.D1 is a good example of outdated advice.Outdated advice might have been considered correct in the past, but in the meantime, new insights emerged that render it incorrect.For example, we found advice to enforce complex composition rules, e.g., passwords should consist of lower-and uppercase letters, numbers, and symbols.Another example is to "Force users to change passwords regularly."However, research uncovered its negative effect on authentication usability and security [6 (1999), 42 (2018), 50 (2010), 55 (2011)].Likewise, official organizations such as NIST changed to less restrictive password policies in 2017 to improve usability and security compared to previous password policies [ 77 (2017)].However, we also found up-to-date advice that is less strict and only suggests enforcing a minimal password length, as also recommended by OWASP [81 (2021)].
We found more outdated advice recommends the use of security questions (D3), despite their usability [12 (2015)] and serious security shortcomings [13 (2012), 99 (2009)], and official guidelines advising against them [77 (2017)].We found other advice that harms (usable) security, such as not allowing to toggle masking in password fields and viewing the password in plain text (D11), automatically terminating all sessions on password change without asking the user (D6), or stating if a user entered an old but not their current password (D7).Some advice is more debatable, and its appropriateness depends on the use case.For example, consider the advice to use text messages (SMS) for authentication purposes (D5).On the one hand, text messages have drawbacks like the possibility of losing access to a phone number [59 (2021), 69 (2021)], SIM swapping attacks [58 (2020)], and being interceptable [78 (2016)].On the other hand, text messages are convenient as smartphones are ubiquitous.Similarly, PGP-encrypted emails for sending reset tokens (D12) provide good security, but PGP has only poor adoption [108 (2022)].Some advice is challenging to assess, like fingerprints always being preferable to facial recognition (D13), while vendors like Apple changed from fingerprint to Face ID [9 (2017)].
Summary: Debatable Advice.In 31.5% of the documents, we found debatable advice that conflicts with research findings and official guidelines.The debatable advice also contains instances of outdated advice, wrong advice, and advice that harms usable security.

Contradicting Advice
Besides recommendable (Sect.4.5) and debatable advice (Sect.4.6), i.e., advice in line or in conflict with literature references, we found several examples of contradicting advice within the document corpus itself.To analyze those, we identified sets of advice that contradict each other (e.g., "Do X" and "Don't do X").We refer to those as contradiction groups.Overall, we identified 15 contradiction groups with a total of 49 distinct pieces of advice (cf.Table 9).Most contradiction groups contained one pair of pieces of advice that directly contradict each other.We illustrate our key findings below.
Most contradiction groups addressed passwords (C2, C4, C5, C7, C11, C13).For example, C4 is about whether to enforce regular password changes for users or not.That contradiction group's advice states two opposite approaches: to "Force users to change passwords regularly" (6 documents) and "Don't enforce regular password updates/changes for users" (2 documents).Similar disagreements can be found in password length maximum (C2) and minimum (C5) requirements or password visibility (C7).But contradicting advice also exists for other authentication areas, like session timeouts (C3, C14), rate limiting (C8, C12), 2FA (C10), or general discussions on whether to use authentication at all (C6).
Another important aspect is how contradicting advice occurs within a single document and within the document corpus, i.e., across multiple documents.Regarding intra-document contradicting advice, we found this to be almost non-existent.Actually, only 2 documents contain contradicting pieces of advice.However, this low intra-document contradiction rate is to be expected because it is very unlikely that an author is inconsistent within a document.Analyzing the inter-document advice contradictions, we identified 60 documents (48.4%) that contained at least one piece of advice belonging to a contradiction group.That said, the document corpus contains many contradictions, indicating inconsistency and unreliability of online advice.
Contradiction groups contain opposite advice by definition.Especially for debatable topics, contrary recommendations and opinions exist that are sometimes mixed with recommendable advice (e.g., C1 or C4) and discussed in the document corpus.Therefore, the prevalence of debatable advice (Sect.4.6) among the contradiction groups is considerably higher at 30.6% compared to the overall 18.0%rate of debatable advice.
Summary: Contradicting Advice.We found 15 groups of contradicting advice.While it is almost non-existent within a single document, 48.4% of the documents contained some piece of contradicting advice.Contradiction groups tend to contain more debatable advice.

DISCUSSION
Below, we summarize our key insights and discuss recommendations for developers, advice writers, and academia.This section concludes with a discussion of future work.

Potential Impact of Advice on Authentication Deployments
Usable and secure web authentication is challenging, and previous work uncovered many issues, such as the widespread use of passwords in general [13,14], the deployment of complicated password • Use generic responses for authentication errors to not weaken security • Tell if user types in an old password C2 Whether to set a maximum password length or not.
• Set a maximum password length • Don't set a maximum password length C3 Keeping users logged in or automatically log out.
• Provide automatic logout/expire session • Keep users signed in 6 9 C4 Whether to enforce regular password changes or not.
• Force users to change passwords regularly • Don't force users to regularly change/update passwords C5 Different advice on minimum password length.
• Use a minimum of 8 characters • Use a minimum of 12-16 characters C6 Whether to use logins or not.
• Always require/enforce authentication • Avoid login completely if possible/not necessary 3 7 C7 Whether to allow toggling password field visibility or not.
• Don't display passwords on the screen • Allow users to toggle password visibility when typing it C8 Whether to use IP addresses or account for rate limits.
• Limit by account, not by IP • Limit by IP C9 Using OAuth or OpenID Connect for authentication.
• Implement 3rd-party authentication via OAuth • Use OpenID Connect for authentication C10 Which 2FA approach to choose.
• Provide OTPs as 2FA option • Use mobile device push notifications as second factor 3 5 C11 Different speed requirements for password hashing.
• Slow down hash functions • Choose work factors so that hashing time <1 sec C12 Different rate limits to prevent brute force attacks.
• Limit to 3 attempts per account in a set timespan • Limit to 20 guesses per minute and IP address 4 4 C13 Entering passwords twice during registration or not.
• Let users confirm passwords by writing twice • Don't ask for password confirmation C14 How long should tokens for authentication live.
Our findings illustrate how online advice for web authentication might contribute to unusable and insecure authentication deployments.When developers seek online advice to solve challenges around online authentication, they likely stumble upon advice that is of limited help to make web authentication usable and secure.Most advice covered technical security implementation details that are not relevant for end-users, while only 272 of 1,208 pieces of advice related to end-users' usable security (Sect.4.2).Contradicting, debatable, and outdated advice might even have a negative effect.
For example, passwords were-despite their known potentially negative impact on security and usability-the most prevalent advice topic and often the only discussed authentication approach (Table 7).The drawbacks and potential alternatives are rarely mentioned.We believe that such advice contributes to passwords' undiminished high popularity.Similarly, many websites do not follow best practices for password policies [60], and advice is also often not in line with best practices (Sect.4.6, Sect.4.7).Developers are not necessarily authentication or usable security experts.They might follow the advice unaware of the drawbacks and alternatives.
Considering that developers rely on online resources for solving security programming challenges [2,27], we think that online advice is a major factor that affects web authentication usable security negatively, e.g., outdated but still followed advice (e.g., changing passwords regularly) or following advice that was always known to counteract usability and security (e.g., using security questions).

Challenges for Developers
Based on our findings, we identified four main challenges for developers.Overall, the results indicate a mixture of advice that could and should be followed and advice that should be carefully considered or avoided at all.Advice Diversity & Priotrization.The advice was highly diverse (Sect.4.4), i.e., we found lots of advice on a single topic (e.g., password policies).This and the high advice volume indicate an overload for developers and a lack of advice prioritization among its authors, as similarly found in related work on end-user advice [91].Scattered Advice.The advice was also distributed over many documents (Sect.4.3).While some documents contained more advice, many contained only a single piece of advice.Moreover, most advice was only covered in a single document.This makes it harder for developers to find relevant advice; they might have to consider many documents to find helpful advice for their needs.Debatable Advice.Debatable advice (Sect.4.6) challenges developers, as they would have to recognize and not blindly follow detrimental advice.However, it cannot be assumed that developers (i) are usable security experts and, for example, know which advice is outdated and should not be followed anymore; or (ii) can recognize the consequences that poor usability of authentication features can have [41].Contradicting Advice.Finding contradicting advice (Sect.4.7) obliges the developer to decide which advice to follow and might be confusing.As with debatable advice, it is unclear whether developers have the skills to make an educated decision and assess related consequences.Even when developers encounter only one piece of contradicting advice, chances are to blindly follow debatable or otherwise disadvantageous advice instead of best practices.

Recommendations
Given the potential impact of advice on authentication deployments and the associated challenges for developers, we advocate to rethink how advice is given and consumed to achieve authentication for end-users that is both more usable and secure.Based on our findings, we make the following recommendations for improving security advice for developers in the future.Developers.We found that online security advice on web authentication is diverse and distributed over many resources on the web.Overall, developers should consider their context and not follow advice blindly, as we found debatable advice that could not be unconditionally recommended (Sect.4.6).Such assessments are mostly impractical for developers since they require lots of expertise, effort, and literature research.Hence, reasonable resources that can be assumed to provide high-quality advice are necessary.
We recommend developers look into the OWASP Cheat Sheets.According to our results, they contain the most advice and, therefore, are the most extensive resource, obviating the need for developers to look at many other resources that provide little or no advice.Another reason is that OWASP Cheat Sheets contain up-to-date advice specifically tailored to software practitioners.Giving Advice.We found several instances of outdated, incorrect, or otherwise debatable advice.Hence, authors should check their advice to avoid such problems before giving advice.This also implies regularly updating advice to prevent giving outdated advice.Unfortunately, checking advice is not straightforward, requiring expertise and considering resources such as the latest usable security research.Since advice is highly dependent on specific contexts, advice givers should describe in which situations advice should be followed or not to prevent developers from following advice that does not suit their project.
As already discussed, we found parallels between advice and the authentication ecosystem.We argue that advice is related to and could influence the authentication landscape.Therefore, advice givers should carefully choose what authentication topics they write about.That does not mean that no advice on passwords should be given.However, the authors should focus on approaches that prevent weaknesses in passwords.For example, when writing about passwords, their drawbacks (e.g., phishing, password reuse) should also be mentioned, and countermeasures (e.g., 2FA) or alternatives (e.g., FIDO2) should be outlined.Official Institutions.While we mentioned OWASP Cheat Sheets, the reader may wonder where official organizations stand as a source of advice.To our surprise, participants rarely reported official documents.For example, no NIST Special Publication was reported.This is especially remarkable as several documents referenced NIST guidelines and considered those authoritative.A closer look at OWASP Cheat Sheets [82] and the related NIST SP [77] quickly reveals major differences.For example, NIST's PDF is much longer at 79 pages than OWASP Cheat Sheets.The language in NIST's documents is also more abstract, e.g., passwords are referred to as "memorized secret authenticators" [77].We hypothesize that these factors lower the accessibility of NIST documents for developers and our participants did not report them for this reason.
Nonetheless, official institutions like NIST are reputable, authoritative organizations whose documents contain reasonable advice.We propose that official institutions provide their advice in a form similar to OWASP cheat sheets.This could be done by condensing the most important advice from their extensive guidelines, standards, and other documents and presenting it in a developerfriendly and actionable way.Authoritative Advice.We obtained 272 distinct pieces of advice that showed high diversity and were scattered all over the internet.This imposes a challenge, as no developer would search 406 documents for advice as we did.Hence, creating an authoritative source with helpful, actionable, carefully constructed web authentication advice for developers might improve accessibility, advice adoption, and end-user usable security.Moreover, this might prevent encountering debatable or contradicting advice.
It is yet unclear who should create and maintain authoritative advice.As already discussed, official institutions like NIST are considered authoritative but lack accessibility.Since this is mainly a presentation issue, we argue that official institutions could take this role.We found OWASP Cheat Sheets [82] to be the most comprehensive knowledge base for authentication advice.Hence, OWASP should continue the Cheat Sheet Series project, while official institutions should start creating similar resources.The advice extracted in this paper might be a starting point for creating (or extending) authoritative usable security authentication advice.
Knowledge Transfer: Academia to Practice.Unsurprisingly, developers do not consider scientific resources like papers helpful (Figure 1).Developers are unlikely to read dozens of papers that might require expert knowledge or are hidden behind paywalls.Instead, developers prefer online resources and heavily draw on contained advice [2,3,5].However, academic papers yield new usable security knowledge and are the main communication channel for scientific results.Their impact is limited if they do not foster any change and gain real-world traction.For developer authentication advice, we can confirm the knowledge gap between academia and practice that Gutfleisch et al. identified [41].To overcome this gap, researchers should aim to provide their knowledge in ways that developers can easily use and access.For example, new research results could be added to OWASP Cheat Sheets.While this can improve the adoption of advice that is grounded on research insights, the adoption is likely limited overall.Recent research indicates that Cheat Sheet advice is only partially followed [51].

Comparison with Related Work
While previous research investigated online security advice for both end-users [15,23,46,52,89,91,93] and software developers [2,3,5,10,11,16,27,28,29,31,39,76,92,120], we are the first to explore developer advice on authentication that affects usable security for end-users.As already discussed, we found similar themes compared to related work, like challenges around advice diversity and prioritization [91], and a usable security knowledge gap between academia and industry [41].Redmiles et al. investigated security advice regarding its quality (e.g., actionability) in a user study [91].As we performed a researcher evaluation, we refrained from evaluating aspects like developer actionability.Such an assessment with developers, similar to Redmiles et al. [91], is potential future work and possible with our dataset.This might be interesting, as our overall impression is that quality is mixed, e.g., some advice is highly concrete (e.g., containing source code) while other is vague.
Moreover, our work might provide some explanations for observations in related work.While not directly comparable, we see parallels to the findings of Lee et al. [60].Lee et al. found that only 13% of websites followed all password policy best practices.A contributing factor to this can be developer advice-due to the plethora of different advice.Some advice might be recommendable, and others debatable, while potentially contradicting each other at the same time.Best practices identified by Lee et al., e.g., block lists and strength meters, were also recommendations in our dataset but rarely occurred, reflecting the low adoption of best practices.This low advice adoption might be caused by the challenges (Sect.5.2) that can become a burden for developers, ultimately resulting in a poor cost-benefit trade-off similar to the one outlined by Herley in 2009 for end-user security advice [47].Overall, we note that not only developers impact usable security of authentication.Also, system administrators configuring authentication have an impact and draw on advice, e.g., consulting standards and guidelines, as recently found by Sahin et al. [98].The authors identified similar challenges compared to our study, like administrators facing and following outdated recommendations.
In our dataset, the fraction of documents from Q&A forums like SO (3.7%) is lower than in related work (e.g., 10-40% [2]).As participants were free to choose any resource, we think the lower prevalence is due to different study characteristics.SO posts are likely most useful for programming as they provide code snippets for copy and paste [27].However, our scenario is a more abstract, conceptual task that does not include programming, while Acar et al. [2] experimented with developers writing code.Other factors might be self-reporting and social desirability biases that prevent participants from reporting SO because the platform is also known to contain low-quality or insecure answers [16,27].While we can only speculate about the reasons, future research is needed to investigate which resources software professionals prefer for abstract, conceptual tasks like ours.

Outlook & Future Work
Changing advice and the authentication landscape can be considered long-term projects requiring suitable technical solutions.The endeavor to replace passwords has been an ongoing effort for decades [6] that turned out to be hard [13] and has not been successful yet.Regarding advice, passwords are the prevailing topic.Therefore, we believe passwords to stay for more years to come.Users also still tend to prefer passwords [122].However, the industry is trying to offer and shift to passwordless alternatives.In 2022, Apple, Microsoft, and Google announced plans to extend FIDO2 and introduce passkeys [25].Passkeys aim to replace passwords by syncing FIDO credentials across the users' devices.While passkeys seem promising, future research is needed to study their effect.The advent of passkeys also calls for a future replication of this study to check how online advice evolves and how passkeys are covered.
While we qualitatively analyzed developer-reported web authentication advice in this paper, our work does not investigate developers' interaction with advice and advice impact on authentication usable security.Conceptional decisions (e.g., whether 2FA is implemented or not) might not be decided by developers alone [41].Hence, further investigation of advice adoption and decision processes is needed.One aspect of this is the prioritization of advice.We refrained from indicating recommendation strength for the advice, as a valid assessment requires a broader expert evaluation beyond the involved researchers.Future work could investigate recommendation strength with experienced software professionals.We also propose to conduct a developer experiment focusing on the impact of advice on the final software to understand how advice should be given to developers.Similarly, future work could investigate the impact of code snippets, as we only focussed on textual (prose) advice.We hope that our dataset can lay the foundation for future work in these directions.

CONCLUSION
In this work, we qualitatively analyzed 272 pieces of advice containing developer advice on usable and secure authentication.We identify several major challenges for developers.For example, the advice is scattered over many resources and, therefore, hard to find.Despite their shortcoming, passwords are the most discussed authentication approach; the advice lacks the discussion of drawbacks and alternatives.Moreover, unconsciously following advice can induce usable security problems, as we found instances of debatable, outdated, and contradicting advice.Overall, our findings suggest that the advice potentially impacts authentication deployments.Our recommendations can inform better advice giving to improve usable and secure authentication on the web.

B SCENARIO & TASK
We gave software developers the following scenario and task to search for online resources on web authentication: For a research project on web development, we collect sources of information and advice that are used by software developers in software projects.As we focus on advice sources that could be relevant for development, this task contains no actual programming, but is about finding advice that might help you or others during development.

Project Description
Imagine you are working on a new software project called Keep-YourHealth.KeepYourHealth is a new web-based health platform that patients can use to store and keep track of their health-related information.This includes diagnoses, medical images (e.g., x-ray), and other data from doctors (similar to an electronic health record; but personal,

D2 Rate Limiting
Several pieces of advice are about limiting login attempts to prevent password guessing and credential stuffing [77 (2017)].However, its implementation can be debated and depends on the use case.We found advice to lock an account permanently or already after three attempts, which might lock legitimate users out of their accounts or could be leveraged for DoS attacks [63 (2018)].Instead, temporary blocks might be more appropriate.However, one needs to choose an appropriate approach, e.g., based on IP addresses, based on accounts, or regarding how many failed attempts are tolerated.
• Lock accounts after three or four failed login attempts D3 Security Questions Some advice (implicitly) suggests to use security questions.However, those pose a security threat as they are often easy to guess (e.g., name of pet) [13 (2012), 99 (2009)], have poor usability [12 (2015)] and official guidelines advise against their use [77 (2017)].
• Consider using a security question • Don't ask all security questions • Don't use security questions alone for password reset 3 5

D4 Authentication Enforcement
Some advice is about enforcement, e.g., always enforcing authentication or enforcing 2FA enrollment on registration.While this does not worsen security and might be an improvement, it is debatable if this is suitable and necessary in all situations.For example, many online shops work entirely with optional registration.Similarly, 2FA enforcement might only be reasonable for sensitive accounts but not important for mostly anonymous forum accounts, for example.
• Enforce 2FA on registration

D6 Terminating Sessions after Password Change
To terminate all running sessions after password change or reset is understandable from a security perspective.Some advice recommends forcing this automatically.However, this could sign out authorized devices without any reason and results in tedious re-logins for the user.Instead, it might be more helpful to give the user a choice whether to invalidate active sessions or not.
• Invalidate all sessions after password change • Force re-login after password reset 3 3

D7 (Not) Entering Passwords Twice
Whether passwords should be entered twice on registration and password changes or not is not well researched.Both variants have pros and cons, like a small probability of typos vs. tedious retyping of passwords.However, we believe that one-time-entry with an option to unmask the password field is more usable (if mistyped passwords can be reset/corrected).
• Let users confirm passwords by writing it twice.
• Don't ask for password confirmation by writing it twice 2 3

D8 Confusion: OAuth vs. OIDC
There is some confusion regarding when to use and what the OpenID Connect and OAuth protocols are.This is common on the internet, but OAuth refers to authorization, not authentication, as the official documentation states: "OAuth 2.0 is not an authentication protocol" [97 (2022)].Therefore, the advice is factually wrong.

D9 Automatic Logout Time Span
Some advice recommends short time spans to terminate a session automatically after inactivity (e.g., for 2-3 minutes).While automatic logouts can be beneficial for security, too short time spans are a usability burden and might only be necessary for special applications (e.g., banking).Official recommendations advise longer time spans, with a minimum of 15 minutes [77 (2017)].

D10 Error Reporting for Login Failures
Another recommendation is to display when someone entered an old password, when it was changed, or generally report that the entered password, username, or email is wrong.This is highly problematic, as it leaks information on account existence or credentials a user used in the past.As passwords are reused, this could weaken security of the user's other accounts [86 (2017)].
• Notify users if they entered an old password • If user typed an old password, output the date it was changed • Output exactly which is wrong (username, email, or password) 3 1

D11 (Not) Unmasking Passwords
There is still advice that indicates to always hide passwords in a password field.However, especially when users have long and unique passwords, the entry is error-prone [77 (2017)].Therefore, offering users an option to unmask the password improves usability.In some cases, displaying an unmasked password has no security implications (e.g., private environment).To account for other situations, however, masking passwords should be the secure default.
• Don't display passwords on the screen 1 1

D12 Using GPG Encrypted Mail
Using GPG encrypted mails, e.g., to send users their reset tokens, seems to be a good idea in terms of security.However, the adoption of email encryption is poor [108 (2022)].Therefore, this should rather be handled as an optional feature.
• Send reset tokens in GPG encrypted mail 1 1

D13 Fingerprint vs. Facial Recognition
One advice is to prefer fingerprints to facial recognition.This advice is highly general, might depend on the technology in use, and could be wrong nowadays.For example, Apple changed from fingerprint Touch ID to facial recognition Face ID and evaluated the latter to be more secure [9 (2017)].
• Prefer fingerprint scanning to facial recognition 1 1
• Use CAPTCHAs voluntary, and under the user's control) and can be supplemented with biometric and other data provided by the users.Patients can gather and upload biometric data such as heartbeat (data recorded by wearables, smartwatches, etc.) for early detection of cardiovascular diseases.KeepYourHealth should be compatible with both desktop and mobile browsers.KeepYourHealth serves as a comprehensive and central instance to keep track of health, share selected data with doctors when needed, and help improve the health of its users.It will be used by people of any age, with different skills and should therefore be very easy-to-use.
At the same time, the platform needs to be secure, as a recent survey found that potential users see the data (e.g., medical data, biometric data, personal identifiable information) that is stored and processed with KeepYourHealth as highly sensitive.
[Page break.]Scenario You are responsible for the architecture and design of an easy-to-use and secure authentication workflow for the KeepYourHealth platform.For a kick-off meeting with other members of the development team, imagine that your job is to prepare a presentation with a concept for the authentication system including user registration and login.Therefore you have to do an online search.Requirements: • The authentication process (and all other mechanisms) of Keep-YourHealth has to be secure.• The authentication process and its security features have to be user-friendly (provide a high level of usability for all kinds of end users like patients, doctors, other health care staff).• You are free to make all necessary decisions including the choice of authentication methods, user interface, registration process, etc.As this project is in an early stage, you can choose any technology, programming languages, frameworks, and approaches that seem reasonable to you.Your Task: • Prepare the kick-off meeting by searching and collecting resources from the web for 30 minutes that you find helpful and you might use for KeepYourHealth's authentication process as well as the requirements from above.• Resources and advice can be all kinds of online documents (e.g., blog posts, official documentation, guidelines, forum discussions, other websites).• Report all resources and advice in the following table we provide, so your team can use these resources and its advice later in the project (like a "knowledge base").• Please include for each resource you found its URL and how you found it (e.g., Google search query, a site you had bookmarked).Please keep in mind that usability and security is in the focus of this project.

Abstract
Our paper comprises human subjects research and a qualitative analysis of online advice for developers on usable and secure authentication.Therefore, we provide study materials and result data in our artifact (but no scripts with experiments, etc.).Hence, we apply only for the Artifact Available badge.Evaluation should be rather quick for the evaluators.We provide the following artifacts: (1) The scenario and task we used, (2) the materials used for recruiting, (3) the consent form, (4) the questionnaire from the developer survey, (5) the list of advice that we extracted from the documents, (6) the detailed advice analysis results, (7) the set of documents, queries, and search results, and (8) the extended version of the paper.To ensure participant privacy, we do not provide any raw questionnaire responses.
Please see the contained README file for a detailed overview of the artifact's content.

Description & Requirements
This artifact has no specific requirements, as it only consists of some study materials and data, including all advice extracted in the paper.

Security, privacy, and ethical concerns
None.There are no scripts to be run that.Hence, there is no risk for evaluators.

How to access
The artifact is hosted at the Open Science Foundation (OSF) and is permanently available via the following DOI that is also linked in the paper: https://doi.org/10.17605/OSF.IO/PFB5H.(Or alternatively via direct OSF link: https:// osf.io/pfb5h/.)

Hardware dependencies
None.

Software dependencies
None.

Set-up
This artifact does not contain any scripts or experiments to be run.It just consists of the data and materials we provide.Therefore, the only set-up step is to download the artifact, so one could access and inspect it.
We note that one might want to use some program to conveniently view the CSV or Markdown files, although it is not necessary.

Installation
The artifact can be accessed and inspected via the abovementioned links.To download the artifact as a compressed ZIP archive from OSF, one can visit https://osf.io/pfb5h/files/osfstorage, click "Download this folder," and extract the archive after downloading.

Basic Test
None.

Evaluation workflow
As we only apply for the Artifact Available badge, the evaluation should be straight-forward by just checking that the materials are available.This can be either done directly on the online platform or more convenient by downloading the materials locally.Overall, the evaluation should be quick, and we expect it to take 15-30 minutes at most.

Notes on Reusability
As the goal of this artifact is to provide the advice we extracted and analyzed in the paper, other researchers might want to inspect the advice or reuse some advice for their own research.

Version
Based on the LaTeX template for Artifact Evaluation V20231005.Submission, reviewing and badging methodology followed for the evaluation of this artifact can be found at https://secartifacts.github.io/acmccs2023/.

Figure 1 :
Figure1: Participants' ranking of information sources for the question: "Which advice or information sources do you find most helpful in general?".From most helpful (1) to least helpful(10), excluding "Other" answers.
Text in Document & Assigned Advice Code "As a developer yourself, it's your responsibility to provide users with the best and most secure experience possible."→ Ensure authentication is secure and has good UX "Make sure to explain each [2FA] step and keep in mind, that not everybody has the same technical knowlegde.For example, HubSpot added a description of Google Authenticator when the user would choose that option."→ Explain each 2FA step "Consider Password-less Authentication [. ..]Some applications are good candidates for a new approach: eliminating the password altogether.[. ..] users [. ..] forget their credentials, password-less authentication is an compelling option."→ Use password alternatives "If your organisation wants to take its login game to the next level, you should definitely consider password alternatives."→ Use password alternatives "Send the user an email informing them that their password has been reset."→ Notify users via email when password was reset Occurrence frequency of advice (in number of documents).

Figure 2 :
Figure 2: Boxplots that describe the advice distribution.

Figure 3 :
Figure 3: Co-occurrence of the topics from Table7within the same document.

18 9 10 †
Container for sub-topics; containing no advice.1Advice occurrences in a topic.2Distinct pieces of advice in a topic.3Documents containing advice from a topic.

4 2 C15
Whether to use CAPTCHAs or not.

1 1 1
Number of distinct pieces of advice. 2 Number of distinct documents.

Table 1 :
Demographics of the 18 web developers.

Table 2 :
Most frequent domains and search query terms and bigrams that participants reported.(a)Top-10 documents domains.
(b) Top-10 terms from search queries.

Table 3 :
Types of the reported documents.

Table 4 :
Examples of document excerpts with concrete advice and assigned advice codes.

Table 6 :
Top-10 most frequent pieces of advice measured by occurrence in documents.

Table 7 :
Categorization of advice in different topics.

Table 8 :
Top-10 most frequent recommendable advice measured by occurrence in documents.

Table 9 :
Overview of the 15 contradiction groups, each with a short description and advice examples (in italics).

Table 11 :
Overview of debatable advice, grouped in categories, each with representative advice examples.
Some advice is about using SMS for authentication, e.g., sending 2FA codes.In general, SMS can be convenient for users, and they are therefore widely used[18 (2018)].However, there are some drawbacks like losing access to a phone number[59 (2021),69 (2021)] or SIM swapping attacks[58 (2020)].NIST also raises concerns[78 (2016)].Overall, usage of SMS should be carefully thought about and alternatives considered.
table looks like and how it could be filled: [Example screenshot of the table.][Page break.]• The authentication process for the KeepYourHealth platform should be user-friendly and secure.• Search 30 minutes for helpful resources on web authentication you might use for the KeepYourHealth platform.• It is sufficient if you skim the resources roughly.You don't have to read those resources in detail.• Report your results by filling the following table.You can also report resources where you found only a small part to be helpful for your task.• You don't have to prepare a presentation for the kick-off meeting.Currently, we are only interested in the resources you find online.Figure 4: A screenshot of the search task's table as used in the questionnaire.ACM CCS'23 Artifact Appendix: "Make Them Change it Every Week!":A Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication