Revocation Speedrun: How the WebPKI Copes with Fraudulent Certificates

The TLS ecosystem depends on certificates to bootstrap secure connections. Certificate Authorities (CAs) are trusted to issue these correctly. However, as a result of security breaches or attacks, certificates may be issued fraudulently and need to be revoked prematurely. Revocation, as a reactive measure, is fundamentally damage control and, as such, time is critical. Therefore, measuring reaction delay is the first step to identifying how well the revocation system functions. In this paper we attempt to characterize the current performance of the WebPKI in dealing with fraudulent certificates. We present measurements of each step in the revocation process: the detection of certificate issuance through Certificate Transparency (CT) monitoring, the administrative revocation process at popular CAs, and the revocation checking behavior of end-user clients, both in a controlled virtualized environment and in the wild. We perform two live measurements, in 2022 and 2023, respectively, to provide a longitudinal comparison. We find that detection and revocation of fraudulent certificates is quick and efficient when leveraging CT and can be completed within 6.5 hours on average. Furthermore, CT is being increasingly enforced by some browsers. However, ∼83% of the clients we observed, across popular browsers, brands and OSes, completely disregard a certificate's status, whileall of the studied browsers still display soft-fail behavior, making them vulnerable to attackers capable of interfering with the network. Of the clients that do check revocation, we find that 35% can be made to accept a revoked certificate through the use of OCSP Stapling. We expect this number to grow with client-side adoption of OCSP Stapling [RFC6961]. Current OCSP expiration times allow a revoked certificate to remain fully valid for up to 7 days for the majority of CAs, exposing clients to attacks.


INTRODUCTION
The SSL/TLS ecosystem relies on public-key cryptography to bootstrap secure connections between clients and servers and prevent Man-in-the-Middle attacks.The link between public-key and server identity is in turn validated and certified by Certificate Authorities (CAs).The security of the Public-Key Infrastructure (PKI) thus builds on trust in the proper functioning of CAs.However, when CAs are rogue or become compromised by adversaries, fraudulent certificates can be issued by adversaries for domains that they do not legitimately own [8,18,19,36].There have also been repeated demonstrations of off-path attacks on Domain Validation (DV), the most common method CAs use to validate control over a domain during certificate issuance [5,6,10].Lastly, malicious actors may also obtain certificates for domains they have hijacked, even for a brief period of time [26,27].
Policing such fraudulent certificates depends on the respective domain owner, as they essentially serve as the ground-truth for asserting whether a certificate issuance is authorized or not.If a certificate is unauthorized, the domain owner can contact the issuing CA and request to have the certificate revoked.Naturally, this process requires the domain owner to be aware of the existence of the fraudulent certificate.
In the past, detecting fraudulent certificates this could have taken months.However, in 2012 Google introduced Certificate Transparency (CT) to accelerate detection of fraudulent certificates and since 2017 all major CAs comply with its regulations.CT requires all issued certificates to be logged in public, auditable log servers.By monitoring these logs, domain owners can be quickly alerted about the issuance of a certificate for their domain and take the necessary actions.
CT is thus designed to allow monitoring and fast detection of certificate misissuance.It does not however prevent misissuance.The security benefit of CT therefore also depends on the performance of the current revocation system.In this work we do the first end-to-end performance analysis of the revocation system, including the efficiency and effectiveness of CT.
Contributions.We characterize the efficacy of CT and the current revocation system in protecting clients by performing extensive measurements covering the entire life cycle of a fraudulent certificate.
• We perform the first measurement of the speed with which current CT monitors detect certificates.We test 15 different monitoring solutions, covering notification services, search services and local tools.We find that detection can occur in a matter of minutes.
• We perform the first measurement of the speed with which CAs respond to administrative revocation requests by domain owners.We test 9 of the top CAs and find an average of 6.5 hours of processing time on the CAs' part.
• We confirm that the findings from [28], concerning poor revocation checking behavior of clients, still hold 8 years later.We expand on the lab measurement by testing for caching effects.We examine 19 current browser implementations across 5 platforms.We confirm that all clients accept a revoked certificate when the CA is unavailable, but also find that most clients will continue accepting the certificate even when the CA is reachable again.
• We additionally perform the first measurements of revocation checking behavior in the wild; an initial measurement in January 2022 and a followup in June 2023 for insight into changes in browsers and OSes in that time.Through the use of an advertisement network over a period of 13 (3) days, respectively, we collect HTTPS requests from nearly 530k (260k) unique devices across 226 (194) countries, in a representative sampling across all continents.We find that at least 87% (82%) of clients accept a revoked certificate.We make this dataset available at https://github.com/jenfrie/certm-data.
• We demonstrate the use of OCSP Stapling by an attacker to extend the validity of a revoked certificate by caching a valid OCSP response.Of the clients that do check revocation, we find that 38% (35%), respectively, can be made to accept a revoked certificate by providing a valid, stapled OCSP response.
• CT rests on the premise that clients reject certificates that were not logged (i.e., do not have SCTs).We therefore perform the first study of clients' verification of SCTs.We find that in 2022 only 30% of clients followed this behavior.This means a rogue CA could have issued an unlogged certificate that would be accepted by 70% of clients.In 2023 only 34% of clients accepted a certificate without SCTs, showing a significant improvement in client-side adoption of CT.
Ethical considerations.We run our evaluations on local installations that we set up in a controlled environment.Our evaluations are carried out in a black box model; we do not attempt to reverse-engineer the systems and implementations that we test.
To evaluate revocation behavior of clients on the Internet we use an advertisement network (adnet).Previous work on system and protocol security has used adnets to attract client queries to the measurement infrastructure [25].Studies involving adnets may create ethical concerns: they may introduce load, expose sensitive user information, or load resources from legally questionable domains.Our study follows the ethical guidelines for network measurements defined in [31], which were additionally vetted and approved by the ethical committee of our scientific organization.We address specific concerns below: • Minimal load: in our evaluation we serve 4 <img> tags, which initiate connections to our server.Each connection attempts a TLS handshake while serving a specific certificate.We check whether the handshake succeeds or fails; if it succeeds, a 1x1 pixel image is returned, keeping transferred resources as small as possible.The volume of the packets we exchange do not exceed those in other such studies [25] [24].
• User participation: from the users' perspective, our measurements are no different from other third-party content loaded when visiting a website.We do not experiment on the users themselves, since our experiment merely verifies if the clients successfully complete the TLS handshake or not, which is independent of the users and their behavior.We provide the option to opt-out of the experiment entirely with a clearly described button.This terminates the measurement (connections that have not been initiated yet are prevented) and signals to the server to discard any measurements taken during this encounter.
• Privacy: the data we collect is processed and analyzed during the communication with the clients.Furthermore, the data we collect is not specific to any individual, but common to many users.We use the client IP address only temporarily to derive the country of origin for aggregate statistics of client distribution, as well as determine the total number of unique IPs and clients.After the measurement completes, all client IPs are discarded and not stored in the resulting dataset.Thus, we do not store any sensitive or personal data.We do not study user behavior and do not collect any data related to user behavior.Our measurement page provides a privacy policy and instructions on how to opt-out of the experiment.
• User security: our measurement infrastructure triggers requests only to the server that we set up and control, with one exception, badssl.com,maintained by the Chromium Project.Hence, no client was redirected by our measurements to any irreputable domain or dangerous content.
Organization.Our work is organized as follows.In Section 2 we review related work and in Section 3 we provide an overview of Certificate Transparency and the revocation mechanism.In Section 4 we evaluate detection of fraudulent certificates using CT monitors.Once a fraudulent certificate is detected it has to be revoked; we study the speed of revocation mechanisms in Section 5.In Section 6 we evaluate in the lab which browsers and operating systems check and respect information on revoked certificates and in Section 7 we measure which clients on the Internet check for revoked certificates.In Section 8 we discuss the implications of fast revocations on attacks and scalability of revocation mechanisms.We conclude this work in Section 9.

RELATED WORK
Previous work on CT has primarily focused on its proper functioning and the security of third-party monitors as an integral part of the CT system [9,22,23].However, we have found no work that has measured the practical effectiveness of CT, specifically, in finding and revoking fraudulent certificates and thus contributing to the security of the WebPKI.
Our work is particularly motivated by attacks on the domain validation methods for issuing fraudulent certificates, as demonstrated by [5,6,10].Since CT is motivated by repeated cases of misissuance, mainly due to the compromise of CAs [20], our goal is to explore and describe how effective CT is in combating these attacks.
Recently, [17] analyzed CAs' responses to reports of certificates used for phishing, including response time.However, with phishing, attackers legitimately own the domain; for the CA there is no conflict of ownership.The previous study therefore does not consider scenarios in which domain owners report unauthorized certificates issued for their own domains.We perform an experiment focusing on this scenario.
On certificate revocation, an extensive study of OS and browser verification behavior was performed in [28] using a virtualized setup.The results showed that no browser-OS combination employed strict, hard-failing revocation checking.Particularly DV certificates could be forced to pass validation by blocking OCSP.We replicate these findings with current browser and OS versions and extend the experiment to test for effects of browser caching.

OVERVIEW: CERTIFICATE DETECTION AND REVOCATION
In this section we provide an overview of Certificate Transparency (CT) and the mechanisms for revocation of certificates.

Certificate Transparency
CT is a Google-initiated effort to facilitate faster detection of fraudulent certificates in the wake of major security incidents involving compromised CAs [20].CT requires CAs to publish all issued certificates in publicly accessible, append-only logs.This makes certificate issuance in the WebPKI open and auditable.Figure 1 provides an overview of the relevant components of the CT and revocation ecosystems.
To enforce logging, log servers provide a signed receipt that a given certificate was logged: a Signed Certificate Timestamp (SCT).SCTs are verified by clients and must be delivered along with the certificate during the TLS handshake for it to be considered valid.Since 2018, prominent browsers like Chrome, Firefox and Safari support this validation.
Any interested party, including domain owners, can monitor CT logs through public APIs to follow all certificate issuances more or less in real time1 [21].
As certificates are stored sequentially in a Merkle tree, certificates are not indexed by domain name on the log servers.Monitoring one's own domain requires looking through all certificates, which requires considerable amounts of storage and bandwidth [23].This has caused an array of third-party monitors to offer search and notification services.As the more attractive monitoring option, third-party monitors are also a de facto cornerstone of CT.

Revocation Checking
Digital certificates are valid for a limited time, as determined by their expiration date.This time frame is generally between a few months to a few years.However, in some cases, e.g., due to the compromise of a private key, certificates need to be invalidated prematurely to avoid misuse.This is referred to as revocation (see Figure 1 for an overview of the relevant components).
In order for clients to know a certificate's revocation status, CAs provide at least one of two mechanisms: a Certificates Revocation List (CRL), which simply lists non-expired, revoked certificates, or an Online Certificate Status Protocol (OCSP) responder, which responds to individual certificate status queries with "good", "revoked" or "unauthorized".However, both methods suffer drawbacks [14,33].
3.2.1 CRL Drawbacks.CRLs are often large, impacting local storage and increasing latency due to processing time and potential in-line downloading.To curb bandwidth usage, CRLs use rather long validity periods.In our study we find that all of the CAs that issue CRLs use weekly expiration dates; see Section 7. Thus, we find that in the worst case, clients that rely on CRLs and abide by the expiration dates would reject a revoked certificate only a week after the actual revocation [7,14,30,33].

OCSP Drawbacks.
With OCSP, clients look up the certificate status on-demand for the domains they are actually visiting.This also adds latency to each connection, as the client establishes an additional out-of-band connection and waits for the status reply from the OCSP responder.When originating from the client, OCSP queries also compromise privacy, as the CA will learn, which domain the client is attempting to connect to.More importantly, however, OCSP responders become a central point of failure for all TLS connections and a prime target for Denial-of-Service (DoS) attacks.Since it is difficult and expensive for CAs to maintain such a high level of availability, browsers typically soft-fail OCSP checks, i.e. certificates are presumed valid when CA responders are unavailable (we reaffirm this experimentally in Section 6).This in turn allows attackers to force clients to accept revoked certificates by simply blocking OCSP connections [7,13,14,30,33].As a result, some browsers, most notably Chrome, have defaulted to forgoing revocation checks altogether.

OCSP Stapling.
OCSP Stapling addresses some of the downsides of OCSP by offloading the status requests to the web server.The server regularly polls its CA's OCSP responder to obtain the most up-to-date status.This status is then forwarded to the client for verification during the TLS handshake.The client can determine whether the status is still valid based on the nextUpdate field.As the status is retrieved asynchronously by the server and cached, the burden on the OCSP responder and the delay during the TLS handshake are reduced, while also protecting clients' privacy.
Despite its advantages, supporting OCSP Stapling is currently not mandatory and thus up to the domain owner.On a malicious server the attacker can decide when and how to use OCSP Stapling.When Stapling is not used, clients fall back on other methods [7,28,30,32,33].

Proprietary CRLs.
Nowadays popular browsers often implement some form of central, proprietary CRL (Chrome's CRLsets, Firefox's OneCRL, Microsoft's disallowedcert.stland Apple's aggregated CRLs), which are assembled from CAs' CRLs and periodically pushed to browsers as updates.However, to combat the size issue, these aggregated CRLs are curated to cover only the most important revocations, i.e. primarily intermediate CA certificates, essentially trading off completeness for efficiency [1,2,14,28,29,32].As demonstrated by [28], they are ineffective for protecting even highly-ranked, popular domains: only 3.9% (10.4%) of revocations recorded by CRLs for certificates across the Alexa Top-1M (Top-1K) domains were included in Google's proprietary CRLSets.To confirm this for the other proprietary CRLs (Microsoft disallowedcert.stland Firefox OneCRL), we retrieved certificates from the Alexa Top-10K domains, as well as 100K domains randomly sampled from the remaining Alexa Top-1M, and found a total of 46K certificates in their CT history for which OCSP responded with "revoked", only 2 of which were included in any of the proprietary CRLs.

How Revocation Works
CAs and certificate resellers typically provide functionality in their web UIs to revoke certificates using the account they were purchased with.However, this is not an option in situations in which a malicious actor issues a certificate using another account.Without the account or private key, the domain owner needs to request administrative revocation.Most CAs therefore provide email addresses or "report abuse" forms on their web pages for reporting fraudulent certificates (see Appendix).The revocations in this study were completed through these methods.
In ACME implementations, revocation can be performed without administrative involvement, as long as all domains listed in the certificate can be validated by the ACME client.However, this automated approach can be easily countered by including a "guard" domain in the certificate that is under the attacker's control.This prevents the ACME client from validating control over all domains listed in the certificate (as required by RFC8555 [4]).Such cases require administrative revocation, as with other CAs.

DETECTION THROUGH CT MONITORS
The first step to policing fraudulent certificates is detecting them.Since CT's widespread adoption by CAs, certificates are publicly visible shortly after issuance.We therefore measure how fast fraudulent certificates can be detected in practice through CT.This is the first such side-by-side comparison of available monitoring solutions and real-world performance measurement of CT, providing us with a lower bound on how fast CT monitoring-based responses can be.

Monitors & CAs
For our study we implement all prominent CT monitoring solutions as of June 2022 (see Table 1), compiled from the official list [11] and top search results.We establish the following taxonomy for these monitoring options: a) Third-Party Monitors continuously monitor CT logs and index discovered certificates by domain name.These services can further be subcategorized into: a1) Notification Services, which allow registered users to specify domain names they wish to be notified about via email, whenever a matching certificate is discovered.a2) Search Services, which provide web pages and/or APIs for retrieving logged certificates by domain name.We query these services in 5 minute intervals to avoid undue load on the servers.
a3) Open-Source Tools, which are run locally and query either third-party databases or CT logs directly and output certificates that match a configured list of domains.
b) Direct Monitoring requires maintaining a list of CT logs to query (we use the union of Apple [3] and Google [12] log lists), implementing requests to the CT endpoints and parsing the responses.We implement a very light-weight option that continuously retrieves all new entries since the last query.We match a certificate by performing a sub-string search for our domain names on the ASCII dump of each response.
We then issue a range of certificates from different CAs for domains under our control and observe how quickly we are notified by each monitor.CAs were selected based on prominence in search results and frequency of appearance in CT logs.Since this work is motivated by hijacks of DV methods, our selection is furthermore restricted to CAs that offer DV certificates (disqualifying some prominent CAs, such as IdenTrust and Entrust).This resulted in the following list: GeoTrust, AlphaSSL and Thawte (all under the umbrella of DigiCert), Sectigo/Comodo, GoDaddy, Let's Encrypt, Network Solutions, SSL.com and RapidSSL.
We reevaluated the representation of these CAs in CT logs in October 2023 by observing 29.5M certificate issuances over 88 hours through Certstream: Let's Encrypt alone accounted for 68.7% of issued certificates, with the remaining studied CAs covering an additional 7.3%, representing a total of 76% of issued certificates.

Measurement
For each certificate, we measure the time between the completion of DV and the moment we are notified by each monitor.For notification services this is when the email arrives in our inbox, for all others it is when an alert is output to the console.
The measurements were conducted in three rounds.During the initial testing round we issued and immediately revoked one certificate per CA.In the second round, we again set up new domains (one for each CA) and issued new certificates, which would serve as "legitimate" certificates.
Then, to simulate an attack, we issued another, "rogue" certificate for each domain from the same respective CA.We used separate accounts to buy these certificates and completed the domain validation from a different IP address.The three rounds were conducted across the span of a week to account for differences in update frequency and timing of the different third-party services.

Results
As shown in Figure 2, the majority of monitoring solutions generally detect new certificates within half an hour of their issuance.Specifically, the median (average) detection speeds across all monitors are 8 (18) minutes, while the fastest is reliably around 1-2 minutes (Certstream).Among the email notification services, CT Advisor and Remme Keyhub failed to notify at all, hence the lack of data.
We find that "rogue" and "legitimate" certificates are not fundamentally differentiable based on detection speed.We also found no difference in issuance or detection speed between CAs.
For domain owners it is worth noting that direct monitoring implementations are not trivial and should parallelize requests to different CT logs to achieve adequate performance.Monitors should also closely follow rate limiting responses for optimal performance; our implementation used a naive timeout for comparability with the "Certificate Search" scraping implementations.As a result, we found that some third-party solutions (most notably Certstream) provided better performance and faster detection than our direct monitoring.
Generally, delays in detection depend on a number of factors, e.g., the implementation of the monitor.Scraping certificate search providers (e.g., Censys, crt.sh) relies on the update frequency of the provider, while using email notification services additionally incurs delays through email transmission and the update frequency of the email client.
CT logs also introduce variation in detection speed.Although all logs currently accepted by Google and Apple are subject to a Maximum Merge Delay (MMD) of 24h, within which each received certificate must be publicly visible, in practice certificates are included in the log much faster.However, how quickly this happens depends on the implementation of the log and may vary depending on load.
Finally, we also verified that all monitors also look for Subject Alternative Names (SANs) in addition to the Common Name (CN) of a certificate.An attacker could otherwise set the CN to a different domain she controls and add a targeted domain as a SAN to avoid detection.

REVOCATION PROCESS
Once a fraudulent certificate is detected, it is up to the domain owner to contact the CA and have it revoked.Delays in this process contribute to the window in which the attacker has a fully valid certificate.As such fraudulent issuances always require administrative verification of the revocation claim, this delay is non-negligible.We therefore measure the duration of this process.

Methodology
After issuing the certificates in the previous measurement, we adopt the role of domain owner and, for each certificate, contact the respective CA to request its revocation.We then measure how long each process takes; i.e. we measure the time between sending the first email and the OCSP revocationTime.
To remove delays on our part from the measurements, we note the time of each email sent and received and deduct the time between receiving a response from the CA and sending our own reply.Thus, each measurement covers only the time taken by the CA to process the revocation and is therefore a lower bound on the duration of the full procedure in practice.
Since many CAs offer DV through one of the domain's standard administrative email addresses (e.g.admin, postmaster), we test whether the revocation process could be accelerated by contacting the CA directly from one of these addresses with the revocation request.We therefore conducted one round of revocations through the respective admin address, while the other two were conducted through non-admin addresses.
The revocation procedure begins with contacting the CA's support or abuse-reporting email address.These addresses are found on the CAs' homepages.For a DV certificate, the CA will reply with a DNS TXT-based domain control challenge.After verifying the challenge, the certificate is revoked.

Results
Whether the CA was contacted from one of the default administrative addresses (in our tests admin@<domain>.xyz)did not affect the procedure for all CAs except GoDaddy.The latter revoked one of our certificates without any further confirmation.While this speeds up the revocation process, it may also open the door for spoofed revocation requests.All other CAs first asked to reconfirm the revocation, as a simple challenge-response step.
Figure 3 summarizes the time in hours taken for each revocation procedure.These measurements discount the time taken for us to respond.We observe a median (average) processing time of 3.18 (6.5) hours, while 90% of revocations were completed within 14.2 hours2 .RapidSSL, SSL.com and Thawte show a noticeably larger variance than the remaining studied CAs, however, it is difficult to say without additional data points which measurements are outliers and what end of this range is more common.Due to budgetary constraints, we can only provide a small number of data points as a rough indication of typical revocation time.
One factor that likely contributes to the observed delays (and the variability of these delays) is the propagation of the DNS TXT records created to complete the domain control challenges.Other factors include the workload per CA support employee at the time of each measurement and the CA's prioritization of incoming revocation requests, which are extremely difficult to measure.
Finally, it is worth noting that these procedures need to balance revocation speed with thorough review of the circumstances to prevent unauthorized revocations.Despite the importance of timely revocation, optimizing purely for speed introduces other vulnerabilities and thus may not be desirable.

REVOCATION CHECKING IN THE LAB
The last step necessary for revocation to be successful is verification by clients.If clients do not check a certificate's status, the revocation process provides no benefit.We therefore attempt to determine the behavior of popular browsers in this regard.Previous work investigated this in 2015 [28] and to a lesser degree in 2021 [32], by setting up combinations of popular OSes and browsers in virtualized environments and determining their response to revoked certificates.The first study also tested how browsers behaved when revocation information was unavailable.We repeat this study to determine if the results still hold in 2023.Additionally, we test for caching effects, which were not considered in either study.

Methodology
We test a variety of browser-OS combinations in virtual environments using the most recent stable versions of both OSes and browsers available at the time of the study.In each configuration, a test domain with a revoked certificate is visited via HTTPS.For each visit we note whether the browser loads the webpage (i.e.accepts the certificate) or displays a warning.
We first set up a fresh installation of the OS and install all browsers we aim to test.We revert to this state between testing different browsers and between each of the following scenarios: Non-blocking (NB).Browsers have access to OCSP/CRL endpoints.Blocking (B).All OCSP/CRL endpoints listed in our certificates are redirected to localhost on the VM-host.
Blocking, then non-blocking (B-NB).The domains were first visited without OCSP/CRL access, the browser was closed, OCSP/CRL endpoints were made available and the domains were visited again.
Blocking, clear cache, then non-blocking (B-CC-NB).The domains were first visited without OCSP/CRL access, the browser's cache was fully cleared, OCSP/CRL endpoints were made available and the domains were visited again.
We reset the VM state to avoid OS-level interaction between browsers that might change their behavior.We implemented this after observing that visiting a site on Windows with Edge or Internet Explorer (IE) while OCSP/CRL was available led Firefox and Chrome to show a warning when subsequently visiting the same site for the first time without access to OCSP/CRL.This provides some evidence that there is some OS-level caching of certificate revocation information.

Results
Our results are summarized in Table 2.In the non-blocking scenario (NB), where OCSP/CRL endpoints are available, we find mostly the same behavior as in [28] and [32].iOS has since introduced revocation checks.However, Chrome, Brave and Opera on Linux and Windows still forgo revocation checking and simply accept revoked certificates.
As in [28], we find that browsers uniformly soft-fail when OCSP and CRLs are unavailable (B).Furthermore, this soft-fail appears to be cached in all cases except Edge and IE on Windows, as certificates continued to be accepted even though revocation endpoints were available again (B-NB).This means that, for most platforms, if an attacker blocks the first revocation check, the certificate will be accepted on subsequent visits without the need to keep revocation endpoints unavailable.This may significantly reduce the effort required, particularly for off-path attackers.
Clearing the browser cache, or invalidating the cached version by altering the web page, caused most browsers that check revocation (with the exception of Brave, Opera and Firefox on MacOSX) to reattempt a revocation check and reject the certificate (B-CC-NB).These behaviors confirm the involvement of the browser cache.
In conclusion, all of the browsers and platforms we tested still soft-fail revocation checks; in addition, some platforms cache this soft-failed result.

REVOCATION CHECKING IN THE WILD
Virtualized tests show that revocation checking behavior varies between OSes and browsers and that many forgo checks by default.Market share data can help estimate the extent to which endusers are vulnerable, but no previous work has attempted to directly measure the behavior of real Internet clients.We therefore perform additional, live measurements using an advertising network (adnet), to determine the extent to which actual end-users are vulnerable to revoked certificates.We perform an initial measurement in January 2022 and follow up with a second measurement in June 2023 to observe changes in client behavior.

Server Setup.
To perform the measurement, we set up a server with several subdomains, each with a unique certificate (issued through Let's Encrypt) and configuration: rev uses a revoked certificate.sta uses a revoked certificate, but includes a valid stapled OCSP response.The OCSP response is cached immediately after certificate issuance and has a validity of 7 days.mus uses a valid certificate with the Must-Staple extension, but does not deliver a stapled OCSP response.sct uses a certificate that does not contain embedded SCTs from a CT log3 , but is otherwise valid.
A client that properly validates certificates should therefore never connect to the rev, mus and sct subdomains and stop connecting to the sta subdomain after the OCSP response expires.
To test the different scenarios for each client, the initial landing page of the server triggers a request (via <img> tags) to each subdomain, which in turn logs every request that reaches the webserver logic.If a request is logged it means the client accepted the respective certificate.Otherwise the request would have failed during the TLS handshake.As a baseline, we also log each request to the landing page, which uses a fully valid certificate.
Subdomain requests are grouped using a random identifier set on the landing page.We identify unique client devices with the tuple (IP address, User-Agent) and deduplicate requests on this basis.We employ basic filtering of non-user agents such as web crawlers by using a blacklist (see Appendix).
In order to have clients across the Internet access the server we use a pop-under adnet, which triggers a request to our landing page on the client.We use a pop-under rather than a pop-up, as these open in the background and are therefore more likely to remain open long enough to trigger requests to all subdomains.It is, however, still possible that the pop-under is closed before all requests are triggered, which means that the requests recorded at the subdomains are a lower bound on the requests that might actually have been made.Therefore, conclusions made based on the presence of requests to the subdomains are fundamentally more reliable than those made based on their absence.

Sampling Representation.
To obtain a more balanced geographic distribution of clients, we run three separate campaigns with equal budget: one for Asia, one for North America, South America and Europe and one for Africa and Oceania.The regions were grouped to achieve comparable numbers of Internet users per region, based on [16].The resulting distribution is summarized in Table 3, showing a broad global spread.Each round of measurements contains requests from ~200 countries.We set the remaining adnet parameters as untargeted as possible to get as close to random sampling of clients as possible.Nonetheless, some sampling bias cannot be avoided, as our measurement can only reach clients visiting sites served by our chosen adnet.
We compare the resulting distribution of browsers and OSes in our measured dataset to a largescale control dataset by Statscounter4 in Figure 4. Overall, our observed distributions in both measurements approximate those seen in the control dataset, indicating a fairly representative sampling.Nonetheless, the following deviations should be noted: Chrome and Android tend to be overrepresented, particularly in the 2023 measurement, while iOS and Safari tend to be underrepresented.
As shown in subsequent analyses, iOS and Safari make up a large portion of clients rejecting revoked certificates.As such, the vulnerability of clients may be overestimated to some degree.However, given the large fraction of observed clients that accept certificates which should be rejected, accounting for this difference in sampling still leaves a large majority of vulnerable clients.

Results
We begin with a broad overview of the collected dataset.4).rev: We found that, in relation to the baseline requests to the landing page, 87.4% (82.8%) of clients, connected to the rev subdomain, showing that the large majority of clients do not check revocation at all.This means that, at the time of writing, even with a fast response to a fraudulent certificate issuance, an attacker could still continue using a revoked certificate and exploit 82.8% of clients.
sta: If we include a valid OCSP response, that number rises slightly to 90.3% (85.5%) (see sta measurement).The fact that it is not 100% may be partially due to a) older clients that do not support OCSP Stapling and b) incompleteness of the measurement.If we discount clients that do not check revocation anyway, we find that 38% (35%) of remaining clients accept the sta certificate on the basis of a valid stapled OCSP response.mus: Additionally, we see that 89.5% (88.3%) of clients disregard the Must-Staple certificate extension (see mus measurement).Enforcing OCSP Stapling through Must-Staple is often discussed as a prime candidate for improving revocation.However, the disregard for Must-Staple suggests that client-side support for this is still severely lacking.This may be partially explained by clients running older browser and OS versions (see Figures 7 and 6), which lack support for newer standards.However, it is not sufficient, as the majority of measured clients run recent versions.In addition, current OCSP validity periods of 4 to 7 days (most commonly 7) still allow a large window of vulnerability, even if OCSP Stapling was strictly enforced.sct: Lastly, the success of CT rests on the premise that CAs are incentivized to log all of their certificates, because clients will otherwise not accept them.In 2022 only 29.6% of clients enforced this, while the remaining 70.4% ignored the absence of SCTs in the certificate.In 2023, however, only 31.2% of clients ignored the absence of SCTs.While there is some difference in OS and browser distribution between the two measurements, it is not large enough to account for the precipitous drop in measured connections accepting the SCT-less certificate.Taking a closer look at the We grouped clients by whether they did or did not connect to the rev subdomain, corresponding to the group labels "insecure" and "secure".For each group, we determined the distribution of OSes and browsers, extracted from the User-Agent (UA) string.Table 5 summarizes the top 5 out of these distributions.We see that the "secure" group contains significantly fewer Android and more iOS and MacOSX devices, which is in line with expectations based on our lab experiments.However, the difference is not as pronounced as expected, with a significant fraction of iOS, MacOSX and Safari agents remaining in the "insecure" group, as well as a large portion of Android and Windows devices in the "secure" group.While the latter can be explained by incompleteness of the measurement (i.e.clients disconnecting early), the former cannot.
This discrepancy with the expectations set by our lab experiment also shows up if we longitudinally compare the 2022 and 2023 measurements.Our overall results (see Table 4) show a 5% reduction in clients connecting to the revoked certificate, implying a slight shift over time to more secure clients.This cannot, however, be simply explained by a large number of Apple devices, since, as shown in Figure 4, the 2023 measurement actually contains a smaller percentage of Apple and a larger percentage of Android devices.
Since many different brands build on the Android platform, we also checked for differences in security of observed Android devices based on brand, but found none.All observed Android brands (e.g.Samsung, Oppo, Google, Xiaomi, Huawei) were fairly equally represented in both the "secure" and "insecure" groups.

Comparison of Client
Versions.Since our lab experiments were done on the newest available versions of each OS and browser, it is possible that the difference between "secure" and "insecure" clients in the wild is due to the software versions in use.To investigate this, we plot the percentage of "insecure" clients observed by major browser (Figure 6) and OS version (Figure 7), based on the User-Agent strings of measured clients.For clarity we plot only the top 8 browser families; for lack of version information for other OSes we plot only the top 4.Because version numbers differ dramatically between software vendors, we plot by release date.Additionally, the size of each data point indicates the total number of clients observed with this version.
Figure 6 clearly shows that newer iOS versions do not accept revoked certificates, whereas older versions do.This decreasing trend is not apparent in Windows, MacOSX or Android.On the browser side, Firefox, Safari Mobile and, to a lesser degree, Safari, show a drop in acceptance of revoked certificates in the more recent versions (see Figure 7), while none of the other prominent browsers show any correlation of revocation checking with version number.This leads us to hypothesize that whether a client performs revocation checking or not is in large part due to individual configuration.While some OS and browser vendors have moved towards increased revocation checking, we still see both the presence and absence of revocation checks across all OSes and browsers in the wild.

OS-Browser Interaction.
As observed in Section 6, revocation checking can be implemented in both the browser and the OS. Figure 8 summarizes this interaction as heatmaps, showing the percentage of "insecure" clients observed as a product of these two variables.We plot one heatmap for each round of measurements (2022 and 2023), which allows for two insights: First, there is an increase in revocation checking from 2022 to 2023 for iOS devices, as well as Firefox, which is in line with the declining degree of vulnerability observed for iOS, Firefox and Mobile Safari versions in Figures 7 and 6.
Second, revocation checking at the OS level appears to be the stronger driver compared to the browser level.The percentage of "insecure" clients is generally more comparable along the OS-axis: from 2022 to 2023 we see a decrease in revocation checking across different browsers on MacOS (with the exception of Firefox), as well as an increase in revocation checking across most browsers on iOS.In contrast, there is no browser that shows a clear presence of revocation checking across multiple platforms.

DISCUSSION
In this section we discuss the limitations of certificate revocation mechanisms and consider directions for research.

Impact of Fast Revocation on Attacks
Attacks based on routing hijacks will be still be unaffected by the current revocation process, as these last for only a few hours, e.g.[15,34].In contrast, existing Man-in-the-Middle positions operate undetected for much longer, but still require certificates to monitor encrypted traffic to targeted sites.For such attacks (e.g.[36]), CT and fast revocation have the greatest impact.
CT can also quickly alert domain owners to the presence of phishing sites targeting their domains, who can then report these to the respective CA.As shown in [17], HTTPS phishing sites are more successful than plain HTTP sites.Having a CA revoke a fraudulent certificate can often be easier than having the respective hosting provider take down the phishing site altogether (such as with bulletproof hosting [35]).Generally, the benefit of revocation depends on whether the duration of an attack is longer or shorter than the overall revocation procedure.

OCSP Expiration & Scaling Revocation
We see that the vast majority of clients is still vulnerable to revoked certificates, because revocation checks are ignored.This is because soft-failing (as shown in Section 6) is the norm.And thus, if revocation is ignored under adversarial conditions, the fraction of situations in which checks are still useful is minute.
In order for revocation to be effective, the WebPKI needs to support hard-failing revocation checks.CT and fast revocation by CAs do not benefit the security of the Internet, if clients do not properly respect certificates' revocation statuses.However, this requires reliable and fast delivery of revocation information.
It is fundamentally an availability problem, requiring robust, performant, DDoS-resilient OCSP responders.OCSP Stapling (in combination with the Must-Staple certificate extension) is often listed as a solution to reducing the strain on OCSP responders, as the number of servers is far fewer than the number of clients, and responses can be queried asynchronously and cached [7,28,30].
However, caching fundamentally trades off accuracy for performance, and choosing a long expiration time allows longer abuse.As we demonstrate in our study, an attacker can easily obtain and cache a valid OCSP response for a newly issued certificate and use it until it expires; most CAs today (7/9 in our study) use a validity of 7 days.In fact, to maximize the certificate lifetime, an attacker can continuously poll and store OCSP responses until a response shows "revoked", and then use the last "valid" response with OCSP Stapling until it expires.From a security perspective, we therefore advocate for shortening the validity period.This will result again in more OCSP requests and increase network load and computational burden on OCSP responders.
In the near-term, this could be done incrementally by CAs while monitoring the load on their OCSP responders, shortening the validity as much as possible until the load becomes unmanageable.In the long-term, a usable revocation system therefore requires more effective scaling.There have been suggestions for blockchain-based solutions, which may provide load-balancing through decentralization, but adds the unnecessary overhead of a consensus protocol and permanent transaction history.
It has also been suggested in an Internet-draft 5 that revocation could leverage the existing, robust infrastructure and mechanisms of DNS to deliver OCSP.DNS-based delivery could be rolled out incrementally without much coordination between servers and clients and further scaling DNS would be easier than creating a similarly robust system for OCSP from scratch.It bears revisiting DNS to provide the necessary performance, as an up-to-date certificate status is as important to a TLS-based Internet as an up-to-date IP address.

CONCLUSION
CT is effective in locating unauthorized certificates and our measurements show an increase in enforcement of SCTs from 2022 to 2023.However, CT is still of limited value for protecting clients from fraudulent certificates as long as the revocation system remains ineffective.Our longitudinal comparison of client revocation checking behavior shows that the ecosystem has changed little in this regard.Adversaries can still make use of revoked certificates for a significant period of time, for most clients until the certificate's expiration, in fact, which we have quantified and summarized in Figure 9.We thus conclude that certificate revocation in the WebPKI is still inadequate to protect the majority of users.
While some OS and browser vendors appear to be moving towards stricter revocation checking, the vast majority of clients (~83% in our dataset) skip revocation checks altogether, unless specifically configured otherwise, and all studied clients were found to soft-fail in response to unavailable revocation information.This renders the entire revocation process redundant.
The degree of vulnerability is extended by browsers caching soft-failed results.Current validity periods for OCSP responses also allow attackers to replay cached (nonce-less) OCSP responses to extend the lifetime of fraudulent certificates by up to 7 days.OCSP Stapling makes this even easier since the attacker does not require a Man-in-the-Middle position to manipulate OCSP communication, but can simply configure his server to include the cached OCSP response in the TLS handshake.
We see the improvement of the availability of revocation information as key to allowing clients to return to revocation checking and implement hard-failing policies and CAs to shorten OCSP expiration.Future work will need to investigate the relationship of shorter validity periods to increased load on OCSP responders to find a suitable compromise in the short term.
c e n s y s a p i c e n s y s w e b c e r t s p o t t e r a p i c e r t s p o t t e r e m a il c e r t s p o t t e r t o o l c e r t s t r e a m c lo u d f la r e e m a il c r t .sh c t -a d v is o r e m a il d ir e c t m o n it o r in g e n t r u s t f a c e b o o k e m a il k e y h u b e m a il p r o t o n m a il t r a n s p a r e n c y r e p o r

c o m o d o g e o t r u s t g lo b a ls ig n g o d a d d y le t s e n c r y p t a d m in le t s e n c r y p t c e r t b o t n e t s o lu t io n s r a p id s s l s s l. c o m t h a w tFig. 3 .
Fig. 3. Length of revocation procedure per CA.

Fig. 4 .
Fig. 4. Measured distribution of browser and OS families compared to large-scale control dataset 4 (matched for time of measurement).

Fig. 5 .
Fig. 5. Percentage of observed clients among the top 5 OSes and top 15 browsers (based on total number of observations) accepting a certificate without SCTs.White spaces indicate combinations that were not observed.

Fig. 8 .
Fig. 8. Percentage of observed clients among the top 5 OSes and top 15 browsers (based on total number of observations) accepting a revoked certificate.White spaces indicate combinations that were not observed.

Fig. 9 .
Fig. 9. Timeline of certificate validity based on measurement data.

Table 1 .
List of CT monitoring solutions studied.

Table 2 .
OS-Browser combinations tested with revoked certificates from eight different CAs.A green check mark indicates that the certificate was correctly rejected, while a red cross indicates that the certificate was mistakenly accepted.

Table 3 .
Geographic distribution of measured clients across continents and countries.

Table 4 .
Overview of measurement results.

Table 5 .
Comparison of OS and browser distribution (top 5) among secure and insecure clients (rounded to nearest percentage point).Comparison of Client Brands, OSes & Browsers.One question that remains is whether there are differences in security (declining revoked certificates) across device and browser families and versions.Our lab experiments in Section 6 set the clear expectation of a divide between Apple and Google (Chrome and Android) devices.