Deep Dive into NTP Pool's Popularity and Mapping

Time synchronization is of paramount importance on the Internet, with the Network Time Protocol (NTP) serving as the primary synchronization protocol. The NTP Pool, a volunteer-driven initiative launched two decades ago, facilitates connections between clients and NTP servers. Our analysis of root DNS queries reveals that the NTP Pool has consistently been the most popular time service. We further investigate the DNS component (GeoDNS) of the NTP Pool, which is responsible for mapping clients to servers. Our findings indicate that the current algorithm is heavily skewed, leading to the emergence of time monopolies for entire countries. For instance, clients in the US are served by 551 NTP servers, while clients in Cameroon and Nigeria are served by only one and two servers, respectively, out of the 4k+ servers available in the NTP Pool. We examine the underlying assumption behind GeoDNS for these mappings and discover that time servers located far away can still provide accurate clock time information to clients. We have shared our findings with the NTP Pool operators, who acknowledge them and plan to revise their algorithm to enhance security.


INTRODUCTION
Global time synchronization underpins modern life.It is crucial to the Internet and to critical systems such as financial markets, power grids, and telecommunications networks [30].In businesses, precise clock information is also vital: distributed systems and applications such as backup systems are entirely dependent on precise clock information [40,70].Operational failures can occur whenever clocks are unsynchronized, potentially leading to data loss [27].
The Network Time Protocol (NTP) [50] is the Internet's default protocol for clock synchronization.It is designed to mitigate the effects of changing network latency (jitter) between client and server.NTP servers synchronize out-of-band with high precision references, such as atomic clocks, radio signals (e.g., DC77 [10]), and satellites (GPS and Galileo).Clients and other secondary NTP servers, in turn, synchronize their clocks with NTP servers over the Internet.Clients either use servers they have been pre-configured with (e.g.,, /etc/ntpd.conf)or servers provided by their networks with DHCP [21,24].The Precision Time Protocol (PTP) [31,32], in turn, improves NTP's precision, but typically requires layer-2 (LAN) access.PTP is often used in financial transactions, mobile phone towers and other industrial networks.
The NTP Pool [65] provides a layer over NTP servers, providing a directory of publicly available NTP servers using DNS [51]; it does not directly operate NTP servers.The NTP servers themselves are run by volunteers, which range from home DSL users to large cloud operators.The NTP Pool currently lists 4,403 volunteer NTP servers, with 3,056 on IPv4 and 1,671 on IPv6 (2023-10-09) [61].It has been operating for more than two decade, being popular among vendors [66,78], including various Linux distributions and Android devices.
Our first contribution ( §3) is to demonstrate that the NTP Pool is not only in active use, but it has consistently been the most popular time-service provider on the Internet, based on DNS traffic at the Root DNS servers [77].This popularity persists even with the introduction of new time services introduced by large vendors in recent years.
Our second contribution is to demonstrate how these mappings are executed and which criteria are employed in this process.We examine GeoDNS [6], the NTP Pool customized DNS software that perform the mapping, complemented by measurements taken from the public Internet Clarifying this process is important given the popularity of the NTP Pool.
Our third contribution ( §5) is to explore the implications of this mapping, from our ability to predict which NTP servers a client will use.We find that assignments can be heavily skewed, producing time service monopolies.Even with more than 4k NTP servers, 27 countries are assigned to a single time provider-one operator serves 767M people and 465M Internet users.In addition, we find that another 101 countries and territories (comprising 260M Internet users) could be monoploized with the deployment of a single NTP server.This monopolization bestows immense power upon a single actor [60], which can then be misused (or exploited) to execute nation-wide scale time-shift attacks, particularly worrisome in today's world where conflicts extend into cyberspace.
fourth and final contribution ( §6) is to show that the current GeoDNS mapping algorithm can be changed to improve server distribution without compromising service quality.Conversations with NTP Pool operators indicate that these mappings are designed to avoid asymmetric routing and alleviate concerns about packet loss.However, our experimental results contradict these apprehensions about substantial packet loss from distant servers: we demonstrate that far away NTP servers can also deliver high-quality timing services with minimal packet loss ratios.Consequently, we recommend that NTP Pool operators consider modifying their mapping algorithm to address these monopolization issues, which could potentially result in a complete or partial time synchronization takeover for entire countries ( §7).

THE NTP POOL PROJECT
The NTP Pool project is a dynamic collection of thousands of NTP servers that provide accurate time via the NTP protocol [50] to clients worldwide.These NTP servers are assigned to clients using DNS, under the pool.ntp.orgzone.It was proposed as a solution to reduce the abuse of publicly available NTP servers [86].Instead of having a long list of public NTP servers (which individually could more easily become overloaded), the NTP Pool project proposed to "load balance" NTP traffic using DNS.The project has been active for more than twenty years.The NTP Pool does not run NTP servers; volunteers run their own NTP servers which they add to the NTP Pool using a web interface [67], where they can also choose how much capacity they want to donate (values ranging from 512Kbps to 1Gbps).We show this process on the left side of Figure 1, where IP addresses are added with their respective  capacity parameter.
To use the NTP Pool for clock synchronization, clients' time software is set with domain names from the NTP Pool DNS zone, as shown in the right side of Figure 1.In this figure, we see a client that uses the domain names [0--3].debian.pool.ntp.org in their config files (e.g, /etc/ntp.conf).Whenever needed, the client will ask its local DNS resolver to fetch the IP addresses for these domain names (B in Figure 1), and the NTP Pool authoritative DNS servers ([a--i].ntpns.org)will answer with a list of IP addresses (C) to the DNS resolvers, which will then forward them to the clients.The NTP Pool runs GeoDNS, their customized authoritative DNS server software [6].Upon receiving NTP servers addresses from the NTP Pool authoritative servers, the client's time synchronization software will contact these NTP servers for accurate time information.
Do clients blindly trust NTP servers?Consider a malicious NTP server that provides wrong time data -say, ten years ahead of the current time.To prevent attackers from tampering with the client's clock, the NTP protocol has built-in algorithms ( §10 and §11 in [50]) that continuously evaluate the timing samples from various NTP servers, disregarding discrepant ones.Moreover, it can combine time information from multiple NTP servers to update the system's clock.Figure 1 shows an example of a NTP reference implementation client status [57].It shows 8 NTP servers under evaluation -all retrieved from the NTP Pool by the client (NTP clients use multiple NTP servers if they are available to select the best ones).Each server has several metrics (offset,

Client
Recursive DNS Resolver

Root
.com apple.com Fig. 2. Time servers domain name resolution.
delay, jitter), which are all part of the NTP protocol filtering specifications.This particular client combines data from all servers with marked with '*' and '+' symbols on the left side of the IP address to synchronize its clock.In this way, NTP implementations prevent the harmful effects of individual malicious NTP servers.The exception occurs when a server reboots and may trust whatever time information is provided withor when ntpd is ran with the -s option.SNTP [49], which is a simplified version of NTP designed to provide basic time synchronization functionality with minimal overhead, will blindly trust the time information provided by time servers.Even if an SNTP client receives multiple NTP servers from the NTP Pool, it will use only a single server.In Figure 1, we show the status of systemd-timesyncd, a SNTP implementation running on Ubuntu, where we see a single NTP server.
How does the NTP Pool prevent malicious volunteers?Anyone can add an NTP server to the NTP Pool.To prevent malicious or bogus NTP servers from being assigned to clients, the NTP Pool operators constantly monitor every volunteer NTP server.Bogus servers are removed from the zone and not served to the clients.(We demonstrate in §4.2 how this system works).
Changing system configurations: clients are typically configured with pre-configured NTP servers.They can manually change it or, if available, use the NTP servers provided by the DHCP [21] protocol, which also enables dynamically setting NTP servers.For example, one of the authors institution provides NTP servers via DHCP, which causes the NTP client on Linux boxes to not use the NTP Pool while connected to the institution network.

EVALUATING THE IMPORTANCE OF THE NTP POOL
Given that several large cloud and content providers now have their own NTP services (Microsoft, Google, Facebook, Cloudflare, and Apple), one may wonder how relevant is NTP Pool still for keeping time on the Internet.In areas such as DNS resolution, no-cost services by commercial providers quickly captured a majory of their market [52].Has the same happened for NTP, reducing the NTP Pool relevance?
The direct way to answer this question would be to compare NTP traffic across different time providers.That, however, would require access to vantage points inaccessible to us.Thus, we address this question by comparing the NTP Pool popularity with other NTP services by analyzing DNS traffic collected at the Root DNS servers [77].

Root DNS and time keeping
Before synchronizing clocks with NTP servers, clients must first resolve the domain names associated with the time service.Consider Figure 2 as an example, where a client first (step 1) sends a DNS query (time.apple.com) to its recursive DNS resolver, typically provided by its local ISP.This recursive resolver converts the name into the IP address providing service, either from its cache or by asking one more more authoritative name servers.
Assuming nothing is cached, the resolver must first contact one of the 13 Root DNS servers, asking for the authoritative server of .comzone authoritative servers (step 2 in Figure 2).The recursive resolver learns where .com,is, then asks the .comauthoritative DNS server for authoritative servers of apple.com(step 3).Next, the apple.comauthoritative servers can tell the resolver which IP addresses are associated with time.apple.com(step 4) and finally can answer its client.The client then uses the IP addresses to synchronize its clock using the NTP protocol.
While we cannot see DNS traffic from each time provider, we have access to traffic from the Root DNS servers with DITL datasets [20], a two-day-a-year traffic capture of the Root DNS servers.The Root DNS traffic provides a view of the top of global DNS traffic [13,28,41].DITL can provide a lower bound estimate of the number of time services users.

Limitations
The DITL datasets from the root DNS have several limitations regarding our research question: They do not see real clients: The root servers only see recursive resolvers used by clients (Figure 2), not actual clients.Since recursive resolvers employ caches and may provide cached results to many clients, observations at the DNS root do not allow us to tell how many are behind the resolver.
Caches also hide repeated requests.Not only will caches hide multiple clients, they also hide repeated requests by single clients (to reduce latency [54,55]).
Query name (qname) minimization hides services: qname minimization [9] improves user privacy by provided only a single DNS component to each authoritative DNS server.For example, instead of querying the root DNS servers for time.apple.com(which the roots cannot directly answer, they can only point to where the .comauthoritative servers), a recursive resolver using qname minimization will only query the root for .com,hiding the full name.While study has shown that qname minimization is still not widespread [17], its use is growing [42].Our method applies only to resolvers that do not use qname minimization.
Localroot avoides Root servers completely: A recent informational RFC suggested a mechanism where recursive resolvers preemptively fetch an copy of the root zone, allowing them to avoid querying root servers entirely [38].We believe localroot is used far less than qname minimization, but we cannot see traffic for recursive resolvers using this mechanism.

Datasets
There are thirteen root DNS servers on the Internet.Each one is referred to by the first letter in its name ([a--m].root-servers.net).We analyze traffic collected at twelve of the thirteen root servers -the Day In The Life of the Internet (DITL) 2022 datasets [20]; we omit I-Root, since it's DITL datasets anonymizes IP addresses, preventing our analysis.In addition, E-root provides only partial data.This dataset was taken from April 12-14, 2022.For a historical comparison, we compare against data at ten servers from DITL 2017 dataset, with data taken from April 11-13, 2017.
For each query , we extract its query name and match it against a list of server names used by time providers we compiled using multiple sources (Table 1).We then compute the number of unique queries, clients, and autonomous systems (ASes) for each time provider (we use CAIDA's Routeviews Prefix2AS datasets to map IP addresses to ASes [12]).
Table 2 shows the DITL datasets after processing.In 2022, we identified 126 million queries from 491 thousand resolvers and 22 thousand ASes that queried for names matching the domain names in Table 1.We notice that the distribution varies per root letter as resolvers employ their own criteria to choose which root server to contact [56].

Comparing time services
Table 1 lists the 10 main and 137 other NTP services we consider, as well is the cache durations (TTLs) for each.We also include their top-level domain (TLD) TTL, such as .comand .net.To appear in DITL, both TLD and server name records must be expired from cache (DNS records are cached independently [54]).
Figure 3a shows the query distribution per time provider from the DITL datasets.We see that NTP Pool receives roughly 90M out of 126M queries in total, being far more popular than all other time providers combined.
These query counts, however, may be inflated in favor of the NTP Pool servers, due to two main reasons: DNS time-to-live [51] values and multiple NTP Pool subzones.
TTL values associated with DNS records determine how long resolvers will cache the results.TTL values can range from 1 second to approximately 65 years [51], but many resolvers cap it at a maximum of two days, and many respect the TTL values [54,55].Operators are free to choose the TTL values that best suit their needs.Table 1 shows a list of time providers domains and their associated TTL values.In the case of the NTP Pool, the DNS records have a TTL of 150 seconds, which means that once a resolver obtains a response, all subsequent client queries within 150s are responded to from the cache instead of triggering new queries.These cache-hit queries we do not observe ( §3.2).
To compensate for the differences in TTL values, we also compare time providers using two other metrics: the number of unique resolvers (source IP addresses) and the number of unique ASes associated with them.Both metrics are not inflated by repetitive queries due to low TTL valuesthey count one resolver or AS once per dataset, regardless of the number of queries sent.For both resolvers and ASes, we still see the same pattern: the NTP Pool is the most popular, followed by NIST (Figure 3b and Figure 3c).
A second reason queries may be artificially high is for pools that use multiple subdomains.In addition to the general pool.ntp.org, the NTP Pool has many subdomains that allow clients to make queries to servers by geography (such as europe.pool.ntp.org) or vendor (such as android.pool.ntp.org).As such, this large number of subdomains increases the query volume in compared to other providers that use one or few domain names.
To rule out the effect of multiple subdomains, we single out queries to pool.ntp.orgonly, which is the general NTP Pool zone (not used by vendors).In total, 25M queries from 138k resolvers and 10.2k ASes have queried for it.For comparison, NIST in 2022 had 6.8M queries from 145k resolvers and 14k ASes -thus more resolvers and ASes than the general NTP Pool zone.
Changes over time: when we compare the results from 2022 with 2017, we see that the NTP Pool was also the most popular service by any metric.We also note that newcomers did not have a significant impact in 2022 (Facebook and Cloudflare did not offer time services in 2017).
Despite the large cloud providers entering the market later, we observe that the NTP Pool is still the most queried service in terms of DNS traffic (at least a the Root level), followed by NIST.Even though Microsoft, Ubuntu, Google, and Apple set their devices to use their own time services, the NTP Pool attracts more resolvers and ASes, at least at the DNS level.
NTP Pool subzone use: We found 158 vendor zones and 104 geographical NTP Pool zones, using the DITL 2022 datasets.Vendor zones had 39.2M queries, while geographical zones had 25.3M queries.The general zone (pool.ntp.org) had 25.2M queries.We show them in §B.
NTP Traffic from one server: we run an NTP server as volunteers within the NTP Pool.Our NTP server [81] serves multiple regions and uses IP anycast.We have collected 24 hours traffic during Jun 22-23 2022 and observed 7.2B queries from 158M resolvers from 52,014 ASes globally [8].For comparison, in 2016, NIST reported 16B daily queries [80].Bear in mind that this is a single server out of the more than 4k listed at the NTP Pool.

CLIENT-TO-SERVER MAPPING
The NTP Pool utilizes GeoDNS for the mapping of clients to volunteer NTP servers (middle box in Figure 1).It could be argued that examining the GeoDNS source code alone should be adequate for comprehending the mapping criteria.While this is a valid point, it is important to note that a code analysis alone cannot be applied to determine the specific servers assigned to real-world clients.This is because such analysis does not encompass the dynamic state of the NTP Pool, which is defined by the list of NTP servers and their performance metrics.These are used to derive the input files of GeoDNS, which are frequently changing.
Hence, it is crucial to consider the state of the NTP Pool, encompassing the list of volunteer servers, their configuration parameters, and their status.This comprehensive understanding can only be attained through active Internet measurements.Bearing this in mind, we perform two types of measurements: (a) in a real-world scenario, employing 9.2k vantage points (VPs) ( §4.1), and (b) in a controlled environment ( §4.2).

View from the wild
The NTP Pool operators list that 4.7k NTP Servers (2023-10-10).We seek to understand the logic between client/server mapping and its implications for real-world clients, given the population of NTP servers.A previous work [78] observed the client for a single VP in Germany was mapped to NTP servers located in Germany by the NTP Pool.However, it did not explore the reasons why and how.
To understand the NTP Pool mappings in practice, we set up two measurements (for IPv4 and IPv6) using 9.2 thousand VPs using RIPE Atlas probes [75,76] (RIPE Atlas probes are hardware devices or virtual machines (VMs) that can be remotely instructed to carry out active measurements).In total, our VPs cover 3,082 ASes in 166 countries.
We configure these 9 thousand Atlas probes to send DNS queries to one of the NTP Pool authoritative servers (b.ntpns.orgover IPv4 -185.120.22.23),so we bypass DNS resolvers (Figure 1) and avoid hitting the resolver's cache.By passing resolvers, we can retrieve new NTP Pool addresses for every new query.The probes are configured to send queries every 5 min -a safe limit that does not overload RIPE Atlas and the NTP Pool authoritative DNS servers.
Table 3 shows the experiments' details.In the first experiment (EnumV4), we configure Atlas probes to retrieve IPv4 NTP servers, whereas in the second (EnumV6) we retrieve IPv6 NTP servers.For both experiments, we see ∼9.2 thousand active VPs, having 9.1 thousand received valid responses (some VPs are blocked or contain bogus responses -a problem reported in Atlas probes in other works [53], which we disregard).These 9.1 thousand VPs provide us with a view from ∼3 thousand ASes, totaling ∼2.5 million DNS queries/responses per experiment.
For each experiment, each Atlas VP sends roughly 275 queries, receiving up to 4 NTP server addresses per response (Table 3).Theoretically, this would allow each probe to retrieve up to 1,100  unique NTP server addresses from the NTP Pool, if the process were completely random and if each client would not receive repeated NTP servers (we refrain from running an experiment that could span over all servers to avoid overloading RIPE Atlas).
Figure 4 shows the cumulative distribution function (CDF) of the number of unique IPs retrieved by each probe.We see a very different distribution of NTP servers per probe for both IPv4 and IPv6.Roughly 10% of the clients see up to 12 NTP servers (EnumV4) and 5 NTP servers (EnumV6).Considering that the NTP Pool has thousands of NTP servers, it is quite remarkable such a limited number of assigned servers.Next, we explain the reasons behind these differences.

Controlled experiment
We first replicate the NTP Pool authoritative DNS server setup and then replay the DNS queries from our experiments in the wild.By doing that, we can obtain the GeoDNS logs from our own instance and use this information to understand how it maps clients to servers.
GeoDNS (v.3.0.2) takes as input a DNS zone file, that lists all zones in the pool (geographical and vendors) and their respective NTP servers.The GeoDNS source code does not have the actual zone file used by the NTP Pool, and we were not able to obtain them from the NTP Pool operators.It contains a demo zone file, which we use as starting point.We refrain from doing source code analysis given it does not include the zone files, which are a product of the NTP Pool monitoring systems and the networking conditions of each server.Therefore, we need to an empirical measurement to determine the status of the servers and understanding how the mapping works in practice.

4.2.1
Reversing the NTP Pool DNS zone.We resort to reverse engineering the NTP Pool zone files (sample in Appendix §C).We start by using the demo zone file available with the GeoDNS source code and populate it with servers that we have found with EnumV4 and EnumV6 experiments, in the following way: (1) Generate a list of all NTP servers from EnumV4 and EnumV6 measurements (2) Retrieve metadata (DNS zones) from each NTP server from the NTP Pool website (3) Populate the demo zone file using the retrieved metadata In the first step, we obtain 3,056 NTP server addresses.We then crawl each of them from the NTP Pool website using each their IP address.Each NTP server in the pool has a dedicated page (in the form of https://www.ntppool.org/scores/IP),which lists the zones the NTP server is associated.For example, the NTP Pool page for 95.217.188.206shows that this NTP server is allocated to the global (@), europe, and Finland's fi zones [64].Then, we assigned this particular IP address to these subzones in our reverse-engineered zone file.We repeat this process for all 3,056 IPv4 and 1,479 IPv6 addresses we found from EnumV4 and EnumV6.
In the GeoDNS zone files, each NTP server has a weight associated with it, which is derived from how much service capacity the volunteer wants to donate to the pool ( in Figure 1).In practice, servers with higher weight values are picked more often.For example, a server with a 100 weight will be seen 100 times more often than a server with one weight.We demonstrate the weights influence in Appendix §D.
Our reverse-engineered zone file has 126 non-empty zones in total -all country and continent zones (we found no vendor zones using this method).We found 125 zones for IPv4 and 112 for IPv6.We also found many countries (101 for IPv4, 145 for IPv6) that have zero servers in their zones.

4.2.2
Validation.The next step consists in replaying the DNS queries from the EnumV4 experiment on our controlled environment.We use the reverse-engineered zone file on GeoDNS and use Maxmind's GeoLite2 country IP2location database [47] from 2021-08-24 -a required input by GeoDNS to operate.
Client setup: To replay the queries from EnumV4, we send spoofed IP packets (forged source IP addresses [22]), using a customized Python script, and run our experiment on our server disconnected from the Internet -so our spoofed packets cause no harm.
Collected datasets: we collect two datasets, namely, network traces (pcap files), and GeoDNS log files (Listing 1, which lists the metadata associated with each DNS query and response), both from the same Linux server.We refer to this experiment as EnumV4-emul.
By analyzing GeoDNS log files, we see how the mapping occurs: first, the client's geographical information is retrieved from MaxMind's database (country and continent).These are used to populate a list of candidate zones that can be used to answer this client, which is shown by the Targets tag (Listing 1).Then, the tag LabelName shows which zone the client has been mapped.For this particular client, we see it could have gone to Israel (il), Asia, or the Global (@) zone, and it was ultimately mapped to Israel's zone.The logs do not show, however, which NTP servers were included in the DNS response.We analyzed the pcap files and confirmed they belong to the Israel's zone.
Table 4 shows the results.We see three main categories: Equal shows that zones and VPs matched perfectly in the wild and our controlled experiments.These comprise 2.2 thousand VPs from 93 zones.The second category is More, in which the VPs in our controlled experiment saw more NTP servers than those in the wild.These comprise most VPs (7.2k, 66 zones).We speculate this can be due to the use of uniform weights (1,000) in our emulation experiment, in which each server gets the same odds of being included in the response.In the NTP Pool, however, these weights vary by a factor of 2,000.As such, our Emulation retrieves most if not all servers in the zone, while in the wild, the distribution would have been shifted to servers with higher weights (see Appendix §D).
The last category is the more concerning one: 47 VPs in our controlled experiment saw fewer NTP servers than in our emulation experiments.We believe this may be due to two reasons: their DNS traffic being intercepted and ultimately to send to resolvers elsewhere, and dynamic changes in the NTP Pool NTP server population along our measurements.Next, we cover the second reason.

NTP Pool monitoring system
The second reason is that the NTP Pool continuously monitors the volunteer's NTP servers.Poorly performing servers (unreachable, providing incorrect time data) have points deducted up to a threshold and are evicted from the NTP Pool zone file if they cross this threshold (10 points).While evicting servers from zones should not change much our results, they change in a specific case: if a country zone has a single server and the server is evicted.If this happens, then the client will, from that point on, be mapped to its respective continent zone.
This case covers 34 VPs that see only one NTP server in our experiment hosted in Cameroon, Guernsey, and Reunion -the latter two islands belonging to the United Kingdom and France, respectively.These VPs are mapped to a single NTP server in their zone that was eventually evicted from the NTP Pool zone due to poor performance.This caused these VPs to fallback to its continent zone (Europe), which has many servers.
We demonstrate that with VP 17580 located in Guernsey.The EnumV4 experiment (in the wild) shows that this probe sees, in total, 21 unique NTP servers -even though its associated territory zone (gg) zone has only one NTP server.We plot the responses seen by this VP in the wild in Table 5.This VP initially receives a single NTP server in the DNS responses -51.255.142.175anNTP server from Guernsey until 20:56.From 21:01 to 21:21, this VP receives 20 different NTP servers in 4 subsequent queries.These 20 servers belong to the europe zone, suggesting t this VP was mapped during this period to europe zone and not gg zone, which seem to have been empty.
While this shows that the probe sees more servers from Europe's zone, it does not show its country zone was empty at the same time.To show that, we analyze the scores associated with this NTP server from the NTP Pool website [63]) -a measurement carried independently from ours.We correlate our measurement results with the NTP Pool's server score logs, as seen in Figure 5.We see that this particular server score dropped below 10 -the minimum value for it to be used in the NTP Pool zones, otherwise a server is evicted -between 20:55:32 and 21:22:04 (2021-08-26, UTC).We show this in the gray area.
This period of scores lower than 10 coincides precisely when this VP receives 4 NTP servers per response (Table 5).Given these low scores, we can infer that this NTP server was likely evicted from the gg zone in this period.Since this was the only server in the zone, GeoDNS mapped this VP to its respective Continent zone (europe).Once the NTP server's score surpasses 10 again, after 21:22, we see the client again receiving 1 NTP server in the DNS response, likely from the NTP server joining the gg zone again.
New monitoring system: On March 2023, the NTP Pool operators released a new monitoring system, which consists of multiple measurement servers across the globe instead of a single one in California [69].(Its beta version has been evaluated in [39]).Each NTP server is now evaluated by five monitoring servers instead of one, and the scores of the five are combined [68].This prevents that failures on a single monitoring server wrongfully evicts well-performing servers.Whereas the scoring logic has changed, the eviction process and reinsert has not.
Vendor zones: vendors zones (such as debian.pool.ntp.org)behave like the default zone: a client will be first mapped to its country of origin, and if its zone it is empty, to its continent.We show it in §E.
Bypassing GeoDNS: Users have the option to bypass GeoDNS mappings by adjusting their system settings to query their preferred zones.For instance, a user in Japan can query it.pool.ntp.orginstead of the default pool.ntp.org to use Italian-only instead of Japanese NTP servers.However, this process requires change to the system settings.As a result, it is reasonable to assume that most users will not undertake this task.Instead, they are likely to rely on their default settings (most likely vendor zones), which by default, resort to GeoDNS mappings.
ECS support: We also found that if a resolver includes the client's EDNS client subnet (ECS) [15] in the DNS query, GeoDNS will then use the ECS IP address to map the client.
Takeaway: we have shown how GeoDNS maps clients to NTP servers.Clients are constrained by GeoDNS to NTP servers available in their respective country subzones.Clients in countries without NTP servers fallback to the continent or global zones.

PREDICTING MAPPINGS FOR ANY CLIENT IN THE WORLD
The experiments from §4 have demonstrated the process by which GeoDNS maps clients to NTP servers.However, these experiments are constrained to the clients for which we had vantage points, encompassing 166 countries in total (Table 3).Now that we understand this mapping process, we can expand our analysis to include all countries globally.This will allow us to assess the fairness and uniformity of the distribution of the 4k+ NTP servers among the entire client population.

Methodology
We first start with the analysis of all the GeoDNS DNS subzones we identified ( §4.2).For each country/territory subzone, we show in Figure 6a the number of NTP servers that its respective GeoDNS subzone has (countries shown in white have zero NTP servers).Next, we aim to determine the subzone to which the client countries will be assigned.As discussed in §4, if a country hosts NTP servers in its subzone (countries depicted in color in Figure 6a), then all its clients will be assigned to the corresponding country subzone (for instance, all Canadian clients will be assigned to ca.pool.ntp.org).This is shown in Figure 6b, represented in gray.
For the remaining countries, we need to determine if they will be mapped to either their continental or the global zone.To do that, we emulate VPs from these countries by identifying IP addresses from these countries and querying our local instance of GeoDNS in a controlled environment.Wy querying the Maxmind database for every /24 on the IPv4 space, we identify a total of 246 countries/territories.This is 80 more than our VPs as referenced in §4.1.For each of these countries/territories, we obtain an IP address.
Then, we send DNS queries using these IP addresses to our local GeoDNS instance and observe the mapping results (Figure 6b).We find that most countries with no NTP servers in their subzones are mapped to their respective continent zone.For instance, users in Morocco are mapped to africa.pool.ntp.org, which hosts 51 servers.Interestingly, we discover that 8 countries and territories, including South Sudan and Antarctica, are mapped to the global zone (highlighted in orange).
With the established mappings (client country → GeoDNS subzone), our next step is to ascertain the number of NTP servers available in each subzone.This will allow us to determine the fraction of the 4k NTP servers that each client will be served by.To achieve this, we query the NTP Pool authoritative DNS servers 1000 times for each previously identified subzone.In total, we send 451k IPv4 queries and 337k IPv6 queries on 2022-11-24.Utilizing the mappings obtained from all countries/territories, we plot the clients' visibility, i.e.,, the effective number of NTP servers out of the 4k+ from the NTP Pool that serves all clients within a country.
To minimize the impact on the NTP Pool authoritative DNS servers, we select a random authoritative server before each query, thereby distributing the load among the 29 available servers, and we incorporate frequent pauses between measurements.

How fair is the NTP Pool mappings?
Figure 7 shows the number of NTP servers allocated to clients from each country.We observe a significant variation in the distribution of NTP servers among clients.For instance, clients in the US and Greenland are served by more than 500 servers, whereas clients in Egypt, Israel, Nigeria, and 24 other countries are served by only 2 NTP servers.Worse, Laos and Cameroon have a single NTP server allocated to it.Given that the NTP Pool comprises more than 4k NTP servers, we deem this skewed distribution as highly skewed and unfair for the client population.
This mapping scheme also discourages countries without NTP servers from contributing to the NTP Pool.It is more advantageous, in terms of diversity, to be mapped to their continental zone (with a larger number of servers) than to be mapped to a few local servers in their country.For instance, Peru, situated in South America, has 3 NTP servers in its zone, hence all its clients are mapped to them.In contrast, Bolivia has none and is therefore mapped to the South American zone, which consists of 50 servers.As a result, Bolivian clients have greater NTP server diversity than Peru, despite having no local NTP servers listed in the NTP Pool.

Time providers per country.
Next, we calculate the number of time service providers for each client country.Considering that a single time provider may operate multiple servers, it is crucial to understand the implications of a time provider failure.To map NTP servers to providers, we utilize their associated AS number, retrieved from CAIDA's IP to AS maps [12].Subsequently, for each client country, we compute the number of unique ASes serving it.
Figure 8 shows the number of time providers serving each client country.We observe that 27 countries, depicted in red, are served by a single AS (also shown in Table 6).Collectively, these countries account for 767M inhabitants and 465M Internet users.Countries in dark orange have all their NTP Pool clients served by only two time providers.These include Uruguay, Saudi Arabia, and Tanzania, encompassing 505M inhabitants and 288M Internet users.

Users per NTP server.
The final metric we employ to evaluate the fairness of the NTP Pool pertains to the number of Internet users assigned to NTP servers.This metric complements the previous ones by providing an estimate of the client population per NTP server.To compute the number of users per NTP server, we divide a country's Internet population (obtained from the ITU entry [33]) by the number of NTP servers that serve that zone (Figure 7).
Figure 9 displays the results, revealing a highly skewed distribution.Nigeria, with only two servers, averages 60M Internet users per server, followed by Egypt (40M) and Iran (34M).In stark contrast, Germany has 150k users per NTP server.These mappings result in a skewed distribution of users by NTP servers, with clients in both wealthier nations and countries without NTP servers in the NTP Pool receiving more NTP servers than the rest.
These time monopolies for entire countries poses security concerns.Ultimately, this concentration is an example of centralization and consolidation on the Internet [3-5, 35, 36, 52, 60, 79, 82].In this case, the main drive is not large companies dominating a market -it is rather due to the current GeoDNS mapping algorithm.Next we discuss its security implications.

Security Implications
Despite the NTP Pool having over 4K+ NTP servers, these servers are distributed unevenly among the client population.This concentration of servers can be exploited by an attacker in two ways.Firstly, an attacker may aim to hijack all traffic from a specific country by hosting a single NTP server within it and adding it to the NTP Pool.We have demonstrated how a single NTP server from Guernsey serves as the sole provider for the entire region ( §4.3).This type of attack was first demonstrated in [78, §6.2],where the authors added an IPv6 server to the NTP Pool.Although they attracted significant traffic with their method, they did not demonstrate a monopoly, as we have done.Notably, all countries in blue in Figure 6b, including Albania, Bolivia, Tunisia, Namibia, Venezuela, and Guatemala.This includes 101 countries and territories for IPv4 and 145 for IPv6, covering about 260M Internet users.
Another attack against the NTP Pool does not require monopolizing traffic; an attacker can simply control a portion of the traffic.The attack involves introducing numerous NTP servers into densely populated zones.This tactic aims to create a race condition, thereby increasing the likelihood of clients being served by their designated NTP servers (while maximizing the  value, as shown in Figure 1).Particularly, in countries with fewer servers, the task of diverting a significant portion of the traffic becomes notably more manageable.
In both cases, an attacker can then execute time-shifting attacks on clients, which involve altering their clocks.In this scenario, an attacker can distinguish between real clients and NTP Pool monitoring servers by initially configuring their NTP server to operate in a "monitor only" mode.In a manner similar to previous studies such as [72] and [39], the attacker can provide accurate time readings to the NTP Pool's monitoring servers while deliberately providing incorrect time to other clients.This manipulation effectively circumvents the NTP Pool's eviction system [72].

GEODNS MAPPINGS: ARE THEY SOUND AND THEIR IMPLICATIONS
In §5, we demonstrated that GeoDNS employs a restrictive client-to-NTP server mapping approach, which imposes limitations on the number of NTP servers available for serving clients.We contacted the NTP Pool operators [7] and they argued the idea behind these mappings is twofold: reduce the risk of asymmetric routing and to minimize packet loss.
Asymmetric routing occurs when incoming and outgoing network traffic for a given connection follows different paths, can have a significant impact NTP and potentially lead to synchronization issues and time inaccuracies [26]  Packet loss can also impact clock synchronization, given NTP responses may simply not arrive.To determine whether these packet loss concerns are sound, we carry out active measurement experiments next.

Can far away NTP services provide good time information?
Is it possible for clients to receive accurate time service from servers located in distant regions, rather than being limited to restrictive in-country mappings?To examine this hypothesis, we undertake an experiment employing 132 RIPE Atlas probes as vantage points.These probes are drawn from 21 countries that are presently solely served by Cloudflare (highlighted in bold within Table 6).It's worth noting that the majority of these countries are situated in Africa, the Middle East, and Southeast Asia, as opposed to regions like the United States or Europe, where more favorable outcomes might be anticipated.
Our objective is to ascertain whether clients in these countries experience no significant packet loss among all servers.To establish a baseline, we compare the service offered by their current sole time provider, Cloudflare, with five additional NTP servers from the NTP Pool.We choose one server per continent, with our choice based on the NTP server that exhibits the highest frequency per zone, as shown in §5.We configure these Atlas VPs to conduct queries every 30 minutes over one week (Dec.[16][17][18][19][20][21][22][23]2022).This extended observation period enhances our chances of identifying potential failures.
The details of our experiments are consolidated in Table 7.It provides an overview of the specific NTP servers for which we configured Atlas probes to sent NTP queries.Over the span of one week, we received a total of 33,000 to 36,000 queries per NTP server, with one notable exception being the NTP server located in South America.This server received 21,000 valid responses but from a reduced pool of only 90 probes.
Lack of NTP Responses: We compute, for each Atlas Probe (VP), the ratio of NTP queries that receive no response.It's important to note that RIPE Atlas does not provide specific reasons for this lack of responses; it could be due to timeouts, filtering, or other factors [37].
In Figure 10, we show a CDF of the Atlas VPs and their respective rate of unanswered queries.We see that 90% of our VPs have no queries loss for the Cloudflare, Europe and North America servers, despite many of the VPs being in Africa, Asia, and Middle East.For the Asia NTP server, we see that 80% of the VPs have up to 10% unanswered queries.Only the South American server has not particularly good results: 40% of VPs have more than 50% of unanswered queries.Even though we cannot determine the reasons why this South American server performed worse than the others for the same VPs, we see in Figure 11 that the same VPs that failed to retrieve responses from the South American server could retrieve responses from the other servers.In this figure, we show the ratio of non-responses per probe (each entry in  axis is a Atlas probe) and per time server ( axis).As such, we can disregard issues with specific Atlas probes, as they could retrieve responses from other servers.In practice, if these they run a NTP client, they would likely disregard the South American server ( §2), so it would cause no harm.
We next set out to compute the quality of timing data provided by each server.For every NTP response, we extract its offset value, indicating the time difference in seconds between the local probe clock the time provided by the NTP server.To mitigate the effects of clock drifting, we exclusively consider results from probes whose clocks were synchronized within a maximum deviation of five minutes by the time our NTP queries were initiated.(Atlas probes are designed to synchronize their clocks approximately every three minutes [29]).
Subsequently, we consolidate all offset data points for each NTP server and compute their aggregate statistics.Our analysis reveals that the average offset for all NTP servers remains under 2.1 seconds (Table 7), which is considered a favorable result.This finding is visually represented in Figure 12, where the majority of offsets fall within the range of ± 2 seconds for all probed servers, signifying reliable performance.
Takeaway: we have shown that NTP servers located in diverse geographical regions and continents can provide dependable time services for clients located elsewhere.Hence, this experiment serves to underscore that the assumption of tethering clients exclusively to servers within their own countries is unnecessary and opens up opportunities for potential attacks.

DISCUSSION
The NTP Pool is a time service provider that relies on the contributions of volunteer NTP server operators worldwide.It has been an active project for over 20 years.As shown in §3, it is and has been the most popular time service on the Internet, measured at the root DNS.It is time to recognize the NTP Pool as one of the most crucial timekeepers on the Internet.
We have also scrutinized GeoDNS, the NTP Pool's DNS component that determines which NTP servers will be assigned to clients.Prior to our work, there was anecdotal evidence that GeoDNS mapped clients to their own countries.We have demonstrated that the NTP Pool does not map individual users, but all users of entire countries ( §4).Our findings reveal a very restrictive mapping: clients are bound to be served by the set of NTP servers in their country, even if there is only one server.We have shown the precise cases where countries fallback to their continent or global zones.
We also identified several issues with the current mapping algorithms.The most significant is that it produces a skewed and unfair server distribution among NTP clients worldwide.Users in wealthier countries, who can afford to donate NTP servers to the NTP Pool, are better off than users in less wealthy countries, in terms of NTP server diversity.In extreme cases, we have seen how the NTP Pool allows for the emergence of time monopolies, by mapping entire countries to one or a few NTP servers/ASes ( §5): it maps all clients in 27 countries, covering 767M inhabitants and 465M Internet users, to a single actor, which can have severe consequences in cases of attacks.
Our results reveal that the GeoDNS in-country mappings introduce unnecessary risks.Discussions with the NTP Pool operators [7] have been productive.They acknowledge the need for changes to enhance system security, which they plan to implement.Additionally, the use of DNS for load balancing across all volunteer servers must be considered when designing the new system.One potential solution could involve eliminating country zones in favor of larger continent zones.

ETHICS, PRIVACY, AND DISCLOSURE
Our paper has three ethical concerns: avoiding negative consequences of our measurements in both clients (Atlas VPs) and DNS/NTP servers, respecting the privacy of these VPs, and disclosing our findings to the NTP Pool operators.
Responsible experimentation: we design our experiments to minimize the impact on clients, measurement platform (RIPE Atlas), and measured DNS and Web servers.Whenever we use RIPE Atlas VPs, we use safe query rates (1 query per 5, 10, or 30 minutes, depending on the experiment).We also crawl web pages related to each NTP server on the NTP Pool website -fewer than 5 thousand pages.To minimize impact, we rate-limit our crawler to 1 webpage/second.Part of our experiments was done in an isolated network ( §5), so no traffic was sent to the NTP Pool servers.
Privacy: We found cases of RIPE VPs that seem to use overseas DNS servers (which may be due to trying to bypass government censorship or DNS hijack).DNS hijack in RIPE VPs has been known for years [53,83].While we cannot determine which is the reason, we do not disclose details about these cases to protect these VPs and their owners, who volunteer to host them.
Disclosure to NTP Pool operators: We shared multiple versions of this manuscript with the NTP Pool operators, who provided valuable feedback.We also have exchanged e-mails and discussion on the public NTP Pool forum [7] .While we did not disclose any new attack models (they have been previously presented [39,72]), we show how current GeoDNS mapping is restrictive and how measured its affected populations.We hope GeoDNS can be changed to introduce more diversity in the number of NTP servers each client sees.

RELATED WORK
NTP Pool measurement studies: our study is the most comprehensive evaluating the inner works and popularity of the NTP Pool.A previous study has also crawled the NTP Pool authoritative servers to enumerate them [78].They used a single VP in Germany to query the NTP Pool authoritative servers.We scrutinize the inner works of GeoDNS and by unveiling how the NTP Pool monitors, evicts, and cleans its zone ( §4), and show how clients all over the world see the NTP Pool ( §5).
NTP Pool vulnerabilities: Previous studies have shown how the NTP Pool can be exploited to hijack traffic from countries with empty zones [78] -they run a brief experiment on IPv6 -but they do not confirm the traffic monopoly.Our measurements from §4.3 confirms it is feasible and demonstrate traffic monopoly, and we provide open datasets (by RIPE Atlas).Another study has shown that an attacker can also control traffic by introducing multiple NTP servers into densely populated zones [72].The latter aims to create a race condition, thereby increasing the likelihood of clients being served by their malicious NTP servers to perform time-shift attacks.
Another work has identified vulnerabilities with the NTP Pool monitoring system [39].The authors present multiple attack methods against the monitoring servers -which include BGP hijack and delay attacks.Another attack model they cover assumes an attacker controls one of the pool monitoring servers.While not directly related to ours, there are several other studies that focused on NTP security.They either cover the NTP protocol vulnerabilities [43][44][45], or show how NTP clients can be vulnerable to malicious time servers [18], or how NTP servers can be used in distributed denial-of-service (DDoS) amplification attacks [16] (in which spoofed queries are sent with the source address of the target, which then receives unsolicited traffic), or study off-path attacks using DNS cache poisoning [34].While related to ours, they do not focus on the NTP Pool itself, as we do.
With regards to NTP traffic characterization, previous studies have characterized traffic at the NIST's NTP servers [80] or running many NTP servers that are part of the NTP Pool [78].We analyze Root DNS traffic to determine how popular time providers are and briefly cover 24 of traffic of a NTP server listed in the NTP Pool.
While NTP traffic is transmitted in clear (and thus prone to tampering), Network Time Security (NTS) [23] protocol provides client-server encryption and therefore eliminates the possibility of tampering between client and server.NTS, however, is currently not supported by the NTP Pool.

CONCLUSION
The NTP Pool has played a pivotal role in ensuring accurate timekeeping on the Internet.Operating as a community-driven initiative, akin to Wikipedia, it has proven to be the most widely used time service on the Internet, as demonstrated by our DITL datasets from the Root servers.We extend our gratitude to the volunteers who have dedicated their time and resources to support this endeavor over the past two decades.
In our investigation, we have delved into the intricate workings of the NTP Pool and highlighted an area of concern.While the client/NTP server mapping was implemented with good intentions (to avoid packet loss and traffic asymmetry), we have shown that these fears may be unfounded.Furthermore, these mappings can be improved to prevent attackers from exploiting current vulnerabilities to monopolize traffic, and to increase server diversity for clients worldwide.
Considering these findings, we raise awareness among NTP Pool operators regarding these shortcomings.We hope to encourage necessary improvements for the continued reliability and security of public and free time synchronization services on the Internet.

A LIST OF TIME PROVIDERS AND SERVERS
Table 9 shows the list of time providers and their respective time services domain names used in §3.We built this list based on a public repository on Github 1 .

D WEIGHTS VALIDATION
Next, we evaluate how each NTP server weight determines the distribution of NTP servers among clients when weight is considered.To do that, we carry out two experiments: a baseline experiment measured in the wild (using the NTP Pool DNS servers), which we compared against a controlled emulation in our setup, , which we use our own GeoDNS instance, as in §4.2.
In the baseline experiment, we query the authoritative server of one of its country subzones -Argentina's ar.We choose it because it has only eight active IPv4 NTP servers (on 2021-08-02 [62]), reducing the number of necessary queries to evaluate the weight's influence.By directly querying Argentina's 3.ar.pool.ntp.org,we bypass GeoDNS's geolocation steps, obtaining records only listed in the ar subzone.(We confirm this behavior experimentally by running a test locally).
As shown in Table 14 (experiment ArgV4), we send 107k queries from 9.2k Atlas VPs.Each valid response received only two A records, and in total, we see eight distinct A records associated with NTP servers under Argentina's zone, as also reported in [62].
Table 15 shows the results.We see that each server receives from 7.2% to 18.3% of all queriesso, in the case of Argentina's subzone, the popular NTP service may appear at least twice as often than the less popular server.We use these results as a baseline.
For the emulation experiment, we create a test zone using the A records from Table 15 and, as weights, we use the counts value.We configure GeoDNS with this zone file on an AWS EC2 Frankfurt Ubuntu VM and use ∼ 9k Atlas probes to query this zone, as shown in Table 3 (dataset ArgV4-Enum), use the same parameters (frequency, duration) in the ArgV4 experiment.
Similarly to EnumV4-Emul experiment, we reproduce the ArvgV4 experiment as ArgV4-Emul.We generate 110k responses from 3128 ASes, as shown in Table 15.We then compute the occurrence of each IP address from our demo zone in the Atlas responses and find that the query distribution per IP is very similar to the original experiment using the production servers of the NTP Pool.We can conclude that frequency counts can be used to infer weights in the NTP Pool zones.

E VENDOR ZONES AND IPV6 CLIENTS
The NTP Pool operators encourage vendors to ask for their own DNS subzones [66].However, we did not find any vendor zones while reverse engineering in NTP Pool zone files (they are not publicly disclosed, and server's report pages do not list them).Are the vendors zones kept apart from the geographical zones?If so, how GeoDNS handles them?We found out experimentally that they are a replica of the geographical zones.Their job is to allow the NTP Pool operators with a easy way to remove problematic vendors from service without affecting other users.
To determine that, we carry out experiments with RIPE Atlas, asking 32 probes located in Argentina to query for the A record of the Android vendor zone (2.android.pool.ntp.org).We analyzed the A records returned to these responses (dataset ArgV4-Android in [73]), and found only 7 distinct IP addresses,as shown in Table 16, all of them belonging also to the ar geographical zone.On the same day (2021-08-23), there were only 7 servers active in the ar zone [62] .
Therefore, we can conclude that the vendor zones seem to be a replica of the geographical zones -only that they give the ability to the NTP Pool operators to remove them in case of a vendor specific errors that can lead to DDoS attacks (CNAME records in DNS can be used to link both zones).As such, clients using vendor subzones are still bound by the geographical zones.
GeoDNS subzone sizes, in number of NTP servers.Country Continent Global (b) Client country and GeoDNS subzone mapping.

Fig. 7 .
Fig. 7. Number of NTP servers assigned to each client country

Fig. 9 .
Fig. 9. Ratio of million Internet users per NTP server.

Table 1 .
Evaluated Time Providers and their records TTL, and their TLD's own TTL

Table 4 .
Validation results per zone.

Table 5 .
Fig. 5. NTP servers per DNS response from VP 17580 and server score from NTP Pool for server NTP server 51.255.142.175:low scores lead to NTP Pool eviction and fallback to the continent zone.Atlas VP 17580 responses (2021-08-26)

Table 6 .
Countries served by a single time provider: Cloudflare and other AS (bold)

Table 7 .
[85] assumes symmetric paths).It is questionable where keep Evaluating NTP servers from clients located in clients only served by Cloudflare.Datasets:[74]clients within a country can reduce route assymetry.Recent work has shown that most Internet paths are asymmetrical[85], so it is not a NTP Pool only problem.Binding client to countries does not consider the large diversity in country sizes -a client in Belgium may be geographically and topologically closer to NTP servers located in neighboring Germany, while a client in Honolulu being served by a NTP server in Boston (both in the US but 8.2k km apart).

Table 9 .
List of Time Providers and their respective server names B NTP POOL SUBZONES FOUND IN THE DITL DATASETSIn this section, we show the NTP Pool subzones found in the DITL 2022 datasets.We only consider valid zones, i.e.,, every subzone we find we resolve to determine if it exists or not (by resolving them using DNS A queries), so we can disregard zones such as debiaan.pool.ntp.org, which show up in the dataset due to typos.Table10shows the breakdown of queries, resolvers, and ASes per subzone type: vendor ( such as ubuntu.pool.ntp.org),geographical (such as spain.pool.ntp.org), and general, which is the pool.ntp.orgzone.For both vendor and geographical zones, we convert [1--3].ZONE.pool.ntp.org to ZONE.pool.ntp.org.We see that most queries are for vendor zones, but most resolvers and ASes query for the general zone.

Table 10 .
NTP Pool subzones categories observed on the DITL 2022 datasets, and query counts.