How I Learned to Stop Worrying About CCA Contention

This paper asks whether inter-flow contention between congestion control algorithms (CCAs) is a dominant factor in determining a flow's bandwidth allocation in today's Internet. We hypothesize that CCA contention typically does not determine a flow's bandwidth allocation, present an initial analysis in support of this hypothesis, propose a measurement technique and study to settle this question, and discuss the implications should the hypothesis prove true.


Introduction
Contention between Internet flows has long been considered a core factor in determining bandwidth allocations.As a result, the community has invested substantial effort into analyzing the long-term bandwidth allocation between competing flows.For example, TFRC [1] was designed to guarantee that UDP flows would share bandwidth evenly with TCP NewReno during congestion avoidance, and BBR has been shown [2] to take more than its long-term fair share of bandwidth when competing against NewReno and Cubic.Further, the research community has proposed prescribing how CCAs should interact with each other, from Floyd and Fall's TCP Friendliness [3], to Jain's fairness index for throughput [4], to Ware et al. 's proposal to center "harm" over a set of metrics.
In this position paper, we hypothesize that there do not remain common scenarios in the modern Internet in which CCA contention is the dominant factor in determining flows' bandwidth allocations.While additional measurement data is required to conclusively resolve this hypothesis, we argue that it is plausible.In other words, if our hypothesis holds, then for most flows on the Internet, contention between CCAs doesn't matter.Instead, we argue that a second set of factors, including in-network bandwidth management techniques (e.g., fair queueing [5], traffic engineering [6][7][8], and capacity planning [9]), along with application traffic limitations, primarily determine bandwidth allocations on the Internet today.
Why is resolving this hypothesis important?First, if our hypothesis is true, this has implications for CCA design: designers of new CCAs should no longer concern themselves with contention properties, and analysis of new and old CCAs should de-emphasize this property.Second, if CCA dynamics do not determine bandwidth allocations on the Internet and outcomes are instead a function of in-network and host rate shaping, how do these mechanisms work in the wild?Are they principled, equitable, and efficient?
In our analysis, it is important to clarify the difference between the terms "contention" and "congestion." Congestion, and even congestion collapse, can occur without CCAs interacting with each other.For example, consider a peering link overloaded with web traffic (consisting of short flows).This link will experience queueing, and the traffic will experience signs of congestion (packet loss and high delays), but the individual flows' CCAs will not interact.Instead, situations such as this one are usually managed by traffic engineering (TE) systems [6][7][8].Contention, on the other hand, occurs when two long-running flows share the same bottleneck link, and their allocated bandwidth is some function of their CCA dynamics.We discuss the precise prerequisites for CCA contention in §2.
In the rest of this paper, we present: (1) A discussion of modern Internet traffic dynamics in which we argue that our hypothesis is plausible ( §2).(2) Initial measurement results using data from a traditional measurement source (M-Lab [10]) and a proposal for a new active measurement technique that seeks to directly measure CCA contention ( §3).(3) A discussion of the implications of our hypothesis should it prove true, both for CCA design and analysis as well as the Internet architecture ( §5).
2 Where is the Contention?
Why do we suspect that flows no longer contend for bandwidth on the Internet?Recall that for contention to occur, at least two flows must: (i) share a path segment, (ii) experience a bottleneck in that path segment, and (iii) use the same queue at the bottleneck link.In this section, we argue that the confluence of these three conditions is rare in the modern Internet for two reasons, summarized in Figure 1:

Short Flows
Application Limited (e.g., video streams) Operator Throttling + Isolation Remaining Cases Economics ( §2.1) Users pay for access bandwidth, and providers provision and configure their networks to provide users with acceptably good network capacity.Furthermore, providers limit how much data a user can send, ensuring that a user's sending rate does not exceed the bandwidth for which they have paid.Note that while operators' scheduling mechanisms, such as fair queueing, cannot change the total amount of traffic in a queue, they can enforce a bandwidth allocation (i.e., max-min fairness) as well as isolation between flows.Thus, a universal deployment of fair queueing (for example) would entirely eliminate the role of CCA dynamics in determining bandwidth allocations.
Access Bottlenecks and App Limitations ( §2.2) While the economic factors described above limit contention between flows from different users, they do not eliminate contention between flows from a single user (since most isolation mechanisms operate on a per-user, not per-flow, basis).However, increases in the amount of available bandwidth and application limitations on offered load limit even contention between a single user's flows.This trend is likely to continue with Moore's law of compute scaling having ended, but Edholm's law of bandwidth scaling continuing to the present day.Further, application bandwidth demands have not grown as much as available bandwidth, so increasingly, an application's bottleneck is not in the network but rather its own offered load.These factors do not eliminate contention in all cases, and inter-flow CCA contention can still, in some cases, determine bandwidth allocations.We discuss these cases and how they might easily evolve to eliminate CCA contention in §2.3.

Economics
The Internet began as a shared communication medium between academic institutions (ARPANet, NSFNet, etc).As a result, Internet users developed techniques for dealing with traffic contention as a set of shared protocols that governed how good Internet citizens would behave when accessing the common infrastructure.After a series of "congestion collapses" in 1986 [11], Jacobson and Karels developed the first congestion control algorithm [12] based on Chiu and Jain's AIMD technique [13].As the Internet grew to encompass commercial traffic, over time, it evolved from a governmentfunded research network to today's commercial Internet.
This transition had a number of effects, including exponential growth in usage, but also operator measures to isolate their customers' traffic.ISPs today have a financial incentive to provision sufficient bandwidth for their customers' usage lest those customers switch to competing ISPs1 .Further, ISPs advertise services based on a promised service quality (e.g., 4K video streaming), and customers are unlikely to be impressed with protestations that this service quality was not met not due to access link provisioning but rather contention with other customers.As a result, many ISPs aggressively upgrade their inter-domain link capacities to maintain link utilization below 60-70% [9].As a recent example, in response to increased traffic volume during the first year of the COVID-19 pandemic, IXP members in Europe and the US increased their peering link capacities by up to 20% [14].
While capacity planning limits the amount of congestion an operator's traffic might encounter in aggregate, operators also use a bevy of traffic management techniques to ensure that they limit users to the level of bandwidth the users paid for, isolate users' traffic, and implement other policies.ISPs commonly offer multiple service levels-Paul et al. [15] report that US ISPs provide between 3 and 11 unique plans with monthly prices ranging from $20 to $120-so throttling user traffic incentivizes users to upgrade.For example, Flach et al. found [16] that traffic policing (in which the ISP's router drops a user's traffic above a configured rate) occurs on 7% of measured paths.They note that directly detecting the presence of shaping (in which the router queues the user's excess traffic rather than dropping it) is challenging.Similarly, Li et al. [17] found that traffic differentiation (i.e., throttling) is also common and identified 77 ISP-App throttling pairs.The prevalence of differentiation implies that bandwidth shaping (a super-set of differentiation) must be at least as prevalent.
At the largest scale, hyperscalers deploy private WANs over leased-lines [18,19] to carry their traffic.Since only one organization controls such a WAN, it can deploy hostbased bandwidth allocation [20] or priority queueing [21] to eliminate inter-flow contention.For example, Google uses BwE [22] to allocate bandwidth in its private WAN.BwE integrates with applications that report their bandwidth demand to centrally determine bandwidth allocations across the entire network.This isolates applications from each other and eliminates inter-flow contention across applications.
Commercial entities appear to have responded to this increasing level of isolation by developing proprietary, more aggressive, and application-specific CCAs.This is a significant departure from the initial introduction of congestion control mechanisms as protocols implemented and deployed with a near-universal agreement to today's custom algorithms.For example, Akamai's FastTCP remains proprietary, and Google developed its BBR and BBRv2 without disclosing details about those CCAs.With user-space CCA implementations gaining popularity [23,24], we can expect this trend to continue.

Access Bottlenecks and App Limitations
Internet measurement studies have observed three properties from which we infer a minimal role for CCA dynamics in determining bandwidth allocations: (a) most paths are short [25], (b) most flows are short [26], and (c) most of the bytes are from video streaming traffic which has bounded throughput demand [14].
First, most paths on the Internet are short, and thus, in most cases, access links-i.e., links within a home network or a device's cellular link-are the only candidates for inter-flow contention.Recent surveys [25,[27][28][29] have found that most ISPs host CDN nodes (from which they can respond to requests) and peer with cloud providers (e.g., Google directly connects to networks representing more than 60% of end-user prefixes) through carefully provisioned and managed links [6][7][8].Furthermore, cloud providers host a significant fraction of the services accessed by most users.Consequently, access links are the only potential location for inter-flow contention for a significant portion of the traffic.On access links, inter-flow contention can affect bandwidth allocation only if a user's applications simultaneously offer enough load to exceed the access link's capacity.Otherwise, each application would simply receive a bandwidth allocation corresponding to its offered load.
Second, most application flows are short [14,26,30].These flows fit within the initial congestion window or, in the case of a persistent connection, within the available congestion window.Since these flows are not limited by any flow-level transmission mechanism (except, perhaps, a rate limit), perflow CCA dynamics cannot affect their bandwidth allocations.Further, only in-network mechanisms can affect these flows; intermediate routers might drop their component packets, or a cellular base station might delay their transmission.
Finally, for inter-flow contention to occur on access links, there must be enough application traffic to exceed the link's provisioned capacity.However, most flows are either applimited or (as discussed above) too short to contend for bandwidth.For example, most transmitted bytes are from video streaming traffic, and even the most aggressive form of video streaming-real-time video game streaming-consumes only 20-30 Mbit/s at the highest bitrates [31].This remains less than the available user bandwidth at broadband speeds in many regions of the world [32].Even if the video stream saturated its access link, rather than inter-flow dynamics determining bandwidth allocations, adaptive bitrate (ABR) algorithms would reduce video streams' throughput demand.
Past measurement studies corroborate these observations.For example, Araújo et al. find that "flow rates are not typically dictated by TCP congestion control alone;" in their measurement dataset, less than 40% of traffic was neither application-, host-, nor receiver-limited [33].Further, Yang et al. found [32] that deployed home Wi-Fi routers commonly reflect the bandwidth local ISPs offer; in other words, the Wi-Fi link itself is as much of a bottleneck as the home access link.This further reduces the likelihood that two flows will contend for bandwidth at the access link.
Of course, there remain plausible scenarios where flows are not application-or receiver-limited and are long enough to contend for bandwidth with other flows.We discuss these below in §2.3.

What about. . . ?
We do not claim that inter-flow bandwidth contention does not occur at all; it can occur in some cases, e.g., persistently backlogged flows (software updates, etc) on access links.Rather, we merely claim that it is not the primary determinant for flows' bandwidth allocations.For the rest of this section, we discuss scenarios in which contention might still occur.
Couldn't contention still happen at access links?Contention can still happen at access links in some cases; we can create these conditions by starting two persistently backlogged connections from behind an access link.However, we note that the user (or their operating system) could use an endhost traffic shaper (e.g., Linux qdisc) to implement isolation.Software implementations are sufficient for most access links, and faster access links are unlikely to be congested in the first place.At the access link itself, recent work has proposed to bring bandwidth shaping to the edge even in cases where the true bottleneck is elsewhere [34,35].Overall, fair queueing and isolation is cheap and easy to implement, and these solve the per-flow contention problem more effectively than any flow-level CCA mechanism can.
What about the developing world?Chen et al. showed that on certain links where the bandwidth-delay product (BDP) is less than one packet, congestion control mechanisms can unfairly allocate bandwidth over short (~20 seconds) timescales [36].This is primarily due to timeout mechanisms that starve an arbitrary set of flows.Users in rural areas and the developing world are the most likely to experience such conditions [37].
We note two factors that minimize the role of flow contention for bandwidth in these scenarios.First, ISPs in both rural regions and in the developing world follow the same economics as those in other parts of the Internet and use similar traffic management techniques to isolate and throttle user traffic.Further, Internet growth in the developing world is dominated by wireless links, whether cellular (most commonly) [37,38], or using commercial aircraft [39] or satellites [40,41].Of course, cellular links already provide flow isolation.Given satellite networks also experience bandwidth and RTT fluctuations [42], they similarly adopt flow isolation and bandwidth management techniques [43], as is already common in other wireless ISP deployments [44,45].
While these factors provide an initial indication that flow contention may not happen in low-bandwidth or sub-packet scenarios, they do not provide a conclusive answer, and studying such scenarios remains an open question.
What about hypergiants?One ostensible source of contention is large content providers overwhelming peering links with large traffic aggregates.One popular example is the 2014 peering dispute between Netflix and Comcast [46].In this case, the dispute's source was not flow-level contention but rather the need for extra link capacity.Since Netflix is a video streaming service and video traffic is generally application-limited, individual flows' CCA dynamics could not have played a significant part in determining bandwidth allocations; those CCAs would control only the allocations at the unloaded edge bottleneck links.Rather, the issue was the aggregate demand over the peering link, which no individual flow's CCA controls.In this example, Netflix and Comcast resolved the peering dispute with a capacity upgrade corresponding to a commercial agreement.
What about datacenters?Some proposals for datacenterspecific flow control mechanisms use CCA mechanisms to allocate bandwidth [47][48][49][50][51], but other more recent works propose active in-network mechanisms to enforce isolation [52,53].Overall, since a single entity-a cloud provider-manages a datacenter, it can choose the bandwidth allocation mechanism that works best for its needs.

Measuring CCA Contention
How can we determine whether this paper's hypothesis is true?We discuss past approaches to measuring CCA dynamics and present an initial analysis using M-Lab [10] data, but we observe that passive measurement approaches cannot conclusively determine the presence (or absence) of CCA contention.Instead, in §3.2, we propose an active measurement approach based on a recently proposed CCA, Nimbus [54].

Passive Measurement
Existing work in measuring CCA behavior takes one of two common approaches: controlled testbed experiments or speedtest-style user-driven bandwidth testing (e.g., from a CDN or dedicated speedtest service).Unfortunately, neither approach is sufficient to conclusively determine CCA contention's impact on flows' bandwidth allocations.

Testbed Experiments
The goal of testbed congestion control analysis work is to gain an understanding of the behavior  of the algorithm by observing its reactions to different inputs.These works [55,56] use controlled environments where researchers can carefully manipulate network conditions to get a comprehensive view of the CCA's actions.This approach is useful for understanding how CCAs interact but cannot determine whether they interact.
Crowdsourced Bandwidth Testing Users who wish to test their connection quality often make use of speedtests.Clients initiate speedtest requests (or requests for content) to specific servers or CDN nodes, respectively.Then, the speedtest operator can measure connection telemetry and store it for future analysis [10,32,57].How much can this data tell us?To find out, we analyzed data from the M-Lab project, which publishes data from user-initiated network data test (NDT) [58,59] measurements.These measurements contain TCPInfo statistics from each data transfer, including the source and destination IP addresses, achieved throughput and RTT, the amount of time the connection was application-limited, etc.Since the M-Lab data contains snapshots of this data over each flow's lifetime, we can attempt to identify cases of CCA contention by searching for cases where a flow's achieved throughput level changed during its lifetime.This might indicate that it was contending for bandwidth with another flow, and the flow's CCA adjusted its sending rate in response.
To perform this analysis, we attempt to remove flows from the dataset that we know were unlikely to have experienced contention: application-or receiver-limited flows and flows we infer to use cellular links.We queried the M-Lab NDT dataset for one month (June 2023) of flow data (9,984 flows).We categorized flows as application-limited if the AppLimited field was greater than zero, and similarly we categorized a flow as receiver-limited if the RWndLimited field was greater than zero.Since the data does not identify whether a flow traversed a cellular link, we use a heuristic to identify these cases: if the maximum RTT and minimum RTT the flow experienced differ by at least 100ms, we classify it as cellular.After categorizing flows, we used the ruptures [60] library's binary segmentation breakpoint detection algorithm to identify periods of time when each flow experienced a change in its achieved throughput.Since the breakpoint algorithm requires a prior on the ideal number of breakpoints (and we have no such prior), we employed it in rounds.In each round, we use binary segmentation to find the next breakpoint, and we stop once the average standard deviation of each segment is less than one-third of the standard deviation across all segments.We show an example in Figure 2b, where we mark the breakpoints with a vertical dotted red line.The breakpoints track large changes in the flow's sending rate.Of the 9,894 flows, we remove 5,761 for having too few data points (≤ 12) for breakpoint analysis.We show results for the remaining flows in Figure 2a.First, corroborating our analysis in §2.2, most flows are application-or receiver-limited (4,008) or traverse cellular links (90); only 35 flows (fewer than 1%) of flows remain afterwards.Of these, there is a clear skew to a small number of breakpoints; i.e., flows generally experience a static bandwidth allocation throughout their lifetime.This provides some initial evidence that most flows do not experience CCA contention.However, this data cannot conclusively determine the absence of CCA contention (indeed, many flows experienced at least one breakpoint) even if it covered all Internet paths.This is because speedtest data uses passive telemetry using fixed CCAs (Cubic and BBR), and thus cannot show what would have happened had the CCA competed for bandwidth more aggressively.

Actively Measuring Elasticity
Instead, we propose an active measurement technique.If we run a speedtest-style experiment but use a CCA which actively determines whether cross traffic on the links it traverses is actively competing for bandwidth, we could use the CCA's report of the cross traffic's aggressiveness as an indicator of bandwidth contention.If the CCA reported no contention on most paths, we could conclude that CCA contention is rare.To this end, we observe that recent CCAs propose mode-switching techniques to determine the nature of cross traffic on their path and enable delay-based congestion control techniques in the absence of such cross traffic.While one such work, Copa [61], simply checks whether a path's cross-traffic adheres to Copa's delay oscillation dynamics, another, Nimbus [54] actively probes for cross traffic that responds to short-term changes in available bandwidth (the authors refer to such traffic as elastic cross traffic).Nimbus estimates a path's elasticity, i.e., the degree to which cross traffic on a path responds to fluctuations in available bandwidth, and uses a delay-based CCA mode when elasticity is low and a loss-based CCA otherwise.
We propose to use this CCA as a measurement tool to determine whether CCA contention occurs.If, for a probe flow, we use Nimbus but disable mode-switching, maintain the bandwidth oscillations, and measure the reported elasticity of Internet paths, we can determine whether cross traffic on those paths contended for bandwidth with that probe flow.We demonstrate a proof-of-concept of this technique in Figure 3.We run five types of traffic for 45 seconds each on a 48Mbps, 100ms emulated Mahimahi [62] link: along with a probe flow using Nimbus, we run (in sequence) two persistently backlogged flows that contend for bandwidth using CCAs Reno and BBR, as well as a video stream, a set of short flows with poisson arrivals, and constant bitrate (CBR) UDP traffic.We can clearly observe higher values for the elasticity metric for the flows that contend for bandwidth.We refer readers to Nimbus [54] for a full description and evaluation of the elasticity measurement algorithm.

Related Work
In addition to measurement and analysis work discussed previously, we call attention to a few additional related works.
Measuring Congestion Dhamdhere et al. [63] and related work [64] use a measurement technique called "time-series latency probes (TSLP)."With TSLP, the observation point sends TTL-limited latency probes to routers just before and after a given link and measures that link's latency differential.By measuring this value longitudinally, researchers using TSLP can identify time periods where individual links experienced inflated queueing delays; the technique infers congestion from the presence of such inflated queueing delays.While this approach is effective at identifying underprovisioned links in a lightweight manner, it cannot discriminate between cases where individual flows contend for bandwidth and cases where aggregates consisting of shorter and application-limited flows overwhelm a given link.Even so, we note Dhamdhere et al.'s finding that ". . .we did not find evidence of widespread endemic congestion on interdomain links between access ISPs and directly connected transit and content providers." Evolving Congestion Control Recent work has proposed extending congestion control mechanisms to implement sharing information or state amongst the flows of a host [65], rack [66], site [34,35], or organization [67].These proposals argue that sharing information is helpful for using the network efficiently as well as prioritizing certain traffic.Effectively, they propose supplanting CCA contention as a mechanism for bandwidth allocation and replacing it with a more centralized mechanism, whether fair queueing or prioritization.
Critiquing Throughput-Centricity In a recent HotNets paper, Ware et al. argued against "throughput centricity" as a means of analyzing CCA contention, since this ignores other critical application metrics such as latency [68].We discuss how CCAs might evolve to fairly contend on metrics other than throughput in §5.2, but we note that our goal in this paper is to analyze whether CCA contention occurs in the first place.

Implications
While further study is needed to test this paper's core hypothesis that contention between CCAs is no longer the dominant factor in determining flows' bandwidth allocations in today's Internet, we have presented initial evidence in its support.Therefore, we conclude by discussing how this hypothesis being true should affect future networking research and protocol design.

Future of Congestion Control
In recent years, development in congestion control has shifted towards providing three goals simultaneously: high throughput, low delay, and fairness [54,61,69].In fact, recent analysis has shown that delay-minimizing methods cannot always avoid starvation, an extreme form of unfairness [70].Another line of congestion control proposals, focusing on congestion control in cellular networks, has sought to implement CCAs that can cope with high variability in link capacities [71,72] while providing low delays.Since these proposals target cellular networks, which implement per-user isolation, they are not concerned with fairness.
If this paper's hypothesis proves to be true and most flows today are isolated from each other, CCA designers should deprioritize concerns about starvation and fairness.Instead, future CCAs should resemble the second group of CCAs, which focus on coping with bandwidth variability while navigating the trade-off between self-inflicted delay and link underutilization.Indeed, as discussed in §2.3, links with variable bandwidth are likely to become more common as the growth of mobile networks continues; e.g., even future fiber networks could embrace bandwidth variability [73].This is not to say that CCAs would become simpler or irrelevant.For example, prior analytical work has shown [74] that even without considering fairness, a given CCA still cannot simultaneously satisfy efficiency and loss-minimization goals.Further, Schapira and Winstein argued [75] that the fundamental tenets of congestion control remain unresolved, e.g., whether CCAs should attempt to model the network, use hill-climbing techniques, etc.

Contention on Alternate Metrics
Even if CCAs no longer determine flows' bandwidth allocations in the Internet today, they might contend on other metrics such as latency and jitter.For example, bursty traffic can vary the instantaneous bandwidth and delay other flows on the same link observe, even if the link uses fair queueing to isolate flows from each other.These variations in delay, i.e., jitter, can degrade QoE for applications such as live video streaming that perform best with consistently low delay.The precise mechanism the operator uses to perform bandwidth shaping affects the way flows contend for low jitter.For example, one popular method of bandwidth shaping is the token-bucket filter, in which a flow accrues "tokens" to send bytes at a fixed rate but can consume these tokens arbitrarily quickly once they are granted; the resulting bursty transmission can cause jitter.We leave for future work the question of designing CCAs that are friendly on jitter and other non-throughput metrics.

How Should We Model the Internet?
Internet users have long operated under a model of the Internet that specifies best-effort packet delivery combined with statistical multiplexing mediated by fair interaction dynamics.Building effective CCAs benefits from such a useful network model [75].Indeed, a 30-year-strong line of research work has sought to understand and analyze these interactions, starting with Chiu and Jain's AIMD and ongoing with theoretical analyses and controlled experiments [2,70,74].If this paper's hypothesis holds, opaque traffic shaping mechanisms governed by operator policies and economic arrangements, rather than CCA fairness dynamics, govern bandwidth allocations today.As an analogy, the Gao-Rexford model [76] of interdomain routing practices has proved useful despite its practical inaccuracy for parts of the Internet; however, if it proved not representative of most of the Internet, it would have to evolve to remain useful.
Thus, how should our model of congestion control evolve to incorporate the observation that CCA dynamics do not affect bandwidth allocations?Clearly, it should incorporate some notion of flow isolation and bandwidth shaping.Second, the unit of bandwidth contention would no longer be an individual flow but rather an economic arrangement that determines a network's bandwidth-shaping policy.A recent HotNets paper proposed one potential model, "Recursive Congestion Shares" [77], rooted in the Internet's existing economic arrangements.Unfortunately, we cannot know how accurate this model or any other model is without data from network operators.We leave the development and validation of such models to future work.

Figure 1 :
Figure 1: Operator policies driven by the economics of the commercial Internet combined with traffic dynamics combine to reduce or eliminate inter-flow contention.
(a) Breakpoint Distribution.(b) Flow Rate with Breakpoint Analysis

Figure 3 :
Figure 3: Active measurement allows directly measuring elasticity, a measure of the level of contention a flow experienced.