research-article

Open access

PLB: congestion signals are simple and effective for network load balancing

Authors:

Mubashir Adnan Qureshi,

Masoud Moshref,

David Wetherall, and

Abdul KabbaniAuthors Info & Claims

SIGCOMM '22: Proceedings of the ACM SIGCOMM 2022 Conference

August 2022

Pages 207 - 218

https://doi.org/10.1145/3544216.3544226

Published: 22 August 2022 Publication History

Abstract

We present a new, host-based design for link load balancing and report the first experiences of link imbalance in datacenters. Our design, PLB (Protective Load Balancing), builds on transport protocols and ECMP/WCMP to reduce network hotspots. PLB randomly changes the paths of connections that experience congestion, preferring to repath after idle periods to minimize packet reordering. It repaths a connection by changing the IPv6 Flow Label on its packets, which switches include as part of ECMP/WCMP. Across hosts, this action drives down hotspots in the network, and lowers the latency of RPCs.

PLB is used fleetwide at Google for TCP and Pony Express traffic. We could deploy it when other designs were infeasible because PLB requires only small transport modifications and switch configuration changes, and is backwards-compatible. It has produced excellent gains: the median utilization imbalance of highly-loaded ToR uplinks in Google datacenters fell by 60%, packet drops correspondingly fell by 33%, and the tail latency (99p) of small RPCs fell by 20%. PLB is also a general solution that works for settings from datacenters to backbone networks, as well as different transports.

Supplementary Material

PDF File (p207-qureshi-supp.pdf)

Supplemental material.

Download
77.66 KB

References

[1]

2018. Tolly Report: Mellanox Spectrum Switch vs. Broadcom Tomahawk. https://community.mellanox.com/s/article/tolly-report-mellanox-spectrum-switch-vs-broadcom-tomahawk. (2018).

[2]

2018. Trident 3 Dynamic Load Balancing. https://www.broadcom.com/video/b468431136744543913129cd6a0caa30. (2018).

[3]

Mohammad Al-Fares, Sivasankar Radhakrishnan, Barath Raghavan, Nelson Huang, Amin Vahdat, et al. 2010. Hedera: dynamic flow scheduling for data center networks. In Nsdi, Vol. 10. San Jose, USA, 89--92.

[4]

Mohammad Alizadeh, Tom Edsall, et al. 2014. CONGA: Distributed congestion-aware load balancing for datacenters. In Proceedings of the 2014 ACM conference on SIGCOMM. 503--514.

Digital Library

[5]

Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. 2010. Data Center TCP (DCTCP). In SIGCOMM.

[6]

S Amante, B Carpenter, S Jiang, and J Rajahalme. 2011. RFC 6437: IPv6 flow label specification. IETF, November (2011).

[7]

Theophilus Benson, Aditya Akella, and David A. Maltz. 2010. Network Traffic Characteristics of Data Centers in the Wild. In IMC. 267--280.

Digital Library

[8]

Theophilus Benson, Ashok Anand, Aditya Akella, and Ming Zhang. 2011. MicroTE: Fine grained traffic engineering for data centers. In Proceedings of the seventh conference on emerging networking experiments and technologies. 1--12.

Digital Library

[9]

Olivier Bonaventure, Christoph Paasch, Gregory Detal, et al. 2017. Use cases and operational experience with multipath TCP. RFC 8041 (2017).

[10]

Neal Cardwell, Yuchung Cheng, et al. 2019. BBR v2: A Model-based Congestion Control. IETF 105. https://datatracker.ietf.org/meeting/105/materials/slides-105-iccrg-bbr-v2-a-model-based-congestion-control-00. (2019).

[11]

Neal Cardwell, Yuchung Cheng, et al. 2020. BBR Update:1: BBR.Swift; 2: Scalable Loss Handling. IETF 109. https://datatracker.ietf.org/meeting/109/materials/slides-109-iccrg-update-on-bbrv2-00. (Nov 2020).

[12]

Brian Carpenter and Shane Amante. 2011. Using the IPv6 flow label for equal cost multipath routing and link aggregation in tunnels. Technical Report. RFC 6438, November.

[13]

Yuchung Cheng, Neal Cardwell, Nandita Dukkipati, and Priyaranjan Jha. 2021. RFC 8985 The RACK-TLP Loss Detection Algorithm for TCP. (2021).

[14]

Andrew R Curtis, Wonho Kim, and Praveen Yalagandula. 2011. Mahout: Low-overhead datacenter traffic management using end-host-based elephant detection. In 2011 Proceedings IEEE INFOCOM. IEEE, 1629--1637.

[15]

Advait Abhay Dixit, Pawan Prakash, Y. Charlie Hu, and Ramana Rao Kompella. 2013. On the impact of packet spraying in data center networks. In INFOCOM.

[16]

William Feller. 1954. Diffusion processes in one dimension. Trans. Amer. Math. Soc. 77, 1 (1954), 1--31.

[17]

Andrew D Ferguson, Steve Gribble, Chi-Yao Hong, Charles Edwin Killian, et al. 2021. Orion: Google's Software-Defined Networking Control Plane. In NSDI. 83--98.

[18]

Sally Floyd, Jamshid Mahdavi, Matt Mathis, and Matt Podolsky. 2000. RFC2883: An extension to the selective acknowledgement (SACK) option for TCP. (2000).

[19]

Yilong Geng, Vimalkumar Jeyakumar, Abdul Kabbani, and Mohammad Alizadeh. 2016. Juggler: a practical reordering resilient network stack for datacenters. In Proceedings of the Eleventh European Conference on Computer Systems. 1--16.

Digital Library

[20]

Soudeh Ghorbani, Zibin Yang, P Brighten Godfrey, Yashar Ganjali, and Amin Firoozshahian. 2017. Drill: Micro load balancing for low-latency data center networks. In Proceedings of the ACM SIGCOMM. 225--238.

Digital Library

[21]

Google. 2022. TCP-PLB source code. (2022). https://github.com/google/plb.

[22]

Douglas Richard Hanks. [n. d.]. Juniper QFX10000 Series. Chapter 4. Performance and Scale. https://www.oreilly.com/library/view/juniper-qfx10000-series/9781491922248/ch04.html. ([n. d.]).

[23]

Chi-Yao Hong, Subhasree Mandal, et al. 2018. B4 and after: managing hierarchy, partitioning, and asymmetry for availability and scale in google's software-defined WAN. In Proceedings of the ACM SIGCOMM. 74--87.

Digital Library

[24]

Abdul Kabbani, Balajee Vamanan, Jahangir Hasan, and Fabien Duchene. 2014. Flowbender: Flow-level adaptive routing for improved latency and throughput in datacenter networks. In Proceedings of ACM CoNEXT. 149--160.

Digital Library

[25]

Abdulkadir Karaagac and Jeroen Hoebeke. [n. d.]. In-band Network Telemetry for 6TiSCH Networks. Internet-Draft draft-karaagac-6tisch-int-00. Internet Engineering Task Force. https://datatracker.ietf.org/doc/html/draft-karaagac-6tisch-int-00 Work in Progress.

[26]

Naga Katta, Aditi Ghag, Mukesh Hira, Isaac Keslassy, Aran Bergman, Changhoon Kim, and Jennifer Rexford. 2017. Clove: Congestion-aware load balancing at the virtual edge. In Proceedings of the 13th International Conference on emerging Networking EXperiments and Technologies. 323--335.

Digital Library

[27]

Naga Katta, Mukesh Hira, Changhoon Kim, Anirudh Sivaraman, and Jennifer Rexford. 2016. Hula: Scalable load balancing using programmable data planes. In Proceedings of the Symposium on SDN Research. 1--12.

Digital Library

[28]

Frank Kelly. 2003. Fairness and stability of end-to-end congestion control. European journal of control 9, 2--3 (2003), 159--176.

[29]

William Knight and D. M. Bloom. 1973. E2386. The American Mathematical Monthly 80, 10 (1973), 1141--1142. http://www.jstor.org/stable/2318556

[30]

Gautam Kumar, Nandita Dukkipati, et al. 2020. Swift: Delay is Simple and Effective for Congestion Control in the Datacenter. In Proceedings of the SIGCOMM.

Digital Library

[31]

Ming Li, Deepak Ganesan, and Prashant Shenoy. 2009. PRESTO: Feedback-driven data management in sensor networks. IEEE/ACM Transactions on Networking 17, 4 (2009), 1256--1269.

Digital Library

[32]

Steven H Low. 2003. A duality model of TCP and queue management algorithms. IEEE/ACM Transactions On Networking 11, 4 (2003), 525--536.

Digital Library

[33]

Michael Marty, Marc de Kruijf, et al. 2019. Snap: a Microkernel Approach to Host Networking. In SOSP.

[34]

Kihong Park, Gitae Kim, and Mark E Crovella. 1997. Effect of traffic self-similarity on network performance. In Performance and Control of Network Systems, Vol. 3231. International Society for Optics and Photonics, 296--310.

[35]

Jonathan Perry, Amy Ousterhout, Hari Balakrishnan, Devavrat Shah, and Hans Fugal. 2014. Fastpass: A centralized zero-queue datacenter network. (2014).

Digital Library

[36]

Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C Snoeren. 2015. Inside the social network's (datacenter) network. In Proceedings of the ACM SIGCOMM. 123--137.

Digital Library

[37]

Siddhartha Sen, David Shue, Sunghwan Ihm, and Michael J Freedman. 2013. Scalable, optimal flow routing in datacenters via local link balancing. In Proceedings of the ninth ACM conference on Emerging networking experiments and technologies. 151--162.

Digital Library

[38]

Arjun Singhvi, Aditya Akella, Dan Gibson, Thomas F Wenisch, et al. 2020. 1RMA: Re-Envisioning Remote Memory Access for Multi-Tenant Datacenters. In Proceedings of the ACM SIGCOMM. 708--721.

Digital Library

[39]

Shan Sinha, Srikanth Kandula, and Dina Katabi. 2004. Harnessing TCP's burstiness with flowlet switching. In Proc. 3rd ACM Workshop on Hot Topics in Networks (Hotnets-III).

[40]

D Thaler and C Hopps. 2000. RFC 2991 Multipath Issues in Unicast and Multicast Next-Hop Selection. (2000).

[41]

Erico Vanini, Rong Pan, Mohammad Alizadeh, Parvin Taheri, and Tom Edsall. 2017. Let it flow: Resilient asymmetric load balancing with flowlet switching. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17). 407--420.

[42]

Peng Wang, Hong Xu, Zhixiong Niu, Dongsu Han, and Yongqiang Xiong. 2016. Expeditus: Congestion-aware load balancing in clos data center networks. In Proceedings of the Seventh ACM Symposium on Cloud Computing. 442--455.

Digital Library

[43]

David Zats, Tathagata Das, Prashanth Mohan, Dhruba Borthakur, and Randy Katz. 2012. DeTail: Reducing the flow completion time tail in datacenter networks. In Proceedings of the ACM SIGCOMM. 139--150.

Digital Library

[44]

Hong Zhang, Junxue Zhang, Wei Bai, Kai Chen, and Mosharaf Chowdhury. 2017. Resilient datacenter load balancing in the wild. In Proceedings of the ACM SIGCOMM. 253--266.

Digital Library

[45]

Junlan Zhou, Malveeka Tewari, Min Zhu, Abdul Kabbani, Leon Poutievski, Arjun Singh, and Amin Vahdat. 2014. WCMP: Weighted cost multipathing for improved fairness in data centers. In Proceedings of the Ninth European Conference on Computer Systems. 1--14.

Digital Library

Cited By

Zadeh SMunir ABahnasy MKetabi SGanjali Y(2023)On Augmenting TCP/IP Stack via eBPFProceedings of the 1st Workshop on eBPF and Kernel Extensions10.1145/3609021.3609300(15-20)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3609021.3609300
Hinz JAddanki VGyörgyi CJepsen TSchmid S(2023)TCP's Third Eye: Leveraging eBPF for Telemetry-Powered Congestion ControlProceedings of the 1st Workshop on eBPF and Kernel Extensions10.1145/3609021.3609295(1-7)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3609021.3609295
Wetherall DKabbani AJacobson VWinget JCheng YMorrey III CMoravapalle UGill PKnight SVahdat ASchulzrinne HKohler EMaltz DMisra V(2023)Improving Network Availability with Protective ReRouteProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604867(684-695)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604867
Show More Cited By

Index Terms

PLB: congestion signals are simple and effective for network load balancing
1. Networks
  1. Network algorithms
    1. Data path algorithms
  2. Network components
    1. End nodes

Recommendations

CONGA: distributed congestion-aware load balancing for datacenters
SIGCOMM '14: Proceedings of the 2014 ACM conference on SIGCOMM

We present the design, implementation, and evaluation of CONGA, a network-based distributed congestion-aware load balancing mechanism for datacenters. CONGA exploits recent trends including the use of regular Clos topologies and overlays for network ...
Read More
CONGA: distributed congestion-aware load balancing for datacenters
SIGCOMM'14

We present the design, implementation, and evaluation of CONGA, a network-based distributed congestion-aware load balancing mechanism for datacenters. CONGA exploits recent trends including the use of regular Clos topologies and overlays for network ...
Read More
Resilient Datacenter Load Balancing in the Wild
SIGCOMM '17: Proceedings of the Conference of the ACM Special Interest Group on Data Communication

Production datacenters operate under various uncertainties such as traffic dynamics, topology asymmetry, and failures. Therefore, datacenter load balancing schemes must be resilient to these uncertainties; i.e., they should accurately sense path ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '22: Proceedings of the ACM SIGCOMM 2022 Conference

August 2022

858 pages

ISBN:9781450394208

DOI:10.1145/3544216

General Chairs:
Fernando Kuipers
Delft University of Technology
,
Ariel Orda
Technion Israel Institute of Technology

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 August 2022

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGCOMM '22

Sponsor:

SIGCOMM

SIGCOMM '22: ACM SIGCOMM 2022 Conference

August 22 - 26, 2022

Amsterdam, Netherlands

Acceptance Rates

Overall Acceptance Rate 554 of 3,547 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
5,866
Total Downloads

Downloads (Last 12 months)2,744
Downloads (Last 6 weeks)226

Other Metrics

View Author Metrics

Citations

Cited By

Zadeh SMunir ABahnasy MKetabi SGanjali Y(2023)On Augmenting TCP/IP Stack via eBPFProceedings of the 1st Workshop on eBPF and Kernel Extensions10.1145/3609021.3609300(15-20)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3609021.3609300
Hinz JAddanki VGyörgyi CJepsen TSchmid S(2023)TCP's Third Eye: Leveraging eBPF for Telemetry-Powered Congestion ControlProceedings of the 1st Workshop on eBPF and Kernel Extensions10.1145/3609021.3609295(1-7)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3609021.3609295
Wetherall DKabbani AJacobson VWinget JCheng YMorrey III CMoravapalle UGill PKnight SVahdat ASchulzrinne HKohler EMaltz DMisra V(2023)Improving Network Availability with Protective ReRouteProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604867(684-695)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604867
Joshi RSong CKhooi XBudhdev NMishra AChan MLeong BSchulzrinne HKohler EMaltz DMisra V(2023)Masking Corruption Packet Losses in Datacenter Networks with Link-local RetransmissionProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604853(288-304)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604853
Song CKhooi XJoshi RChoi ILi JChan MSchulzrinne HKohler EMaltz DMisra V(2023)Network Load Balancing with In-network Reordering Support for RDMAProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604849(816-831)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604849
Qureshi MYan JCheng YYeganeh SSeung YCardwell NDe Bruijn WJacobson VKaur JWetherall DVahdat ASchulzrinne HKohler EMaltz DMisra V(2023)Fathom: Understanding Datacenter Application Network PerformanceProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604815(394-405)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604815
Liu MBaumann ACrooks NSchwarzkopf M(2023)Fabric-Centric ComputingProceedings of the 19th Workshop on Hot Topics in Operating Systems10.1145/3593856.3595907(118-126)Online publication date: 22-Jun-2023
https://dl.acm.org/doi/10.1145/3593856.3595907
Huang HZhangB YWang RXiang QWang WQue XXu K(2023)FRAVaR: A Fast Failure Recovery Framework for Inter-DC Network2023 IEEE Wireless Communications and Networking Conference (WCNC)10.1109/WCNC55385.2023.10119088(1-6)Online publication date: Mar-2023
https://doi.org/10.1109/WCNC55385.2023.10119088

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents