research-article

Congestion Control for Large-Scale RDMA Deployments

Authors:

Daniel Firestone,

Chuanxiong Guo,

Marina Lipshteyn,

Yehonatan Liron,

Jitendra Padhye,

Shachar Raindel,

Mohamad Haj Yahia, and

Ming ZhangAuthors Info & Claims

ACM SIGCOMM Computer Communication Review, Volume 45, Issue 4

Pages 523 - 536

https://doi.org/10.1145/2829988.2787484

Published: 17 August 2015 Publication History

Abstract

Modern datacenter applications demand high throughput (40Gbps) and ultra-low latency (< 10 μs per hop) from the network, with low CPU overhead. Standard TCP/IP stacks cannot meet these requirements, but Remote Direct Memory Access (RDMA) can. On IP-routed datacenter networks, RDMA is deployed using RoCEv2 protocol, which relies on Priority-based Flow Control (PFC) to enable a drop-free network. However, PFC can lead to poor application performance due to problems like head-of-line blocking and unfairness. To alleviates these problems, we introduce DCQCN, an end-to-end congestion control scheme for RoCEv2. To optimize DCQCN performance, we build a fluid model, and provide guidelines for tuning switch buffer thresholds, and other protocol parameters. Using a 3-tier Clos network testbed, we show that DCQCN dramatically improves throughput and fairness of RoCEv2 RDMA traffic. DCQCN is implemented in Mellanox NICs, and is being deployed in Microsoft's datacenters.

Supplementary Material

WEBM File (p523-zhu.webm)

Download
159.20 MB

References

[1]

M. Alizadeh, B. Atikoglu, A. Kabbani, A. Lakshmikantha, R. Pan, B. Prabhakar, and M. Seaman. Data center transport mechanisms: Congestion control theory and IEEE standardization. In Allerton, 2008.

[2]

M. Alizadeh, A. Greenberg, D. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, and M. Sridharan. Data Center TCP (DCTCP). In SIGCOMM, 2010.

Digital Library

[3]

M. Alizadeh, A. Javanmard, and B. Prabhakar. Analysis of DCTCP: Stability, convergence and fairness. In SIGMETRICS, 2011.

Digital Library

[4]

M. Alizadeh, A. Kabbani, B. Atikoglu, and B. Prabhakar. Stability analysis of QCN: the averaging principle. In SIGMETRICS, 2011.

Digital Library

[5]

M. Alizadeh, A. Kabbani, T. Edsall, B. Prabhakar, A. Vahdat, and M. Yasuda. Less is more: Trading a little bandwidth for ultra-low latency in the data center. In NSDI, 2012.

Digital Library

[6]

M. Alizadeh, S. Yang, M. Sharif, S. Katti, N. McKeown, B. Prabhakar, and S. Shenker. pFabric: Minimal near-optimal datacenter transport. In SIGCOMM, 2013.

Digital Library

[7]

A. K. Choudhury and E. L. Hahne. Dynamic queue length thresholds for shared-memory packet switches. IEEE/ACM Transactions on Networking, 6(2), 1998.

Digital Library

[8]

Cisco. Priority flow control: Build reliable layer 2 infrastructure. http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-542809_ns783_Networking_Solutions_White_Paper.html.

[9]

P. Devkota and A. L. N. Reddy. Performance of quantized congestion notification in TCP incast scenarios of data centers. In MASCOTS, 2012.

Digital Library

[10]

A. Dragojevic, D. Narayanan, O. Hodson, and M. Castro. FaRM: Fast remote memory. In NSDI, 2014.

Digital Library

[11]

J. Duetto, I. Johnson, J. Flich, F. Naven, P. Garcia, and T. Nachiondo. A new scalable and cost-effective congestion management strategy for lossless multistage interconnection networks. In HPCA, 2005.

Digital Library

[12]

N. Dukkipati. Rate control protocol (RCP): Congestion control to make flows complete quickly. In PhD diss., Stanford University, 2007.

Digital Library

[13]

S. Floyd and V. Jacobson. Random early detection gateways for congestion avoidance. IEEE/ACM Transactions on Networking, 1:397--413, 1993.

Digital Library

[14]

E. G. Gran, M. Eimot, S.-A. Reinemo, T. Skeie, O. Lysne, L. P. Huse, and G. Shainer. First experiences with congestion control in infiniband hardware. In IPDPS, 2010.

[15]

A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta. VL2: A scalable and flexible data center network. In SIGCOMM, 2009.

Digital Library

[16]

C. Huang, H. Simitci, Y. Xu, A. Ogus, B. Calder, P. Gopalan, J. Li, and S. Yekhanin. Erasure coding in Windows Azure storage. In USENIX ATC, 2012.

Digital Library

[17]

IEEE. 802.11Qau. Congestion notification, 2010.

[18]

IEEE. 802.11Qbb. Priority based flow control, 2011.

[19]

Infiniband Trade Association. InfiniBand architecture volume 1, general specifications, release 1.2.1, 2008.

[20]

Infiniband Trade Association. Supplement to InfiniBand architecture specification volume 1 release 1.2.2 annex A16: RDMA over converged ethernet (RoCE), 2010.

[21]

Infiniband Trade Association. InfiniBand architecture volume 2, physical specifications, release 1.3, 2012.

[22]

Infiniband Trade Association. Supplement to InfiniBand architecture specification volume 1 release 1.2.2 annex A17: RoCEv2 (IP routable RoCE), 2014.

[23]

E. Jeong, S. Woo, A. Jamshed, H. Jeong, S. Ihm, D. Han, and K. Park. mTCP: a highly scalable user-level TCP stack for multicore systems. In NSDI, 2014.

Digital Library

[24]

S. Kamil, L. Oliker, A. Pinar, and J. Shalf. Communication requirements and interconnect optimization for high-end scientific applications. IEEE TPDS, 21:188--202, 2009.

Digital Library

[25]

S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken. The nature of datacenter traffic: Measurements and analysis. In IMC, 2009.

Digital Library

[26]

A. Kangarlou et al. vSnoop: Improving TCP throughput in virtualized environments via acknowledgement offload. In SC, 2010.

Digital Library

[27]

S. Larsen, P. Sarangam, and R. Huggahalli. Architectural breakdown of end-to-end latency in a TCP/IP network. In SBAC-PAD, 2007.

[28]

Luigi Rizzo. netmap: a novel framework for fast packet I/O. In USENIX ATC, 2012.

Digital Library

[29]

I. Marinos, R. N. Watson, and M. Handley. Network stack specialization for performance. In SIGCOMM, 2014.

Digital Library

[30]

C. Mitchell, Y. Geng, and J. Li. Using one-sided rdma reads to build a fast, cpu-efficient key-value store. In USENIX ATC, 2013.

Digital Library

[31]

R. Mitta, E. Blem, N. Dukkipati, T. Lam, A. Vahdat, Y. Wang, H. Wassel, D. Wetherall, D. Zats, and M. Ghobadi. TIMELY: RTT-based congestion control for the datacenter. In SIGCOMM, 2015.

Digital Library

[32]

G. Mora, P. J. Garcia, J. Flich, and J. Duato. RECN-IQ: A cost-effective input-queued switch architecture with congestion management. In ICPP, 2007.

Digital Library

[33]

J. Perry, A. Ousterhout, H. Balakrishnan, D. Shah, and H. Fugal. Fastpass: A centralized, zero-queue datacenter network. In SIGCOMM, 2014.

Digital Library

[34]

K. Ramakrishnan, S. Floyd, and D. Black. The addition of explicit congestion notification (ECN). RFC 3168.

Digital Library

[35]

M. Recio, B. Metzler, P. Culley, J. Hilland, and D. Garcia. A remote direct memory access protocol specification. RFC 5040.

[36]

A. Sivaraman, K. Winstein, P. Thaker, and H. Balakrishnan. An experimental study of the learnability of congestion control. In SIGCOMM, 2014.

Digital Library

[37]

B. Stephens, A. Cox, A. Singla, J. Carter, C. Dixon, and W. Felter. Practical DCB for improved data center networks. In INFOCOMM, 2014.

[38]

H. Subramoni, S. Potluri, K. Kandalla, B. Barth, J. Vienne, J. Keasler, K. Tomko, K. Schulz, A. Moody, and D. Panda. Design of a scalable InfiniBand topology service to enable network-topology-aware placement of processes. In SC, 2012.

Digital Library

[39]

V. Vasudevan, A. Phanishayee, H. Shah, E. Krevat, D. G. Andersen, G. R. Ganger, G. A. Gibson, and B. Mueller. Safe and effective fine-grained TCP retransmissions for datacenter communication. In SIGCOMM, 2009.

Digital Library

[40]

K. Winstein and H. Balakrishnan. TCP ex machina: computer-generated congestion control. In SIGCOMM, 2013.

Digital Library

[41]

X. Wu, D. Turner, G. Chen, D. Maltz, X. Yang, L. Yuan, and M. Zhang. Netpilot: Automating datacenter network failure mitigation. In SIGCOMM, 2012.

Digital Library

[42]

D. Zats, T. Das, P. Mohan, D. Borthakur, and R. Katz. Detail: Reducing the flow completion time tail in datacenter networks. In SIGCOMM, 2012.

Digital Library

[43]

Chelsio Terminator 5 ASIC. http://www.chelsio.com/nic/rdma-iwarp/.

[44]

ConnectX-4 single/dual-port adapter supporting 100gb/s. http://www.mellanox.com/.

[45]

Intel data direct I/O technology. http://www.intel.com/content/www/us/en/io/data-direct-i-o-technology-brief.html.

[46]

Iperf - the TCP/UDP bandwidth measurement tool. http://iperf.fr.

[47]

Offloading the segmentation of large TCP packets. http://msdn.microsoft.com/en-us/library/windows/hardware/ff568840(v=vs.85).aspx.

[48]

QueryPerformanceCounter function. http://msdn.microsoft.com/en-us/library/windows/desktop/ms644904(v=vs.85).aspx.

[49]

Receive Side Scaling (RSS). http://technet.microsoft.com/en-us/library/hh997036.aspx.

Cited By

Lei FWang YChen JYang S(2024)Symmetrical Data Recovery: FPGA-Based Multi-Dimensional Elastic Recovery Acceleration for Multiple Block Failures in Ceph SystemsSymmetry10.3390/sym1606067216:6(672)Online publication date: 30-May-2024
https://doi.org/10.3390/sym16060672
Li WRen MLiu YLi CQian HZhang Z(2024)Congestion Control Mechanism Based on Backpressure Feedback in Data Center NetworksFuture Internet10.3390/fi1604013116:4(131)Online publication date: 15-Apr-2024
https://doi.org/10.3390/fi16040131
He XLiang FFan WWang JHan LXiao FDou W(2024)Accurate and fast congestion feedback in MEC-enabled RDMA datacentersJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-024-00642-813:1Online publication date: 23-Mar-2024
https://dl.acm.org/doi/10.1186/s13677-024-00642-8
Show More Cited By

Index Terms

Congestion Control for Large-Scale RDMA Deployments
1. Networks
  1. Network protocols
    1. Transport protocols

Recommendations

Congestion Control for Large-Scale RDMA Deployments
SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

Modern datacenter applications demand high throughput (40Gbps) and ultra-low latency (< 10 μs per hop) from the network, with low CPU overhead. Standard TCP/IP stacks cannot meet these requirements, but Remote Direct Memory Access (RDMA) can. On IP-...
Read More
ECN or Delay: Lessons Learnt from Analysis of DCQCN and TIMELY
CoNEXT '16: Proceedings of the 12th International on Conference on emerging Networking EXperiments and Technologies

Data center networks, and especially drop-free RoCEv2 networks require efficient congestion control protocols. DCQCN (ECN-based) and TIMELY (delay-based) are two recent proposals for this purpose. In this paper, we analyze DCQCN and TIMELY using fluid ...
Read More
Host Congestion Control
ACM SIGCOMM '23: Proceedings of the ACM SIGCOMM 2023 Conference

The conventional wisdom in systems and networking communities is that congestion happens primarily within the network fabric. However, adoption of high-bandwidth access links and relatively stagnant technology trends for resources within hosts have led ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM SIGCOMM Computer Communication Review

ACM SIGCOMM Computer Communication Review Volume 45, Issue 4

SIGCOMM'15

October 2015

659 pages

ISSN:0146-4833

DOI:10.1145/2829988

Editors:
Konstantina Papagiannaki
Telefonica Research, Barcelona, Spain
,
Katerina Argyraki
EPFL, Switzerland
,
Hitesh Ballani
Microsoft Research Cambridge, UK
,
Fabián Bustamante
Northwestern University, USA
,
Joseph Camp
SMU, USA
,
Augustin Chaintreau
Columbia University, USA
,
Phillipa Gill
Stony Brook University, USA
,
Marco Mellia
Politecnico di Torino, Italy
,
Bhaskaran Raman
IIT Bombay, India
,
Joel Sommers
Colgate University, USA
,
Aline Carneiro Viana
INRIA, France

Issue’s Table of Contents

SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication
August 2015
684 pages
ISBN:9781450335423
DOI:10.1145/2785956
General Chairs:
Steve Uhlig
Queen Mary University of London, UK
,
Olaf Maennel
Tallinn U. of Technology in Estonia, Estonia
,
Program Chairs:
Brad Karp
University College London, UK
,
Jitendra Padhye
Microsoft, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2015

Published in SIGCOMM-CCR Volume 45, Issue 4

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

416
Total Citations
View Citations
4,864
Total Downloads

Downloads (Last 12 months)1,317
Downloads (Last 6 weeks)178

Other Metrics

View Author Metrics

Citations

Cited By

Lei FWang YChen JYang S(2024)Symmetrical Data Recovery: FPGA-Based Multi-Dimensional Elastic Recovery Acceleration for Multiple Block Failures in Ceph SystemsSymmetry10.3390/sym1606067216:6(672)Online publication date: 30-May-2024
https://doi.org/10.3390/sym16060672
Li WRen MLiu YLi CQian HZhang Z(2024)Congestion Control Mechanism Based on Backpressure Feedback in Data Center NetworksFuture Internet10.3390/fi1604013116:4(131)Online publication date: 15-Apr-2024
https://doi.org/10.3390/fi16040131
He XLiang FFan WWang JHan LXiao FDou W(2024)Accurate and fast congestion feedback in MEC-enabled RDMA datacentersJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-024-00642-813:1Online publication date: 23-Mar-2024
https://dl.acm.org/doi/10.1186/s13677-024-00642-8
Gupta JKant KPal ABiswas J(2024)Configuring and Coordinating End-to-end QoS for Emerging Storage InfrastructureACM Transactions on Modeling and Performance Evaluation of Computing Systems10.1145/36316069:1(1-32)Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1145/3631606
Bai SZheng HTian CWang XLiu CJin XXiao FXiang QDou WChen G(2024)Unison: A Parallel-Efficient and User-Transparent Network Simulation KernelProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629574(115-131)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3627703.3629574
Yang XZhang YChen HLi FWang BFang JSun CWang YBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)PolarDB-MP: A Multi-Primary Cloud-Native Database via Disaggregated Shared MemoryCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3653377(295-308)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3653377
Luo CGu HYu XZhu LZhou ZZhang HHou W(2024)Alarm: An Adaptive Routing Algorithm Based on One-Way Delay for InfinibandIEEE Transactions on Network Science and Engineering10.1109/TNSE.2024.338229511:4(3653-3666)Online publication date: Jul-2024
https://doi.org/10.1109/TNSE.2024.3382295
Luo CGu HZhu LZhang H(2024)FlowStar: Fast Convergence Per-Flow State Accurate Congestion Control for InfiniBandIEEE/ACM Transactions on Networking10.1109/TNET.2024.336365832:3(2662-2674)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3363658
Zhang JWang YZhong XYu MPan HZhang YGuan ZChe BWan ZPan THuang T(2024)PACC: A Proactive CNP Generation Scheme for Datacenter NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2024.336177132:3(2586-2599)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3361771
Qiao PZhang ZLi YYuan YWang SWang GYu J(2024)AStore: Uniformed Adaptive Learned Index and Cache for RDMA-enabled Key-Value StoreIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.3355100(1-18)Online publication date: 2024
https://doi.org/10.1109/TKDE.2024.3355100
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents