Sizing router buffers

Abstract

All Internet routers contain buffers to hold packets during times of congestion. Today, the size of the buffers is determined by the dynamics of TCP's congestion control algorithm. In particular, the goal is to make sure that when a link is congested, it is busy 100% of the time; which is equivalent to making sure its buffer never goes empty. A widely used rule-of-thumb states that each link needs a buffer of size B = overlineRTT x C, where overlineRTT is the average round-trip time of a flow passing across the link, and C is the data rate of the link. For example, a 10Gb/s router linecard needs approximately 250ms x 10Gb/s = 2.5Gbits of buffers; and the amount of buffering grows linearly with the line-rate. Such large buffers are challenging for router manufacturers, who must use large, slow, off-chip DRAMs. And queueing delays can be long, have high variance, and may destabilize the congestion control algorithms. In this paper we argue that the rule-of-thumb (B = (overlineRTT x C) is now outdated and incorrect for backbone routers. This is because of the large number of flows (TCP connections) multiplexed together on a single backbone link. Using theory, simulation and experiments on a network of real routers, we show that a link with n flows requires no more than B = (overlineRTT x C) √n, for long-lived or short-lived TCP flows. The consequences on router design are enormous: A 2.5Gb/s link carrying 10,000 flows could reduce its buffers by 99% with negligible difference in throughput; and a 10Gb/s link carrying 50,000 flows requires only 10Mbits of buffering, which can easily be implemented using fast, on-chip SRAM.

References

1. C. Villamizar and C. Song. High performance tcp in ansnet. ACM Computer Communications Review, 24(5):45--60, 1994 1994.]]
2. Cisco line cards. http://www.cisco.com/en/US/products/hw/modules/ps2710/products_data_sheets_list.html.]]
3. R. Bush and D. Meyer. RFC 3439: Some internet architectural guidelines and philosophy, December 2003.]]
4. C. J. Fraleigh. Provisioning Internet Backbone Networks to Support Latency Sensitive Applications. PhD thesis, Stanford University, Department of Electrical Engineering, June 2002.]]
5. D. Ferguson. {e2e} Queue size of routers. Posting to the end-to-end mailing list, January 21, 2003.]]
6. S. H. Low, F. Paganini, J. Wang, S. Adlakha, and J. C. Doyle. Dynamics of tcp/red and a scalable control. In Proceedings of IEEE INFOCOM 2002, New York, USA, June 2002.]]
7. G. Appenzeller, I. Keslassy, and N. McKeown. Sizing router buffers. Technical Report TR04-HPNG-06-08-00, Stanford University, June 2004. Extended version of the paper published at SIGCOMM 2004.]]
8. S. Iyer, R. R. Kompella, and N. McKeown. Analysis of a memory architecture for fast packet buffers. In Proceedings of IEEE High Performance Switching and Routing, Dallas, Texas, May 2001.]]
9. C. Dovrolis. {e2e} Queue size of routers. Posting to the end-to-end mailing list, January 17, 2003.]]
10. S. Shenker, L. Zhang, and D. Clark. Some observations on the dynamics of a congestion control algorithm. ACM Computer Communications Review, pages 30--39, Oct 1990.]]
11. The network simulator - ns-2. http://www.isi.edu/nsnam/ns/.]]
12. S. Floyd and V. Jacobson. Random early detection gateways for congestion avoidance. IEEE/ACM Transactions on Networking, 1(4):397--413, 1993.]]
13. L. Zhang and D. D. Clark. Oscillating behaviour of network traffic: A case study simulation. Internetworking: Research and Experience, 1:101--112, 1990.]]
14. L. Qiu, Y. Zhang, and S. Keshav. Understanding the performance of many tcp flows. Comput. Networks, 37(3-4):277--306, 2001.]]
15. G. Iannaccone, M. May, and C. Diot. Aggregate traffic performance with active queue management and drop from tail. SIGCOMM Comput. Commun. Rev., 31(3):4--13, 2001.]]
16. V. Paxson and S. Floyd. Wide area traffic: the failure of Poisson modeling. IEEE/ACM Transactions on Networking, 3(3):226--244, 1995.]]
17. A. Feldmann, A. C. Gilbert, and W. Willinger. Data networks as cascades: Investigating the multifractal nature of internet WAN traffic. In SIGCOMM, pages 42--55, 1998.]]
18. R. W. Wolff. Stochastic Modelling and the Theory of Queues, chapter 8. Prentice Hall, October 1989.]]
19. F. P. Kelly. Notes on Effective Bandwidth, pages 141--168. Oxford University Press, 1996.]]
20. J. Cao, W. Cleveland, D. Lin, and D. Sun. Internet traffic tends to poisson and independent as the load increases. Technical report, Bell Labs, 2001.]]
21. S. Floyd and V. Paxson. Difficulties in simulating the internet. IEEE/ACM Transactions on Networking, February 2001.]]
22. R. Morris. Scalable tcp congestion control. In Proceedings of IEEE INFOCOM 2000, Tel Aviv, USA, March 2000.]]
23. S. B. Fredj, T. Bonald, A. Proutière, G. Régnié, and J. Roberts. Statistical bandwidth sharing: a study of congestion at flow level. In Proceedings of SIGCOMM 2001, San Diego, USA, August 2001.]]
24. Personal communication with stanford networking on characteristics of residential traffic.]]
25. Cisco 12000 series routers. http://www.cisco.com/en/US/products/hw/routers/ps167/.]]
26. J. Sommers, H. Kim, and P. Barford. Harpoon: A flow-level traffic generator for router and network test. In Proceedings of ACM SIGMETRICS, 2004. (to appear).]]
27. I. Cisco~Systems. Netflow services solution guido, July 2001. http://www.cisco.com/.]]
28. L. Zhang, S. Shenker, and D. D. Clark. Observations on the dynamics of a congestion control algorithm: The effects of two-way traffic. In Proceedings of ACM SIGCOMM, pages 133--147, September 1991.]]
29. Microsoft. Tcp/ip and nbt configuration parameters for windows xp. Microsoft Knowledge Base Article - 314053, November 4, 2003.]]
30. K. McCloghrie and M. T. Rose. RFC 1213: Management information base for network management of TCP/IP-based internets:MIB-II, March 1991. Status: STANDARD.]]
31. R. Morris. Tcp behavior with many flows. In Proceedings of the IEEE International Conference on Network Protocols, Atlanta, Georgia, October 1997.]]
32. K. Avrachenkov, U. Ayesta, E. Altman, P. Nain, and C. Barakat. The effect of router buffer size on the tcp performance. In Proceedings of the LONIIS Workshop on Telecommunication Networks and Teletraffic Theory, pages 116--121, St.Petersburg, Russia, January 2002.]]
33. M. Garetto and D. Towsley. Modeling, simulation and measurements of queueing delay under long-tail internet traffic. In Proceedings of SIGMETRICS 2003, San Diego, USA, June 2003.]]

Index Terms

• Networks

  • Network components

    • Intermediate nodes

      • Routers