Abstract
Load balancers choose among load-balanced paths to distribute traffic as if it makes no difference using one path or another. This work shows that the latency difference between load-balanced paths (called latency imbalance ), previously deemed insignificant, is now prevalent from the perspective of the cloud and affects various latency-sensitive applications. In this work, we present the first large-scale measurement study of latency imbalance from a cloud-centric view. Using public cloud around the globe, we measure latency imbalance both between data centers (DCs) in the cloud and from the cloud to the public Internet. Our key findings include that 1) Amazon's and Alibaba's clouds together have latency difference between load-balanced paths larger than 20ms to 21% of public IPv4 addresses; 2) Google's secret in having lower latency imbalance than other clouds is to use its own well-balanced private WANs to transit traffic close to the destinations and that 3) latency imbalance is also prevalent between DCs in the cloud, where 8 pairs of DCs are found to have load-balanced paths with latency difference larger than 40ms. We further evaluate the impact of latency imbalance on three applications (i.e., NTP, delay-based geolocation and VoIP) and propose potential solutions to improve application performance. Our experiments show that all three applications can benefit from considering latency imbalance, where the accuracy of delay-based geolocation can be greatly improved by simply changing how \textttping measures the minimum path latency.
- Brice Augustin, Timur Friedman, and Renata Teixeira. Measuring load-balanced paths in the Internet. In IMC, 2007.Google Scholar
Digital Library
- Sandeep Kumar Singh, Tamal Das, and Admela Jukan. A survey on Internet multipath routing and provisioning. IEEE Communications Surveys & Tutorials, 17(4):2157--2175, 2015.Google Scholar
Digital Library
- Brice Augustin, Xavier Cuvellier, Benjamin Orgogozo, Fabien Viger, Timur Friedman, Matthieu Latapy, Clémence Magnien, and Renata Teixeira. Avoiding traceroute anomalies with Paris Traceroute. In IMC, 2006.Google Scholar
Digital Library
- Kevin Vermeulen, Stephen D. Strowes, Olivier Fourmaux, and Timur Friedman. Multilevel MDA-Lite Paris Traceroute. In IMC, 2018.Google Scholar
Digital Library
- Imtiaz Parvez, Ali Rahmati, Ismail Guvenc, Arif I. Sarwat, and Huaiyu Dai. A survey on low latency towards 5G: RAN, core network and caching solutions. IEEE Communications Surveys & Tutorials, 20(4):3098--3130, 2018.Google Scholar
Digital Library
- Rupa Krishnan, Harsha V. Madhyastha, Sridhar Srinivasan, Sushant Jain, Arvind Krishnamurthy, Thomas Anderson, and Jie Gao. Moving beyond end-to-end path information to optimize CDN performance. In IMC, pages 190--201, 2009.Google Scholar
Digital Library
- Zachary Weinberg, Shinyoung Cho, Nicolas Christin, Vyas Sekar, and Phillipa Gill. How to catch when proxies lie: Verifying the physical locations of network proxies with active geolocation. In IMC, pages 203--217, 2018.Google Scholar
Digital Library
- Frank Dabek, Russ Cox, Frans Kaashoek, and Robert Morris. Vivaldi: A decentralized network coordinate system. In SIGCOMM CCR, pages 15--26, 2004.Google Scholar
- Sathiya Kumaran Mani, Ramakrishnan Durairajan, Paul Barford, and Joel Sommers. MNTP: Enhancing time synchronization for mobile devices. In IMC, 2016. Proc. ACM Meas. Anal. Comput. Syst., Vol. 4, No. 2, Article 32. Publication date: June 2020. 32:24 Yibo Pi et al.Google Scholar
- Cristina D. Murta, Pedro R. Torres Jr, and Prasant Mohapatra. Qrpp1--4: Characterizing quality of time and topology in a time synchronization network. In Globecom, pages 1--5, 2006.Google Scholar
Cross Ref
- Cristel Pelsser, Luca Cittadini, Stefano Vissicchio, and Randy Bush. From Paris to Tokyo: On the suitability of ping to measure latency. In IMC, 2013.Google Scholar
Digital Library
- Mark Gondree and Zachary NJ Peterson. Geolocation of data in the cloud. In Proceedings of the third ACM conference on Data and application security and privacy, pages 25--36, 2013.Google Scholar
Digital Library
- Junchen Jiang et al. Via: Improving Internet telephony call quality using predictive relay selection. In SIGCOMM, 2016.Google Scholar
- Our tool and dataset. https://github.com/yibopi/latency-imbalance.Google Scholar
- Yuval Shavitt and Noa Zilberman. A geolocation databases study. IEEE Journal on Selected Areas in Communications, 29(10):2044--2056, 2011.Google Scholar
Cross Ref
- David L. Mills. Simple network time protocol (SNTP) version 4 for IPv4, IPv6 and OSI. https://tools.ietf.org/html/rfc4330, 2006.Google Scholar
- Jonathan Ledlie, Paul Gardner, and Margo I. Seltzer. Network coordinates in the wild. In NSDI, volume 7, pages 299--311, 2007.Google Scholar
Digital Library
- Zhiruo Cao, Zheng Wang, and Ellen Zegura. Performance of hashing-based schemes for Internet load balancing. In INFOCOM, pages 332--341, 2000.Google Scholar
- Darryl Veitch, Brice Augustin, Renata Teixeira, and Timur Friedman. Failure control in multipath route tracing. In INFOCOM, 2009.Google Scholar
Cross Ref
- Rafael Almeida, Osvaldo Fonseca, Elverton Fazzion, Dorgival Guedes, Wagner Meira, and Ítalo Cunha. A characterization of load balancing on the IPv6 Internet. In PAM, 2017.Google Scholar
Cross Ref
- Yves Vanaubel, Pascal Mérindol, Jean-Jacques Pansiot, and Benoit Donnet. MPLS under the microscope: Revealing actual transit path diversity. In IMC, 2015.Google Scholar
Digital Library
- Eric Elena, Jean-Louis Rougier, and Stefano Secci. Characterisation of AS-level path deviations and multipath in Internet routing. In 6th EURO-NGI Conference on Next Generation Internet, 2010.Google Scholar
Cross Ref
- Matthew Luckie, Young Hyun, and Bradley Huffaker. Traceroute probe method and forward IP path inference. In IMC, 2008.Google Scholar
Digital Library
- Zhe Wu, Curtis Yu, and Harsha V. Madhyastha. CosTLO: Cost-effective redundancy for lower latency variance on cloud storage services. In NSDI, 2015.Google Scholar
Digital Library
- Bamba Gueye, Artur Ziviani, Mark Crovella, and Serge Fdida. Constraint-based geolocation of Internet hosts. TON, 14(6):1219--1232, 2006.Google Scholar
- Fangfei Chen, Ramesh K. Sitaraman, and Marcelo Torres. End-user mapping: Next generation request routing for content delivery. In CCR, 2015.Google Scholar
- Yibo Pi, Sugih Jamin, Peter Danzig, and Jacob Shaha. AP-Atoms: A high-accuracy data-driven client aggregation for global load balancing. IEEE/ACM Transactions on Networking, 26(6):2748--2761, December 2018.Google Scholar
Digital Library
- Mohammed S. Elbamby, Cristina Perfecto, Mehdi Bennis, and Klaus Doppler. Toward low-latency and ultra-reliable virtual reality. IEEE Network, 32(2):78--84, 2018.Google Scholar
Cross Ref
- Roy D. Yates, Mehrnaz Tavan, Yi Hu, and Dipankar Raychaudhuri. Timely cloud gaming. In INFOCOM, 2017.Google Scholar
Cross Ref
- Zhiruo Cao, Zheng Wang, and Ellen Zegura. Performance of hashing-based schemes for Internet load balancing. In INFOCOM, 2000.Google Scholar
- Yilong Geng, Shiyu Liu, Zi Yin, Ashish Naik, Balaji Prabhakar, Mendel Rosenblum, and Amin Vahdat. Exploiting a natural network effect for scalable, fine-grained clock synchronization. In NSDI, pages 81--94, 2018.Google Scholar
- Ethan Katz-Bassett, Harsha V. Madhyastha, Vijay Kumar Adhikari, Colin Scott, Justine Sherry, Peter Van Wesep, Thomas E. Anderson, and Arvind Krishnamurthy. Reverse traceroute. In NSDI, 2010.Google Scholar
Digital Library
- MDA-Lite Gitlab repository. https://gitlab.planet-lab.eu/cartography/multilevel-mda-lite, Sep 2018.Google Scholar
- Abhinav Pathak, Himabindu Pucha, Ying Zhang, Y. Charlie Hu, and Z. Morley Mao. A measurement study of Internet delay asymmetry. In PAM, 2008.Google Scholar
Cross Ref
- Robert Beverly. Yarrp'ing the Internet: Randomized high-speed active topology discovery. In IMC, pages 413--420, 2016.Google Scholar
Digital Library
- Perry R. Hinton. Statistics explained. Routledge, 2014.Google Scholar
Cross Ref
- Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4):764--766, 2013.Google Scholar
Cross Ref
- Youndo Lee and Neil Spring. Identifying and aggregating homogeneous IPv4 /24 blocks with Hobbit. In IMC, pages 151--165, 2016.Google Scholar
Digital Library
- Ki Suh Lee, Han Wang, Vishal Shrivastav, and Hakim Weatherspoon. Globally synchronized time via datacenter networks. In SIGCOMM, 2016.Google Scholar
Digital Library
- IP latency statistics. https://enterprise.verizon.com/terms/latency/, July 2019. Proc. ACM Meas. Anal. Comput. Syst., Vol. 4, No. 2, Article 32. Publication date: June 2020. Latency Imbalance Among Internet Load-Balanced Paths: A Cloud-Centric View 32:25Google Scholar
- Sadjad Fouladi, John Emmons, Emre Orbay, Catherine Wu, Riad S. Wahby, and Keith Winstein. Salsify: low-latency network video through tighter integration between a video codec and a transport protocol. In NSDI, 2018.Google Scholar
- CAIDA. Routeviews prefix to AS mappings dataset for IPv4 and IPv6. https://www.caida.org/data/routing/routeviewsprefix2as. xml, Sep 2018.Google Scholar
- Timothy Wood, Prashant J. Shenoy, Alexandre Gerber, Jacobus E. van der Merwe, and Kadangode K. Ramakrishnan. The case for enterprise-ready virtual private clouds. In HotCloud, 2009.Google Scholar
- Sathiya Kumaran Mani, Ramakrishnan Durairajan, Paul Barford, and Joel Sommers. An architecture for IoT clock synchronization. In Proceedings of the 8th International Conference on the Internet of Things, 2018.Google Scholar
Digital Library
- Alexander Marder, Matthew Luckie, Amogh Dhamdhere, Bradley Huffaker, kc claffy, and Jonathan M. Smith. Pushing the boundaries with bdrmapit: Mapping router ownership at Internet scale. In IMC, 2018.Google Scholar
Digital Library
- CAIDA. The CAIDA Internet Topology Data Kit. https://www.caida.org/data/internet-topology-data-kit, August 2018.Google Scholar
- AFRINIC Extended Allocation and Assignment Reports. ftp://ftp.afrinic.net/pub/stats/afrinic/, 2019.Google Scholar
- RIPE Extended Allocation and Assignment Reports. ftp://ftp.afrinic.net/pub/stats/ripencc/, 2019.Google Scholar
- LACNIC Extended Allocation and Assignment Reports. ftp://ftp.afrinic.net/pub/stats/lacnic/, 2019.Google Scholar
- APNIC Extended Allocation and Assignment Reports. ftp://ftp.afrinic.net/pub/stats/apnic/, 2019.Google Scholar
- ARIN Extended Allocation and Assignment Reports. ftp://ftp.afrinic.net/pub/stats/arin/, 2019.Google Scholar
- Euro-IX IXP Directory. https://www.euro-ix.net/tools/ixp-directory, 2019.Google Scholar
- PeeringDB. https://peeringdb.com/api, 2019.Google Scholar
- Packet Clearing House: Internet Exchange Directory. https://www.pch.net/applications/ixpdir/menu_download.php, 2019.Google Scholar
- CAIDA. The CAIDA AS Relationships Dataset. https://www.caida.org/data/as-relationships/, Feb 2019.Google Scholar
- Google cloud network service tiers. https://cloud.google.com/network-tiers#tab1.Google Scholar
- Matthew Luckie, Amogh Dhamdhere, kc Claffy, and David Murrell. Measured impact of crooked traceroute. In CCR, pages 14--21, 2011.Google Scholar
Digital Library
- Ramesh Govindan and Vern Paxson. Estimating router ICMP generation delays. In PAM, 2002.Google Scholar
- Yves Vanaubel, Pascal Mérindol, Jean-Jacques Pansiot, and Benoit Donnet. Through the wormhole: Tracking invisible MPLS tunnels. In IMC, 2017.Google Scholar
Digital Library
- Joel Sommers, Paul Barford, and Brian Eriksson. On the prevalence and characteristics of MPLS deployments in the open Internet. In IMC, 2011.Google Scholar
Digital Library
- Benoit Donnet, Matthew Luckie, Pascal Mérindol, and Jean-Jacques Pansiot. Revealing MPLS tunnels obscured from traceroute. In SIGCOMM CCR, 2012.Google Scholar
Digital Library
- Abhinav Pathak, Ming Zhang, Y. Charlie Hu, Ratul Mahajan, and Dave Maltz. Latency inflation with MPLS-based traffic engineering. In IMC, 2011.Google Scholar
Digital Library
- Chi-Yao Hong, Srikanth Kandula, Ratul Mahajan, Ming Zhang, Mohan Nanduri, and Roger Wattenhofer. Achieving high utilization with software-driven WAN. In SIGCOMM CCR, 2013.Google Scholar
Digital Library
- Sushant Jain et al. B4: Experience with a globally-deployed software defined WAN. In SIGCOMM, pages 3--14, 2013.Google Scholar
- Ramakrishnan Durairajan, Sathiya Kumaran Mani, Joel Sommers, and Paul Barford. Time's Forgotten: Using NTP to understand Internet latency. In HotNets, 2015.Google Scholar
Digital Library
- Amazon AWS IoT. https://aws.amazon.com/iot/.Google Scholar
- Karyn Benson, Rafael Dowsley, and Hovav Shacham. Do you know where your cloud files are? In Proceedings of the 3rd ACM workshop on Cloud computing security workshop, pages 73--82, 2011.Google Scholar
Digital Library
- Yang Xu, Chenguang Yu, Jingjiang Li, and Yong Liu. Video telephony for end-consumers: Measurement study of Google+, iChat, and Skype. In IMC, pages 371--384, 2012.Google Scholar
Digital Library
- Osama Haq, Mamoon Raja, and Fahad R. Dogar. Measuring and improving the reliability of wide-area cloud paths. In WWW, 2017.Google Scholar
Digital Library
- Venkata N. Padmanabhan and Lakshminarayanan Subramanian. An investigation of geographic mapping techniques for Internet hosts. In SIGCOMM, 2001.Google Scholar
- Planetlab locations. https://www.planet-lab.org/db/pub/sites.php, April, 2019.Google Scholar
- iPlane project dataset. http://web.eecs.umich.edu/~harshavm/iplane/iplane_logs/data/.Google Scholar
- G.107. The E-Model, a computational model for user in transmission planning, 2017. https://www.itu.int/rec/T-REC-G.107.Google Scholar
- Haiyong Xie and Yang Richard Yang. A measurement-based study of the skype peer-to-peer VoIP performance. In IPTPS, 2012.Google Scholar
Index Terms
Latency Imbalance Among Internet Load-Balanced Paths: A Cloud-Centric View
Recommendations
Latency Imbalance Among Internet Load-Balanced Paths: A Cloud-Centric View
SIGMETRICS '20: Abstracts of the 2020 SIGMETRICS/Performance Joint International Conference on Measurement and Modeling of Computer SystemsLoad balancers choose among load-balanced paths to distribute traffic as if it makes no difference using one path or another. This work shows that the latency difference between load-balanced paths (called latency imbalance), previously deemed ...
Latency Imbalance Among Internet Load-Balanced Paths: A Cloud-Centric View
Load balancers choose among load-balanced paths to distribute traffic as if it makes no difference using one path or another. This work shows that the latency difference between load-balanced paths (called latency imbalance), previously deemed ...
Load-balanced three-stage switch
A load-balanced two-stage switch is scalable and can provide close to 100% throughput. Its major problem is that packets can be mis-sequenced when they arrive at outputs. In a recent work, the packet mis-sequencing problem is elegantly solved by a ...






Comments