Abstract
Benchmarking the performance of public cloud providers is a common research topic. Previous work has already extensively evaluated the performance of different cloud platforms for different use cases, and under different constraints and experiment setups. In this article, we present a principled, large-scale literature review to collect and codify existing research regarding the predictability of performance in public Infrastructure-as-a-Service (IaaS) clouds. We formulate 15 hypotheses relating to the nature of performance variations in IaaS systems, to the factors of influence of performance variations, and how to compare different instance types. In a second step, we conduct extensive real-life experimentation on four cloud providers to empirically validate those hypotheses. We show that there are substantial differences between providers. Hardware heterogeneity is today less prevalent than reported in earlier research, while multitenancy has a dramatic impact on performance and predictability, but only for some cloud providers. We were unable to discover a clear impact of the time of the day or the day of the week on cloud performance.
- Sayaka Akioka and Yoichi Muraoka. 2010. HPC benchmarks on Amazon EC2. In Proceedings of the 2010 IEEE 24th International Conference on Advanced Information Networking and Applications Workshops (WAINA’10). 1029--1034. DOI:http://dx.doi.org/10.1109/WAINA.2010.166 Google Scholar
Digital Library
- Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2010. A view of cloud computing. Communications of the ACM 53, 4, 50--58. Google Scholar
Digital Library
- Gültekin Ataş and Vehbi Cagri Gungor. 2014. Performance evaluation of cloud computing platforms using statistical methods. Computers and Electrical Engineering 40, 5, 1636--1649. DOI:http://dx.doi.org/10.1016/j.compeleceng.2014.03.017 Google Scholar
Digital Library
- Sean Kenneth Barker and Prashant Shenoy. 2010. Empirical evaluation of latency-sensitive application performance in the cloud. In Proceedings of the 1st Annual ACM SIGMM Conference on Multimedia Systems (MMSys’10). ACM, New York, NY, 35--46. DOI:http://dx.doi.org/10.1145/1730836.1730842 Google Scholar
Digital Library
- Amir Hossein Borhani, Philipp Leitner, Bu-Sung Lee, Xiaorong Li, and Terence Hung. 2014. WPress: Benchmarking infrastructure-as-a-service cloud computing systems for on-line transaction processing applications. In Proceedings of the 18th IEEE International Enterprise Distributed Object Computing Conference (EDOC’14).Google Scholar
- Rajkumar Buyya, Chee Shin Yeo, Srikumar Venugopal, James Broberg, and Ivona Brandic. 2009. Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Future Generation Computing Systems 25, 6, 599--616. Google Scholar
Digital Library
- Davide Cerotti, Marco Gribaudo, Pietro Piazzolla, and Giuseppe Serazzi. 2012. Flexible CPU provisioning in clouds: A new source of performance unpredictability. International Conference on Quantitative Evaluation of Systems 0, 230--237. DOI:http://dx.doi.org/10.1109/QEST.2012.23 Google Scholar
Digital Library
- Nuttapong Chakthranont, Phonlawat Khunphet, Ryousei Takano, and Tsutomu Ikegami. 2014. Exploring the performance impact of virtualization on an HPC cloud. In Proceedings of the 2014 IEEE 6th International Conference on Cloud Computing Technology and Science (CloudCom). 426--432. DOI:http://dx.doi.org/10.1109/CloudCom.2014.71 Google Scholar
Digital Library
- Matheus Cunha, Nabor Mendonca, and Americo Sampaio. 2013. A declarative environment for automatic performance evaluation in IaaS clouds. In Proceedings of the 2013 IEEE 6th International Conference on Cloud Computing (CLOUD’13). 285--292. DOI:http://dx.doi.org/10.1109/CLOUD.2013.12 Google Scholar
Digital Library
- Jiang Dejun, Guillaume Pierre, and Chi-Hung Chi. 2009. EC2 performance analysis for resource provisioning of service-oriented applications. In Proceedings of the 2009 International Conference on Service-Oriented Computing (ICSOC/ServiceWave’09). Springer, Berlin, 197--207. http://dl.acm.org/citation.cfm?id=1926618.1926641 Google Scholar
Digital Library
- Roberto R. Exposito, Guillermo L. Taboada, Xoan C. Pardo, Juan Tourino, and Ramon Doallo. 2013. Running scientific codes on Amazon EC2: A performance analysis of five high-end instances. Computer Science and Technology 13, 3.Google Scholar
- Benjamin Farley, Ari Juels, Venkatanathan Varadarajan, Thomas Ristenpart, Kevin D. Bowers, and Michael M. Swift. 2012. More for your money: Exploiting performance heterogeneity in public clouds. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC’12). ACM, New York, NY, Article 20, 14 pages. DOI:http://dx.doi.org/10.1145/2391229.2391249 Google Scholar
Digital Library
- Florian Fittkau, Sören Frey, and Wilhelm Hasselbring. 2012. CDOSim: Simulating cloud deployment options for software migration support. In Proceedings of the 6th IEEE International Workshop on the Maintenance and Evolution of Service-Oriented and Cloud-Based Systems (MESOCA’12). 37--46. DOI:http://dx.doi.org/10.1109/MESOCA.2012.6392599Google Scholar
Cross Ref
- Marc Frincu, Stephane Genaud, and Julien Gossa. 2014. On the efficiency of several VM provisioning strategies for workflows with multi-threaded tasks on clouds. Computing 1--28. DOI:http://dx.doi.org/10.1007/s00607-014-0410-0 Google Scholar
Digital Library
- Devarshi Ghoshal, Richard Shane Canon, and Lavanya Ramakrishnan. 2011. I/O performance of virtualized cloud environments. In Proceedings of the 2nd International Workshop on Data Intensive Computing in the Clouds (DataCloud-SC’11). ACM, New York, NY, 71--80. DOI:http://dx.doi.org/10.1145/2087522.2087535 Google Scholar
Digital Library
- Lee Gillam, Bin Li, John O’Loughlin, and Anuz Pratap Singh Tomar. 2013. Fair benchmarking for cloud computing systems. Journal of Cloud Computing: Advances, Systems and Applications 2, 6. Google Scholar
Digital Library
- Lazaros Gkatzikis and Iordanis Koutsopoulos. 2013. Migrate or not? Exploiting dynamic task migration in mobile cloud computing systems. IEEE Wireless Communication 20, 3, 24--32.Google Scholar
Cross Ref
- Scott Hazelhurst. 2008. Scientific computing using virtual high-performance computing: A case study using the Amazon elastic computing cloud. In Proceedings of the 2008 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries (SAICSIT’08), Vol. 338. ACM, 94--103. Google Scholar
Digital Library
- Zach Hill and Marty Humphrey. 2009. A quantitative analysis of high performance computing with Amazon’s EC2 infrastructure: The death of the local cluster? In GRID. IEEE, 26--33.Google Scholar
- Zach Hill, Jie Li, Ming Mao, Arkaitz Ruiz-Alvarez, and Marty Humphrey. 2010. Early observations on the performance of Windows Azure. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing (HPDC’10). ACM, New York, NY, 367--376. DOI:http://dx.doi.org/10.1145/1851476.1851532 Google Scholar
Digital Library
- Shigeru Imai, Thomas Chestna, and Carlos A. Varela. 2013. Accurate resource prediction for hybrid IaaS clouds using workload-tailored elastic compute units. In Proceedings of the 6th IEEE/ACM International Conference on Utility and Cloud Computing (UCC’13). Dresden, Germany. Google Scholar
Digital Library
- Alexandru Iosup, Simon Ostermann, Nezih Yigitbasi, Radu Prodan, Thomas Fahringer, and Dick Epema. 2011a. Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Transactions on Parallel and Distributed Systems 22, 6, 931--945. DOI:http://dx.doi.org/10.1109/TPDS.2011.66 Google Scholar
Digital Library
- Alexandru Iosup, Nezih Yigitbasi, and Dick Epema. 2011b. On the performance variability of production cloud services. In Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid’11). 104--113. DOI:http://dx.doi.org/10.1109/CCGrid.2011.22 Google Scholar
Digital Library
- Keith R. Jackson, Lavanya Ramakrishnan, Krishna Muriki, Shane Canon, Shreyas Cholia, John Shalf, Harvey J. Wasserman, and Nicholas J. Wright. 2010. Performance analysis of high performance computing applications on the Amazon web services cloud. In Proceedings of the 2010 IEEE 2nd International Conference on Cloud Computing Technology and Science (CLOUDCOM’10). IEEE Computer Society, Washington, DC, 159--168. DOI:http://dx.doi.org/10.1109/CloudCom.2010.69 Google Scholar
Digital Library
- Deepal Jayasinghe, Galen Swint, Simon Malkowski, Jack Li, Qingyang Wang, Junhee Park, and Calton Pu. 2012. Expertus: A generator approach to automate performance testing in IaaS clouds. In Proceedings of the’12 IEEE 5th International Conference on Cloud Computing (CLOUD’12). IEEE Computer Society, Washington, DC, 115--122. DOI:http://dx.doi.org/10.1109/CLOUD.2012.98 Google Scholar
Digital Library
- Barbara A. Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-based software engineering. In Proceedings of the 26th International Conference on Software Engineering (ICSE’04). IEEE Computer Society, Washington, DC, 273--281. http://dl.acm.org/citation.cfm?id=998675.999432 Google Scholar
Digital Library
- Donald Kossmann, Tim Kraska, and Simon Loesing. 2010. An evaluation of alternative architectures for transaction processing in the cloud. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (SIGMOD’10). ACM, New York, NY, 579--590. DOI:http://dx.doi.org/10.1145/1807167.1807231 Google Scholar
Digital Library
- Lars Kotthoff. 2014. Reliability of computational experiments on virtualised hardware. Journal of Experimental & Theoretical Artificial Intelligence 26, 1 33--49. DOI:http://dx.doi.org/10.1080/0952813X.2013.784812Google Scholar
Cross Ref
- Philipp Leitner, Benjamin Satzger, Waldemar Hummer, Christian Inzinger, and Schahram Dustdar. 2012. CloudScale: A novel middleware for building transparently scaling cloud applications. In Proceedings of the 27th Annual ACM Symposium on Applied Computing (SAC’12). 434--440. http://doi.acm.org/10.1145/2245276.2245360 Google Scholar
Digital Library
- Philipp Leitner and Joel Scheuner. 2015. Bursting with possibilities—an empirical study of credit-based bursting cloud instance types. In Proceedings of the 8th IEEE/ACM International Conference on Utility and Cloud Computing (UCC’15).Google Scholar
- Alexander Lenk, Michael Menzel, Johannes Lipsky, Stefan Tai, and Philipp Offermann. 2011. What are you paying for? Performance benchmarking for infrastructure-as-a-service offerings. In IEEE CLOUD, Ling Liu and Manish Parashar (Eds.). 484--491. Google Scholar
Digital Library
- Ang Li, Xiaowei Yang, Srikanth Kandula, and Ming Zhang. 2010. CloudCmp: Comparing public cloud providers. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement (IMC’10). ACM, New York, NY, 1--14. DOI:http://dx.doi.org/10.1145/1879141.1879143 Google Scholar
Digital Library
- Zheng Li, Liam O’Brien, Rajiv Ranjan, and Miranda Zhang. 2013b. Early observations on performance of Google compute engine for scientific computing. In Proceedings of the 2013 IEEE International Conference on Cloud Computing Technology and Science (CLOUDCOM’13). IEEE Computer Society, Washington, DC, 1--8. DOI:http://dx.doi.org/10.1109/CloudCom.2013.7 Google Scholar
Digital Library
- Zheng Li, L. O’Brien, and He Zhang. 2013a. CEEM: A practical methodology for cloud services evaluation. In Proceedings of the 2013 IEEE 9th World Congress on Services (SERVICES’13). 44--51. DOI:http://dx.doi.org/10.1109/SERVICES.2013.73 Google Scholar
Digital Library
- Zheng Li, Liam O’Brien, He Zhang, and Rainbow Cai. 2012. On a catalogue of metrics for evaluating commercial cloud services. In Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing (GRID’12). IEEE Computer Society, Washington, DC, 164--173. DOI:http://dx.doi.org/10.1109/Grid.2012.15 Google Scholar
Digital Library
- Zheng Li, Liam O’Brien, He Zhang, and Rainbow Cai. 2013. On the conceptualization of performance evaluation of IaaS services. IEEE Transactions on Services Computing.Google Scholar
- Ming Mao and Marty Humphrey. 2012. A performance study on the VM startup time in the cloud. In Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing (CLOUD’12). IEEE Computer Society, Washington, DC, 423--430. DOI:http://dx.doi.org/10.1109/CLOUD.2012.103 Google Scholar
Digital Library
- Piyush Mehrotra, Jahed Djomehri, Steve Heistand, Robert Hood, Haoqiang Jin, Arthur Lazanoff, Subhash Saini, and Rupak Biswas. 2012. Performance evaluation of Amazon EC2 for NASA HPC applications. In Proceedings of the 3rd Workshop on Scientific Cloud Computing (ScienceCloud’12). ACM, New York, NY, 41--50. DOI:http://dx.doi.org/10.1145/2287036.2287045 Google Scholar
Digital Library
- John OLoughlin and Lee Gillam. 2013. Towards performance prediction for public infrastructure clouds: An EC2 case study. Proceedings of the 2013 IEEE 5th International Conference on Cloud Computing Technology and Science (CloudCom’13) 1, 475--480. DOI:http://dx.doi.org/10.1109/CloudCom.2013.69 Google Scholar
Digital Library
- Zhonghong Ou, Hao Zhuang, Andrey Lukyanenko, Jukka K. Nurminen, Pan Hui, Vladimir Mazalov, and Antti Yla-Jaaski. 2013. Is the same instance type created equal? Exploiting heterogeneity of public clouds. IEEE Transactions on Cloud Computing 1, 2, 201--214. DOI:http://dx.doi.org/10.1109/TCC.2013.12 Google Scholar
Digital Library
- Stephen C. Phillips, Vegard Engen, and Juri Papay. 2011. Snow white clouds and the seven dwarfs. In Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science (CLOUDCOM’11). IEEE Computer Society, Washington, DC, 738--745. DOI:http://dx.doi.org/10.1109/CloudCom.2011.114 Google Scholar
Digital Library
- Iman Sadooghi, Jesus Hernandez Martin, Tonglin Li, Kevin Brandstatter, Ketan Maheshwari, Tiago Pais Pitta de Lacerda Ruivo, Gabriele Garzoglio, Steven Timm, Yong Zhao, and Ioan Raicu. 2015. Understanding the performance and potential of cloud computing for scientific applications. IEEE Transactions on Cloud Computing PP, 99. DOI:http://dx.doi.org/10.1109/TCC.2015.2404821Google Scholar
- Jörg Schad, Jens Dittrich, and Jorge-Arnulfo Quiané-Ruiz. 2010. Runtime measurements in the cloud: Observing, analyzing, and reducing variance. Proceedings of the VLDB Endowment 3, 1--2, 460--471. http://dl.acm.org/citation.cfm?id=1920841.1920902 Google Scholar
Digital Library
- Joel Scheuner, Jürgen Cito, Philipp Leitner, and Harald C. Gall. 2015. Cloud workbench: Benchmarking IaaS providers based on infrastructure-as-code. In Proceedings of the 24th International World Wide Web Conference (WWW 2015)—Companion Volume. 239--242. DOI:http://dx.doi.org/10.1145/2740908.2742833 Google Scholar
Digital Library
- Joel Scheuner, Philipp Leitner, Jürgen Cito, and Harald Gall. 2014. Cloud workbench—infrastructure-as-code based cloud benchmarking. In Proceedings of the 6th IEEE International Conference on Cloud Computing Technology and Science (CloudCom’14). Google Scholar
Digital Library
- Nicolas Serrano, Gorka Gallardo, and Josune Hernantes. 2015. Infrastructure as a service and cloud technologies. IEEE Software 32, 2, 30--36. DOI:http://dx.doi.org/10.1109/MS.2015.43Google Scholar
Cross Ref
- Ignacio Silva-Lepe, Revathi Subramanian, Isabelle Rouvellou, Thomas Mikalsen, Judah Diament, and Arun Iyengar. 2008. SOAlive service catalog: A simplified approach to describing, discovering and composing situational enterprise services. In Proceedings of the 6th International Conference on Service-Oriented Computing (ICSOC’08). Springer, Berlin, 422--437. DOI:http://dx.doi.org/10.1007/978-3-540-89652-4_32 Google Scholar
Digital Library
- Will Sobel, Shanti Subramanyam, Akara Sucharitakul, Jimmy Nguyen, Hubert Wong, Arthur Klepchukov, Sheetal Patil, Armando Fox, and David Patterson. 2008. Cloudstone: Multi-platform, multi-language benchmark and measurement tools for Web 2.0. In Cloud Computing and Its Applications.Google Scholar
- Venkatanathan Varadarajan, Thawan Kooburat, Benjamin Farley, Thomas Ristenpart, and Michael M. Swift. 2012. Resource-freeing attacks: Improve your cloud performance (at your neighbor’s expense). In Proceedings of the 2012 ACM Conference on Computer and Communications Security (CCS’12). ACM, New York, NY, 281--292. DOI:http://dx.doi.org/10.1145/2382196.2382228 Google Scholar
Digital Library
- Edward Walker. 2008. Benchmarking Amazon EC2 for high-performance scientific computing. LOGIN 33, 5, 18--23.Google Scholar
- Guohui Wang and T. S. Eugene Ng. 2010. The impact of virtualization on network performance of Amazon EC2 data center. In Proceedings of the 29th Conference on Information Communications (INFOCOM’10). IEEE Press, Piscataway, NJ, 1163--1171. http://dl.acm.org/citation.cfm?id=1833515.1833691 Google Scholar
Digital Library
- Zhilin Wang, Xinhuai Tang, and Xiangfeng Luo. 2011. Policy-based SLA-aware cloud service provision framework. In Proceedings of the 7th International Conference on Semantics Knowledge and Grid (SKG’11). 114--121. Google Scholar
Digital Library
- Jiawei Wen, Lei Lu, Giuliano Casale, and Evgenia Smirni. 2015. Less can be more: Micro-managing VMs in Amazon EC2. In Proceedings of the 8th IEEE International Conference on Cloud Computing (CLOUD). 317--324. DOI:http://dx.doi.org/10.1109/CLOUD.2015.50 Google Scholar
Digital Library
- Rostyslav Zabolotnyi, Philipp Leitner, Waldemar Hummer, and Schahram Dustdar. 2015. JCloudScale: Closing the Gap Between IaaS and PaaS. ACM Transactions on Internet Technology 15, 3, Article 10 (July 2015), 20 pages. DOI:http://dx.doi.org/10.1145/2792980 Google Scholar
Digital Library
Index Terms
Patterns in the Chaos—A Study of Performance Variation and Predictability in Public IaaS Clouds
Recommendations
Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench
ICPE '19: Companion of the 2019 ACM/SPEC International Conference on Performance EngineeringThe continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services with different performance characteristics. To support service selection, researchers and practitioners conduct cloud performance benchmarking by ...
On the Performance Evaluation of IaaS Cloud Services With System-Level Benchmarks
Infrastructure-as-a-service is a cloud service model that allows customers to outsource computing resources such as servers and storage. This article evaluates four IaaS cloud services-Amazon EC2, Microsoft Azure, Google Compute Engine and Rackspace ...
Optimal scheduling across public and private clouds in complex hybrid cloud environment
The hybrid cloud extends the private cloud model by using both local and remote resources. The private cloud will rely on the resources leased from public cloud providers for the execution of private cloud applications. The paper presents optimal ...






Comments