skip to main content
research-article

Patterns in the Chaos—A Study of Performance Variation and Predictability in Public IaaS Clouds

Authors Info & Claims
Published:19 April 2016Publication History
Skip Abstract Section

Abstract

Benchmarking the performance of public cloud providers is a common research topic. Previous work has already extensively evaluated the performance of different cloud platforms for different use cases, and under different constraints and experiment setups. In this article, we present a principled, large-scale literature review to collect and codify existing research regarding the predictability of performance in public Infrastructure-as-a-Service (IaaS) clouds. We formulate 15 hypotheses relating to the nature of performance variations in IaaS systems, to the factors of influence of performance variations, and how to compare different instance types. In a second step, we conduct extensive real-life experimentation on four cloud providers to empirically validate those hypotheses. We show that there are substantial differences between providers. Hardware heterogeneity is today less prevalent than reported in earlier research, while multitenancy has a dramatic impact on performance and predictability, but only for some cloud providers. We were unable to discover a clear impact of the time of the day or the day of the week on cloud performance.

References

  1. Sayaka Akioka and Yoichi Muraoka. 2010. HPC benchmarks on Amazon EC2. In Proceedings of the 2010 IEEE 24th International Conference on Advanced Information Networking and Applications Workshops (WAINA’10). 1029--1034. DOI:http://dx.doi.org/10.1109/WAINA.2010.166 Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2010. A view of cloud computing. Communications of the ACM 53, 4, 50--58. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Gültekin Ataş and Vehbi Cagri Gungor. 2014. Performance evaluation of cloud computing platforms using statistical methods. Computers and Electrical Engineering 40, 5, 1636--1649. DOI:http://dx.doi.org/10.1016/j.compeleceng.2014.03.017 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Sean Kenneth Barker and Prashant Shenoy. 2010. Empirical evaluation of latency-sensitive application performance in the cloud. In Proceedings of the 1st Annual ACM SIGMM Conference on Multimedia Systems (MMSys’10). ACM, New York, NY, 35--46. DOI:http://dx.doi.org/10.1145/1730836.1730842 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Amir Hossein Borhani, Philipp Leitner, Bu-Sung Lee, Xiaorong Li, and Terence Hung. 2014. WPress: Benchmarking infrastructure-as-a-service cloud computing systems for on-line transaction processing applications. In Proceedings of the 18th IEEE International Enterprise Distributed Object Computing Conference (EDOC’14).Google ScholarGoogle Scholar
  6. Rajkumar Buyya, Chee Shin Yeo, Srikumar Venugopal, James Broberg, and Ivona Brandic. 2009. Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Future Generation Computing Systems 25, 6, 599--616. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Davide Cerotti, Marco Gribaudo, Pietro Piazzolla, and Giuseppe Serazzi. 2012. Flexible CPU provisioning in clouds: A new source of performance unpredictability. International Conference on Quantitative Evaluation of Systems 0, 230--237. DOI:http://dx.doi.org/10.1109/QEST.2012.23 Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Nuttapong Chakthranont, Phonlawat Khunphet, Ryousei Takano, and Tsutomu Ikegami. 2014. Exploring the performance impact of virtualization on an HPC cloud. In Proceedings of the 2014 IEEE 6th International Conference on Cloud Computing Technology and Science (CloudCom). 426--432. DOI:http://dx.doi.org/10.1109/CloudCom.2014.71 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Matheus Cunha, Nabor Mendonca, and Americo Sampaio. 2013. A declarative environment for automatic performance evaluation in IaaS clouds. In Proceedings of the 2013 IEEE 6th International Conference on Cloud Computing (CLOUD’13). 285--292. DOI:http://dx.doi.org/10.1109/CLOUD.2013.12 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Jiang Dejun, Guillaume Pierre, and Chi-Hung Chi. 2009. EC2 performance analysis for resource provisioning of service-oriented applications. In Proceedings of the 2009 International Conference on Service-Oriented Computing (ICSOC/ServiceWave’09). Springer, Berlin, 197--207. http://dl.acm.org/citation.cfm?id=1926618.1926641 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Roberto R. Exposito, Guillermo L. Taboada, Xoan C. Pardo, Juan Tourino, and Ramon Doallo. 2013. Running scientific codes on Amazon EC2: A performance analysis of five high-end instances. Computer Science and Technology 13, 3.Google ScholarGoogle Scholar
  12. Benjamin Farley, Ari Juels, Venkatanathan Varadarajan, Thomas Ristenpart, Kevin D. Bowers, and Michael M. Swift. 2012. More for your money: Exploiting performance heterogeneity in public clouds. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC’12). ACM, New York, NY, Article 20, 14 pages. DOI:http://dx.doi.org/10.1145/2391229.2391249 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Florian Fittkau, Sören Frey, and Wilhelm Hasselbring. 2012. CDOSim: Simulating cloud deployment options for software migration support. In Proceedings of the 6th IEEE International Workshop on the Maintenance and Evolution of Service-Oriented and Cloud-Based Systems (MESOCA’12). 37--46. DOI:http://dx.doi.org/10.1109/MESOCA.2012.6392599Google ScholarGoogle ScholarCross RefCross Ref
  14. Marc Frincu, Stephane Genaud, and Julien Gossa. 2014. On the efficiency of several VM provisioning strategies for workflows with multi-threaded tasks on clouds. Computing 1--28. DOI:http://dx.doi.org/10.1007/s00607-014-0410-0 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Devarshi Ghoshal, Richard Shane Canon, and Lavanya Ramakrishnan. 2011. I/O performance of virtualized cloud environments. In Proceedings of the 2nd International Workshop on Data Intensive Computing in the Clouds (DataCloud-SC’11). ACM, New York, NY, 71--80. DOI:http://dx.doi.org/10.1145/2087522.2087535 Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Lee Gillam, Bin Li, John O’Loughlin, and Anuz Pratap Singh Tomar. 2013. Fair benchmarking for cloud computing systems. Journal of Cloud Computing: Advances, Systems and Applications 2, 6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Lazaros Gkatzikis and Iordanis Koutsopoulos. 2013. Migrate or not? Exploiting dynamic task migration in mobile cloud computing systems. IEEE Wireless Communication 20, 3, 24--32.Google ScholarGoogle ScholarCross RefCross Ref
  18. Scott Hazelhurst. 2008. Scientific computing using virtual high-performance computing: A case study using the Amazon elastic computing cloud. In Proceedings of the 2008 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries (SAICSIT’08), Vol. 338. ACM, 94--103. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Zach Hill and Marty Humphrey. 2009. A quantitative analysis of high performance computing with Amazon’s EC2 infrastructure: The death of the local cluster? In GRID. IEEE, 26--33.Google ScholarGoogle Scholar
  20. Zach Hill, Jie Li, Ming Mao, Arkaitz Ruiz-Alvarez, and Marty Humphrey. 2010. Early observations on the performance of Windows Azure. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing (HPDC’10). ACM, New York, NY, 367--376. DOI:http://dx.doi.org/10.1145/1851476.1851532 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Shigeru Imai, Thomas Chestna, and Carlos A. Varela. 2013. Accurate resource prediction for hybrid IaaS clouds using workload-tailored elastic compute units. In Proceedings of the 6th IEEE/ACM International Conference on Utility and Cloud Computing (UCC’13). Dresden, Germany. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Alexandru Iosup, Simon Ostermann, Nezih Yigitbasi, Radu Prodan, Thomas Fahringer, and Dick Epema. 2011a. Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Transactions on Parallel and Distributed Systems 22, 6, 931--945. DOI:http://dx.doi.org/10.1109/TPDS.2011.66 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Alexandru Iosup, Nezih Yigitbasi, and Dick Epema. 2011b. On the performance variability of production cloud services. In Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid’11). 104--113. DOI:http://dx.doi.org/10.1109/CCGrid.2011.22 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Keith R. Jackson, Lavanya Ramakrishnan, Krishna Muriki, Shane Canon, Shreyas Cholia, John Shalf, Harvey J. Wasserman, and Nicholas J. Wright. 2010. Performance analysis of high performance computing applications on the Amazon web services cloud. In Proceedings of the 2010 IEEE 2nd International Conference on Cloud Computing Technology and Science (CLOUDCOM’10). IEEE Computer Society, Washington, DC, 159--168. DOI:http://dx.doi.org/10.1109/CloudCom.2010.69 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Deepal Jayasinghe, Galen Swint, Simon Malkowski, Jack Li, Qingyang Wang, Junhee Park, and Calton Pu. 2012. Expertus: A generator approach to automate performance testing in IaaS clouds. In Proceedings of the’12 IEEE 5th International Conference on Cloud Computing (CLOUD’12). IEEE Computer Society, Washington, DC, 115--122. DOI:http://dx.doi.org/10.1109/CLOUD.2012.98 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Barbara A. Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-based software engineering. In Proceedings of the 26th International Conference on Software Engineering (ICSE’04). IEEE Computer Society, Washington, DC, 273--281. http://dl.acm.org/citation.cfm?id=998675.999432 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Donald Kossmann, Tim Kraska, and Simon Loesing. 2010. An evaluation of alternative architectures for transaction processing in the cloud. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (SIGMOD’10). ACM, New York, NY, 579--590. DOI:http://dx.doi.org/10.1145/1807167.1807231 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Lars Kotthoff. 2014. Reliability of computational experiments on virtualised hardware. Journal of Experimental & Theoretical Artificial Intelligence 26, 1 33--49. DOI:http://dx.doi.org/10.1080/0952813X.2013.784812Google ScholarGoogle ScholarCross RefCross Ref
  29. Philipp Leitner, Benjamin Satzger, Waldemar Hummer, Christian Inzinger, and Schahram Dustdar. 2012. CloudScale: A novel middleware for building transparently scaling cloud applications. In Proceedings of the 27th Annual ACM Symposium on Applied Computing (SAC’12). 434--440. http://doi.acm.org/10.1145/2245276.2245360 Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Philipp Leitner and Joel Scheuner. 2015. Bursting with possibilities—an empirical study of credit-based bursting cloud instance types. In Proceedings of the 8th IEEE/ACM International Conference on Utility and Cloud Computing (UCC’15).Google ScholarGoogle Scholar
  31. Alexander Lenk, Michael Menzel, Johannes Lipsky, Stefan Tai, and Philipp Offermann. 2011. What are you paying for? Performance benchmarking for infrastructure-as-a-service offerings. In IEEE CLOUD, Ling Liu and Manish Parashar (Eds.). 484--491. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Ang Li, Xiaowei Yang, Srikanth Kandula, and Ming Zhang. 2010. CloudCmp: Comparing public cloud providers. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement (IMC’10). ACM, New York, NY, 1--14. DOI:http://dx.doi.org/10.1145/1879141.1879143 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Zheng Li, Liam O’Brien, Rajiv Ranjan, and Miranda Zhang. 2013b. Early observations on performance of Google compute engine for scientific computing. In Proceedings of the 2013 IEEE International Conference on Cloud Computing Technology and Science (CLOUDCOM’13). IEEE Computer Society, Washington, DC, 1--8. DOI:http://dx.doi.org/10.1109/CloudCom.2013.7 Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Zheng Li, L. O’Brien, and He Zhang. 2013a. CEEM: A practical methodology for cloud services evaluation. In Proceedings of the 2013 IEEE 9th World Congress on Services (SERVICES’13). 44--51. DOI:http://dx.doi.org/10.1109/SERVICES.2013.73 Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Zheng Li, Liam O’Brien, He Zhang, and Rainbow Cai. 2012. On a catalogue of metrics for evaluating commercial cloud services. In Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing (GRID’12). IEEE Computer Society, Washington, DC, 164--173. DOI:http://dx.doi.org/10.1109/Grid.2012.15 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Zheng Li, Liam O’Brien, He Zhang, and Rainbow Cai. 2013. On the conceptualization of performance evaluation of IaaS services. IEEE Transactions on Services Computing.Google ScholarGoogle Scholar
  37. Ming Mao and Marty Humphrey. 2012. A performance study on the VM startup time in the cloud. In Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing (CLOUD’12). IEEE Computer Society, Washington, DC, 423--430. DOI:http://dx.doi.org/10.1109/CLOUD.2012.103 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Piyush Mehrotra, Jahed Djomehri, Steve Heistand, Robert Hood, Haoqiang Jin, Arthur Lazanoff, Subhash Saini, and Rupak Biswas. 2012. Performance evaluation of Amazon EC2 for NASA HPC applications. In Proceedings of the 3rd Workshop on Scientific Cloud Computing (ScienceCloud’12). ACM, New York, NY, 41--50. DOI:http://dx.doi.org/10.1145/2287036.2287045 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. John OLoughlin and Lee Gillam. 2013. Towards performance prediction for public infrastructure clouds: An EC2 case study. Proceedings of the 2013 IEEE 5th International Conference on Cloud Computing Technology and Science (CloudCom’13) 1, 475--480. DOI:http://dx.doi.org/10.1109/CloudCom.2013.69 Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Zhonghong Ou, Hao Zhuang, Andrey Lukyanenko, Jukka K. Nurminen, Pan Hui, Vladimir Mazalov, and Antti Yla-Jaaski. 2013. Is the same instance type created equal? Exploiting heterogeneity of public clouds. IEEE Transactions on Cloud Computing 1, 2, 201--214. DOI:http://dx.doi.org/10.1109/TCC.2013.12 Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Stephen C. Phillips, Vegard Engen, and Juri Papay. 2011. Snow white clouds and the seven dwarfs. In Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science (CLOUDCOM’11). IEEE Computer Society, Washington, DC, 738--745. DOI:http://dx.doi.org/10.1109/CloudCom.2011.114 Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Iman Sadooghi, Jesus Hernandez Martin, Tonglin Li, Kevin Brandstatter, Ketan Maheshwari, Tiago Pais Pitta de Lacerda Ruivo, Gabriele Garzoglio, Steven Timm, Yong Zhao, and Ioan Raicu. 2015. Understanding the performance and potential of cloud computing for scientific applications. IEEE Transactions on Cloud Computing PP, 99. DOI:http://dx.doi.org/10.1109/TCC.2015.2404821Google ScholarGoogle Scholar
  43. Jörg Schad, Jens Dittrich, and Jorge-Arnulfo Quiané-Ruiz. 2010. Runtime measurements in the cloud: Observing, analyzing, and reducing variance. Proceedings of the VLDB Endowment 3, 1--2, 460--471. http://dl.acm.org/citation.cfm?id=1920841.1920902 Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Joel Scheuner, Jürgen Cito, Philipp Leitner, and Harald C. Gall. 2015. Cloud workbench: Benchmarking IaaS providers based on infrastructure-as-code. In Proceedings of the 24th International World Wide Web Conference (WWW 2015)—Companion Volume. 239--242. DOI:http://dx.doi.org/10.1145/2740908.2742833 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Joel Scheuner, Philipp Leitner, Jürgen Cito, and Harald Gall. 2014. Cloud workbench—infrastructure-as-code based cloud benchmarking. In Proceedings of the 6th IEEE International Conference on Cloud Computing Technology and Science (CloudCom’14). Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Nicolas Serrano, Gorka Gallardo, and Josune Hernantes. 2015. Infrastructure as a service and cloud technologies. IEEE Software 32, 2, 30--36. DOI:http://dx.doi.org/10.1109/MS.2015.43Google ScholarGoogle ScholarCross RefCross Ref
  47. Ignacio Silva-Lepe, Revathi Subramanian, Isabelle Rouvellou, Thomas Mikalsen, Judah Diament, and Arun Iyengar. 2008. SOAlive service catalog: A simplified approach to describing, discovering and composing situational enterprise services. In Proceedings of the 6th International Conference on Service-Oriented Computing (ICSOC’08). Springer, Berlin, 422--437. DOI:http://dx.doi.org/10.1007/978-3-540-89652-4_32 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Will Sobel, Shanti Subramanyam, Akara Sucharitakul, Jimmy Nguyen, Hubert Wong, Arthur Klepchukov, Sheetal Patil, Armando Fox, and David Patterson. 2008. Cloudstone: Multi-platform, multi-language benchmark and measurement tools for Web 2.0. In Cloud Computing and Its Applications.Google ScholarGoogle Scholar
  49. Venkatanathan Varadarajan, Thawan Kooburat, Benjamin Farley, Thomas Ristenpart, and Michael M. Swift. 2012. Resource-freeing attacks: Improve your cloud performance (at your neighbor’s expense). In Proceedings of the 2012 ACM Conference on Computer and Communications Security (CCS’12). ACM, New York, NY, 281--292. DOI:http://dx.doi.org/10.1145/2382196.2382228 Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Edward Walker. 2008. Benchmarking Amazon EC2 for high-performance scientific computing. LOGIN 33, 5, 18--23.Google ScholarGoogle Scholar
  51. Guohui Wang and T. S. Eugene Ng. 2010. The impact of virtualization on network performance of Amazon EC2 data center. In Proceedings of the 29th Conference on Information Communications (INFOCOM’10). IEEE Press, Piscataway, NJ, 1163--1171. http://dl.acm.org/citation.cfm?id=1833515.1833691 Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Zhilin Wang, Xinhuai Tang, and Xiangfeng Luo. 2011. Policy-based SLA-aware cloud service provision framework. In Proceedings of the 7th International Conference on Semantics Knowledge and Grid (SKG’11). 114--121. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Jiawei Wen, Lei Lu, Giuliano Casale, and Evgenia Smirni. 2015. Less can be more: Micro-managing VMs in Amazon EC2. In Proceedings of the 8th IEEE International Conference on Cloud Computing (CLOUD). 317--324. DOI:http://dx.doi.org/10.1109/CLOUD.2015.50 Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Rostyslav Zabolotnyi, Philipp Leitner, Waldemar Hummer, and Schahram Dustdar. 2015. JCloudScale: Closing the Gap Between IaaS and PaaS. ACM Transactions on Internet Technology 15, 3, Article 10 (July 2015), 20 pages. DOI:http://dx.doi.org/10.1145/2792980 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Patterns in the Chaos—A Study of Performance Variation and Predictability in Public IaaS Clouds

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Internet Technology
        ACM Transactions on Internet Technology  Volume 16, Issue 3
        August 2016
        156 pages
        ISSN:1533-5399
        EISSN:1557-6051
        DOI:10.1145/2926746
        • Editor:
        • Munindar P. Singh
        Issue’s Table of Contents

        Copyright © 2016 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 19 April 2016
        • Accepted: 1 January 2016
        • Revised: 1 November 2015
        • Received: 1 January 2015
        Published in toit Volume 16, Issue 3

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!