skip to main content
research-article
Public Access

Bolt: I Know What You Did Last Summer... In The Cloud

Published:04 April 2017Publication History
Skip Abstract Section

Abstract

Cloud providers routinely schedule multiple applications per physical host to increase efficiency. The resulting interference on shared resources often leads to performance degradation and, more importantly, security vulnerabilities. Interference can leak important information ranging from a service's placement to confidential data, like private keys. We present Bolt, a practical system that accurately detects the type and characteristics of applications sharing a cloud platform based on the interference an adversary sees on shared resources. Bolt leverages online data mining techniques that only require 2-5 seconds for detection. In a multi-user study on EC2, Bolt correctly identifies the characteristics of 385 out of 436 diverse workloads. Extracting this information enables a wide spectrum of previously-impractical cloud attacks, including denial of service attacks (DoS) that increase tail latency by 140x, as well as resource freeing (RFA) and co-residency attacks. Finally, we show that while advanced isolation mechanisms, such as cache partitioning lower detection accuracy, they are insufficient to eliminate these vulnerabilities altogether. To do so, one must either disallow core sharing, or only allow it between threads of the same application, leading to significant inefficiencies and performance penalties.

References

  1. Amazon ec2. http://aws.amazon.com/ec2/.Google ScholarGoogle Scholar
  2. Aman Bakshi and Yogesh B. Dujodwala. Securing cloud from ddos attacks using intrusion detection system in virtual machine. In Proc. of the 2010 Second International Conference on Communication Software and Networks (ICCSN). 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Luiz Barroso and Urs Hoelzle. The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines. MC Publishers, 2009.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Naomi Benger, Joop van de Pol, Nigel P. Smart, and Yuval Yarom. "ooh aah... just a little bit" : A small amount of side channel can go a long way. In Proc. of the International Cryptographic Hardware and Embedded Systems Workshop (CHES). Busan, South Korea, 2014.Google ScholarGoogle Scholar
  5. Major Bhadauria and Sally A. McKee. An approach to resource-aware co-scheduling for cmps. In Proc. of the 24th ACM International Conference on Supercomputing (ICS). Tsukuba, Japan, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Leon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of the International Conference on Computational Statistics (COMPSTAT). Paris, France, 2010. Google ScholarGoogle ScholarCross RefCross Ref
  7. Eric Brewer. Kubernetes: The path to cloud native. http://goo.gl/QgkzYB, SOCC Keynote, August 2015.Google ScholarGoogle Scholar
  8. Martin A. Brown. Traffic control howto. http://linux-ip.net/articles/Traffic-Control-HOWTO/.Google ScholarGoogle Scholar
  9. Robin Burke. Hybrid recommender systems: Survey and experiments. User Modeling and User-Adapted Interaction, 12(4):331--370, November 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Apache cassandra. http://cassandra.apache.org/.Google ScholarGoogle Scholar
  11. InteltextcircledR64 and IA-32 Architecture Software Developer's Manual, vol3B: System Programming Guide, Part 2, September 2014.Google ScholarGoogle Scholar
  12. Ludmila Cherkasova, Diwaker Gupta, and Amin Vahdat. Comparison of the three cpu schedulers in xen. SIGMETRICS Perform. Eval. Rev., 35(2):42--51, September 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Scott A. Crosby and Dan S. Wallach. Denial of service via algorithmic complexity attacks. In Proceedings of the 12th Conference on USENIX Security. Washington, DC, 2003.Google ScholarGoogle Scholar
  14. Marwan Darwish, Abdelkader Ouda, and Luiz Fernando Capretz. Cloud-based ddos attacks and defenses. In Proc. of i-Society. Toronto, ON, 2013.Google ScholarGoogle Scholar
  15. Christina Delimitrou and Christos Kozyrakis. iBench: Quantifying Interference for Datacenter Workloads. In Proceedings of the 2013 IEEE International Symposium on Workload Characterization (IISWC). Portland, OR, September 2013. Google ScholarGoogle ScholarCross RefCross Ref
  16. Christina Delimitrou and Christos Kozyrakis. Paragon: QoS-Aware Scheduling for Heterogeneous Datacenters. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Houston, TX, USA, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Christina Delimitrou and Christos Kozyrakis. Quasar: Resource-Efficient and QoS-Aware Cluster Management. In Proceedings of the Nineteenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Salt Lake City, UT, USA, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Christina Delimitrou and Christos Kozyrakis. HCloud: Resource-Efficient Provisioning in Shared Cloud Systems. In Proceedings of the Twenty First International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), April 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Jake Edge. Denial of service via hash collisions. http://lwn.net/Articles/474912/, January 2012.Google ScholarGoogle Scholar
  20. Benjamin Farley, Ari Juels, Venkatanathan Varadarajan, Thomas Ristenpart, Kevin D. Bowers, and Michael M. Swift. More for your money: Exploiting performance heterogeneity in public clouds. In Proc. of the ACM Symposium on Cloud Computing (SOCC). San Jose, CA, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Alexandra Fedorova, Margo Seltzer, and Michael D. Smith. Improving performance isolation on chip multiprocessors via an operating system scheduler. In Proceedings of the 16th Intl. Conference on Parallel Architecture and Compilation Techniques (PACT). Brasov, Romania, 2007. Google ScholarGoogle ScholarCross RefCross Ref
  22. Alexander Felfernig and Robin Burke. Constraint-based recommender systems: Technologies and research issues. In Proceedings of the ACM International Conference on Electronic Commerce (ICEC). Innsbruck, Austria, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Brad Fitzpatrick. Distributed caching with memcached. In Linux Journal, Volume 2004, Issue 124, 2004.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Oded Goldreich and Rafail Ostrovsky. Software protection and simulation on oblivious rams. J. ACM, 43(3):431--473, May 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Google container engine. https://cloud.google.com/container-engine.Google ScholarGoogle Scholar
  26. Asela Gunawardana and Christopher Meek. A unified approach to building hybrid recommender systems. In Proc. of the Third ACM Conference on Recommender Systems (RecSys). New York, NY, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Sanchika Gupta and Padam Kumar. Vm profile based optimized network attack pattern detection scheme for ddos attacks in cloud. In Proc. of SSCC. Mysore, India, 2013. Google ScholarGoogle ScholarCross RefCross Ref
  28. Yi Han, Tansu Alpcan, Jeffrey Chan, and Christopher Leckie. Security games for virtual machine allocation in cloud computing. In 4th International Conference on Decision and Game Theory for Security. Fort Worth, TX, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Amir Herzberg, Haya Shulman, Johanna Ullrich, and Edgar Weippl. Cloudoscopy: Services discovery and topology mapping. In Proceedings of the ACM Workshop on Cloud Computing Security Workshop (CCSW). Berlin, Germany, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Ben Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, and Ion Stoica. Mesos: A platform for fine-grained resource sharing in the data center. In Proceedings of NSDI. Boston, MA, 2011.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Jingwei Huang, David M. Nicol, and Roy H. Campbell. Denial-of-service threat to hadoop/yarn clusters with multi-tenancy. In Proc. of the IEEE International Congress on Big Data. Washington, DC, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Alexandru Iosup, Nezih Yigitbasi, and Dick Epema. On the performance variability of production cloud services. In Proceedings of CCGRID. Newport Beach, CA, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Vimalkumar Jeyakumar, Mohammad Alizadeh, David Mazières, Balaji Prabhakar, Changhoon Kim, and Albert Greenberg. Eyeq: Practical network performance isolation at the edge. In Proc. of the 10th USENIX Conference on Networked Systems Design and Implementation (NSDI). Lombard, IL, 2013.Google ScholarGoogle Scholar
  34. Yaakoub El Khamra, Hyunjoo Kim, Shantenu Jha, and Manish Parashar. Exploring the performance fluctuations of hpc workloads on clouds. In Proceedings of CloudCom. Indianapolis, IN, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Krzysztof C. Kiwiel. Convergence and efficiency of subgradient methods for quasiconvex minimization. In Mathematical Programming (Series A) (Berlin, Heidelberg: Springer) 90 (1): pp. 1--25, 2001. Google ScholarGoogle ScholarCross RefCross Ref
  36. Ruby B. Lee. Rethinking computers for cybersecurity. IEEE Computer, 48(4):16--25, 2015. Google ScholarGoogle ScholarCross RefCross Ref
  37. Jacob Leverich and Christos Kozyrakis. Reconciling high server utilization and sub-millisecond quality-of-service. In Proceedings of EuroSys. Amsterdam, The Netherlands, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Host server cpu utilization in amazon ec2 cloud. http://goo.gl/2LTx4T.Google ScholarGoogle Scholar
  39. Fangfei Liu, Yuval Yarom, Qian Ge, Gernot Heiser, and Ruby B. Lee. Last-level cache side-channel attacks are practical. In Proc. of IEEE Symposium on Security and Privacy (S&P). San Jose, CA, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Fei Liu, Lanfang Ren, and Hongtao Bai. Mitigating cross-vm side channel attack on multiple tenants cloud platform. In Journal of Computers, Vol 9, No 4 (2014), 1005--1013, April 2014. Google ScholarGoogle ScholarCross RefCross Ref
  41. David Lo, Liqun Cheng, Rama Govindaraju, Luiz André Barroso, and Christos Kozyrakis. Towards energy proportionality for large-scale latency-critical workloads. In Proceedings of the 41st Annual International Symposium on Computer Architecuture (ISCA). Minneapolis, MN, 2014. Google ScholarGoogle ScholarCross RefCross Ref
  42. David Lo, Liqun Cheng, Rama Govindaraju, Parthasarathy Ranganathan, and Christos Kozyrakis. Heracles: Improving resource efficiency at scale. In Proc. of the 42Nd Annual International Symposium on Computer Architecture (ISCA). Portland, OR, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Mahout. http://mahout.apache.org/.Google ScholarGoogle Scholar
  44. Dave Mangot. Ec2 variability: The numbers revealed. http://tech.mangot.com/roller/dave/entry/ec2_variability_the_numbers_re%vealed.Google ScholarGoogle Scholar
  45. Jason Mars and Lingjia Tang. Whare-map: heterogeneity in "homogeneous" warehouse-scale computers. In Proceedings of ISCA. Tel-Aviv, Israel, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Robert Martin, John Demme, and Simha Sethumadhavan. Timewarp: Rethinking timekeeping and performance monitoring mechanisms to mitigate side-channel attacks. In Proceedings of the International Symposium on Computer Architecture (ISCA). Portland, OR, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. David Meisner, Christopher M. Sadler, Luiz André Barroso, Wolf-Dietrich Weber, and Thomas F. Wenisch. Power management of online data-intensive services. In Proceedings of the 38th annual international symposium on Computer architecture, pages 319--330, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Jelena Mirkovic and Peter Reiher. A taxonomy of ddos attack and ddos defense mechanisms. ACM SIGCOMM Computer Communication Review (CCR), April 2004.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Thomas Moscibroda and Onur Mutlu. Memory performance attacks: Denial of memory service in multi-core systems. In Proc. of 16th USENIX Security Symposium on USENIX Security Symposium (SS). Boston, MA, 2007.Google ScholarGoogle Scholar
  50. Ripal Nathuji, Aman Kansal, and Alireza Ghaffarkhah. Q-clouds: Managing performance interference effects for qos-aware clouds. In Proceedings of EuroSys. Paris,France, 2010.Google ScholarGoogle Scholar
  51. Simon Ostermann, Alexandru Iosup, Nezih Yigitbasi, Radu Prodan, Thomas Fahringer, and Dick Epema. A performance analysis of ec2 cloud computing services for scientific computing. In Lecture Notes on Cloud Computing. Volume 34, p.115--131, 2010. Google ScholarGoogle ScholarCross RefCross Ref
  52. Tao Peng, Christopher Leckie, and Kotagiri Ramamohanarao. Survey of network-based defense mechanisms countering the dos and ddos problems. ACM Comput. Surv., 39(1), April 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Diego Perez-Botero, Jakub Szefer, and Ruby B. Lee. Characterizing hypervisor vulnerabilities in cloud computing servers. In Proceedings of the 2013 International Workshop on Security in Cloud Computing, [email protected]. Hangzhou, China, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Moinuddin K. Qureshi and Yale N. Patt. Utility-based cache partitioning: A low-overhead, high-performance, runtime mechanism to partition shared caches. In Proceedings of the 39th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 39, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Himanshu Raj, Ripal Nathuji, Abhishek Singh, and Paul England. Resource management for isolation enhanced cloud services. In Proc. of the ACM Workshop on Cloud Computing Security (CCSW). Chicago, IL, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Suhail Rehman and Majd Sakr. Initial findings for provisioning variation in cloud computing. In Proceedings of CloudCom. Indianapolis, IN, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Thomas Ristenpart, Eran Tromer, Hovav Shacham, and Stefan Savage. Hey, you, get off of my cloud: Exploring information leakage in third-party compute clouds. In Proc. of the ACM Conference on Computer and Communications Security (CCS). Chicago, IL, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Daniel Sanchez and Christos Kozyrakis. Vantage: Scalable and Efficient Fine-Grain Cache Partitioning. In Proceedings of the 38th annual International Symposium in Computer Architecture (ISCA-38). San Jose, CA, June, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Jörg Schad, Jens Dittrich, and Jorge-Arnulfo Quiané-Ruiz. Runtime measurements in the cloud: Observing, analyzing, and reducing variance. Proceedings VLDB Endow., 3(1--2):460--471, September 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, and John Wilkes. Omega: flexible, scalable schedulers for large compute clusters. In Proceedings of EuroSys. Prague, Czech Republic, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Alan Shieh, Srikanth Kandula, Albert Greenberg, and Changhoon Kim. Seawall: Performance isolation for cloud datacenter networks. In Proc. of the USENIX Conference on Hot Topics in Cloud Computing (HotCloud). Boston, MA, 2010.Google ScholarGoogle Scholar
  62. David Shue, Michael J. Freedman, and Anees Shaikh. Performance isolation and fairness for multi-tenant cloud storage. In Proc. of the 10th USENIX Conference on Operating Systems Design and Implementation (OSDI). Hollywood, CA, 2012.Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Dan Tsafrir, Yoav Etsion, and Dror G. Feitelson. Secretly monopolizing the cpu without superuser privileges. In Proc. of 16th USENIX Security Symposium on USENIX Security Symposium. Boston, MA, 2007.Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Venkatanathan Varadarajan, Thawan Kooburat, Benjamin Farley, Thomas Ristenpart, and Michael M. Swift. Resource-freeing attacks: Improve your cloud performance (at your neighbor's expense). In Proc. of the ACM Conference on Computer and Communications Security (CCS). Raleigh, NC, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Venkatanathan Varadarajan, Thomas Ristenpart, and Michael Swift. Scheduler-based defenses against cross-vm side-channels. In Proc. of the 23rd Usenix Security Symposium. San Diego, CA, 2014.Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Venkatanathan Varadarajan, Yinqian Zhang, Thomas Ristenpart, and Michael Swift. A placement vulnerability study in multi-tenant public clouds. In Proc. of the 24th USENIX Security Symposium (USENIX Security). Washington, DC, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Huaibin Wang, Haiyun Zhou, and Chundong Wang. Virtual machine-based intrusion detection system framework in cloud computing environment. In Journal of Computers, October 2012. Google ScholarGoogle ScholarCross RefCross Ref
  68. Hui Wang, Canturk Isci, Lavanya Subramanian, Jongmoo Choi, Depei Qian, and Onur Mutlu. A-drm: Architecture-aware distributed resource management of virtualized clusters. In Proceedings of the 11th ACM SIGPLAN/SIGOPS international conference on Virtual Execution Environments (VEE). Istanbul, Turkey, 2015.Google ScholarGoogle Scholar
  69. Ian H. Witten, Eibe Frank, and Geoffrey Holmes. Data Mining: Practical Machine Learning Tools and Techniques. 3rd Edition.Google ScholarGoogle Scholar
  70. Zhenyu Wu, Zhang Xu, and Haining Wang. Whispers in the hyper-space: High-speed covert channel attacks in the cloud. In Proc. of the 21st USENIX Conference on Security Symposium (USENIX Security). Bellevue, WA, 2012.Google ScholarGoogle Scholar
  71. Yunjing Xu, Michael Bailey, Farnam Jahanian, Kaustubh Joshi, Matti Hiltunen, and Richard Schlichting. An exploration of l2 cache covert channels in virtualized environments. In Proc. of the 3rd ACM Workshop on Cloud Computing Security Workshop (CCSW). Chicago, IL, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  72. Zhang Xu, Haining Wang, and Zhenyu Wu. A measurement study on co-residence threat inside the cloud. In Proc. of the 24th USENIX Security Symposium (USENIX Security). Washington, DC, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. Yuval Yarom and Katrina Falkner. FlushGoogle ScholarGoogle Scholar
  74. reload: a high resolution, low noise, l3 cache side-channel attack. In Proc. of the 23rd Usenix Security Symposium. San Diego, CA, 2014.Google ScholarGoogle Scholar
  75. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J. Franklin, Scott Shenker, and Ion Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of NSDI. San Jose, CA, 2012.Google ScholarGoogle Scholar
  76. Yinqian Zhang, Ari Juels, Alina Oprea, and Michael K. Reiter. Homealone: Co-residency detection in the cloud via side-channel analysis. In Proc. of the IEEE Symposium on Security and Privacy. Oakland, CA, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  77. Yinqian Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. Cross-tenant side-channel attacks in paas clouds. In Proc. of the ACM SIGSAC Conference on Computer and Communications Security (CCS). Scottsdale, AZ, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  78. Yinqian Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. Cross-vm side channels and their use to extract private keys. In Proceedings of the ACM Conference on Computer and Communications Security (CCS). Raleigh, NC, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  79. Yinqian Zhang and Michael K. Reiter. Duppel: retrofitting commodity operating systems to mitigate cache side channels in the cloud. In Proc. of the ACM Conference on Computer and Communications Security (CCS). Berlin, Germany, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  80. Fangfei Zhou, Manish Goel, Peter Desnoyers, and Ravi Sundaram. Scheduler vulnerabilities and coordinated attacks in cloud computing. J. Comput. Secur., 21(4):533--559, July 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  81. Jieming Zhu, Pinjia He, Zibin Zheng, and Michael R. Lyu. Towards online, accurate, and scalable qos prediction for runtime service adaptation. In Proc. of the IEEE International Conference on Distributed Computing Systems (ICDCS). Madrid, Spain, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  82. Sergey Zhuravlev, Sergey Blagodurov, and Alexandra Fedorova. Addressing shared resource contention in multicore processors via scheduling. In Proc. of the Fifteenth Edition of ASPLOS on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Pittsburgh, PA, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Bolt: I Know What You Did Last Summer... In The Cloud

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!