skip to main content
research-article

An In-depth Comparative Analysis of Cloud Block Storage Workloads: Findings and Implications

Published:06 March 2023Publication History
Skip Abstract Section

Abstract

Cloud block storage systems support diverse types of applications in modern cloud services. Characterizing their input/output (I/O) activities is critical for guiding better system designs and optimizations. In this article, we present an in-depth comparative analysis of production cloud block storage workloads through the block-level I/O traces of billions of I/O requests collected from two production systems, Alibaba Cloud and Tencent Cloud Block Storage. We study their characteristics of load intensities, spatial patterns, and temporal patterns. We also compare the cloud block storage workloads with the notable public block-level I/O workloads from the enterprise data centers at Microsoft Research Cambridge, and we identify the commonalities and differences of the three sources of traces. To this end, we provide 6 findings through the high-level analysis and 16 findings through the detailed analysis on load intensity, spatial patterns, and temporal patterns. We discuss the implications of our findings on load balancing, cache efficiency, and storage cluster management in cloud block storage systems.

REFERENCES

  1. [1] Ahmad Irfan. 2007. Easy and efficient disk I/O workload characterization in VMware ESX server. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC’07). 149158.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Alibaba. 2022. Alibaba Block Traces. Retrieved from https://github.com/alibaba/block-traces.Google ScholarGoogle Scholar
  3. [3] Alibaba. 2022. Alibaba Cloud Block Storage. Retrieved from https://www.alibabacloud.com/help/doc-detail/63136.htm.Google ScholarGoogle Scholar
  4. [4] Amazon. 2022. Amazon EBS. Retrieved from https://aws.amazon.com/ebs/.Google ScholarGoogle Scholar
  5. [5] Arteaga Dulcardo, Cabrera Jorge, Xu Jing, Sundararaman Swaminathan, and Zhao Ming. 2016. CloudCache: On-demand flash cache management for cloud computing. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 355369.Google ScholarGoogle Scholar
  6. [6] Arteaga Dulcardo and Zhao Ming. 2014. Client-side flash caching for cloud systems. In Proceedings of the 7th ACM International Systems and Storage Conference (SYSTOR’14). 111.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Atikoglu Berk, Xu Yuehai, Frachtenberg Eitan, Jiang Song, and Paleczny Mike. 2012. Workload analysis of a large-scale key-value store. In Proceedings of the ACM Special Interest Group for the Computer Performance Evaluation Community (SIGMETRICS’12). 5364.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Beaver Doug, Kumar Sanjeev, Li Harry C., Sobel Jason, and Vajgel Peter. 2010. Finding a needle in Haystack: Facebook’s photo storage. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI’10). 4760.Google ScholarGoogle Scholar
  9. [9] Bhadkamkar Medha, Guerra Jorge, Useche Luis, Burnett Sam, Liptak Jason, Rangaswami Raju, and Hristidis Vagelis. 2009. BORG: Block-reORGanization for self-optimizing storage systems. In Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST’09). 183196.Google ScholarGoogle Scholar
  10. [10] Cai Yu, Luo Yixin, Haratsch Erich F., Mai Ken, and Mutlu Onur. 2015. Data retention in MLC NAND flash memory: Characterization, optimization, and recovery. In Proceedings of the 21st IEEE International Symposium on High Performance Computer Architecture (HPCA’15). IEEE, 551563.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Cao Zhichao, Dong Siying, Vemuri Sagar, and Du David H. C.. 2020. Characterizing, modeling, and benchmarking RocksDB key-value workloads at Facebook. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST’20). 209223.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Chan Jeremy C. W., Ding Qian, Lee Patrick P. C., and Chan Helen H. W.. 2014. Parity logging with reserved space: Towards efficient updates and recovery in erasure-coded clustered storage. In Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST’14). 163176.Google ScholarGoogle Scholar
  13. [13] Chiueh Tzi-cker, Tsao Weafon, Sun Hou-Chiang, Chien Ting-Fang, Chang An-Nan, and Chen Cheng-Ding. 2014. Software orchestrated flash array. In Proceedings of the 7th ACM International Systems and Storage Conference (SYSTOR’14). 111.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Desnoyers Peter. 2012. Analytic modeling of SSD write performance. In Proceedings of the 5th ACM International Systems and Storage Conference (SYSTOR’12). 110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Han Shujie, Lee Patrick P. C., Xu Fan, Liu Yi, He Cheng, and Liu Jiongzhou. 2021. An in-depth study of correlated failures in production SSD-based data centers. In Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST’21). 417429.Google ScholarGoogle Scholar
  16. [16] Harter Tyler, Salmon Brandon, Liu Rose, Arpaci-Dusseau Andrea C., and Arpaci-Dusseau Remzi H.. 2016. Slacker: Fast distribution with lazy docker containers. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 181195.Google ScholarGoogle Scholar
  17. [17] He Jun, Kannan Sudarsun, Arpaci-Dusseau Andrea C., and Arpaci-Dusseau Remzi H.. 2017. The unwritten contract of solid state drives. In Proceedings of the 12th ACM European Conference on Computer Systems (EuroSys’17). 127144.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Hsu W. W. and Smith A. J.. 2003. Characteristics of I/O traffic in personal computer and server workloads. IBM Syst. J. 42, 2 (2003), 347372.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Kavalanekar Swaroop, Worthington Bruce, Zhang Qi, and Sharda Vishal. 2008. Characterization of storage workload traces from production windows servers. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC’08). 119128.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Lee Chunghan, Kumano Tatsuo, Matsuki Tatsuma, Endo Hiroshi, Fukumoto Naoto, and Sugawara Mariko. 2017. Understanding storage traffic characteristics on enterprise virtual desktop infrastructure. In Proceedings of the 10th ACM International Systems and Storage Conference (SYSTOR’17). 111.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Li Huiba, Zhang Yiming, Li Dongsheng, Zhang Zhiming, Liu Shengyun, Huang Peng, Qin Zheng, Chen Kai, and Xiong Yongqiang. 2019. URSA: Hybrid block storage for cloud-scale virtual disks. In Proceedings of the 14th ACM European Conference on Computer Systems (EuroSys’19). 117.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Li Jinhong, Wang Qiuping, Lee Patrick P. C., and Shi Chao. 2020. An in-depth analysis of cloud block storage workloads in large scale production. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC’20). 3747.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Li Qiao, Shi Liang, Xue Chun Jason, Wu Kaijie, Ji Cheng, Zhuge Qingfeng, and Sha Edwin H.-M.. 2016. Access characteristic guided read and write cost regulation for performance improvement on flash memory. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 125132.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Liu Ren Shuo, Yang Chia Lin, and Wu Wei. 2012. Optimizing NAND flash-based SSDs via retention relaxation. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12). 111.Google ScholarGoogle Scholar
  25. [25] Liu Shuyang, Wang Shucheng, Cao Qiang, Lu Ziyi, Jiang Hong, Yao Jie, Dong Yuanyuan, and Yang Puyuan. 2019. Analysis of and optimization for write-dominated hybrid storage nodes in cloud. In Proceedings of ACM Symposium on Cloud Computing (SoCC’19). 403415.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Liu Zaoxing, Bai Zhihao, Liu Zhenming, Li Xiaozhou, Kim Changhoon, Braverman Vladimir, Jin Xin, and Stoica Ion. 2019. DistCache: Provable load balancing for large-scale storage systems with distributed caching. In Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST’19). 143157.Google ScholarGoogle Scholar
  27. [27] Maneas Stathis, Mahdaviani Kaveh, Emami Tim, and Schroeder Bianca. 2020. A study of SSD reliability in large scale enterprise storage deployments. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST’20). 137149.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Meyer Dutch T., Aggarwal Gitika, Cully Brendan, Lefebvre Geoffrey, Feeley Michael J., Hutchinson Norman C., and Warfield Andrew. 2008. Parallax: Virtual disks for virtual machines. In Proceedings of the 3rd ACM European Conference on Computer Systems (EuroSys’08). 4154.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Mickens James, Nightingale Edmund B., Elson Jeremy, Nareddy Krishna, Gehring Darren, Fan Bin, Kadav Asim, Chidambaram Vijay, and Khan Osama. 2014. Blizzard: Fast, cloud-scale block storage for cloud-oblivious applications. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI’14). 257273.Google ScholarGoogle Scholar
  30. [30] Microsoft. 2022. MSR Cambridge Traces. Retrieved from http://iotta.snia.org/traces/388.Google ScholarGoogle Scholar
  31. [31] Min Changwoo, Kim Kangnyeon, Cho Hyunjin, Lee Sang-Won, and Eom Young Ik. 2012. SFS: Random write considered harmful in solid state drives. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12). 116.Google ScholarGoogle Scholar
  32. [32] Mishra Asit K., Hellerstein Joseph L., Cirne Walfredo, and Das Chita R.. 2010. Towards characterizing cloud backend workloads: Insights from Google compute clusters. In Proceedings of the ACM Special Interest Group for the Computer Performance Evaluation Community (SIGMETRICS’10). 3441.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Narayanan Dushyanth, Donnelly Austin, and Rowstron Antony. 2008. Write off-loading: Practical Power Management for Enterprise Storage. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST’08). 253267.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Riska Alma and Riedel Erik. 2006. Disk drive level workload characterization. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’06). 97102.Google ScholarGoogle Scholar
  35. [35] Rosenblum Mendel and Ousterhout John K.. 1992. The design and implementation of a log-structured file system. ACM Trans. Comput. Syst. 10, 1 (1992), 2652.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Saxena Mohit, Swift Michael M., and Zhang Yiying. 2012. FlashTier: A lightweight, consistent and durable storage cache. In Proceedings of the 7th ACM European Conference on Computer Systems (EuroSys’12). 267280.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Soundararajan Gokul, Prabhakaran Vijayan, Balakrishnan Mahesh, and Wobber Ted. 2010. Extending SSD lifetimes with disk-based write caches. In Proceedings of the 8th USENIX Conference on File and Storage Technologies (FAST’10). 101114.Google ScholarGoogle Scholar
  38. [38] Spearman C.. 1987. The proof and measurement of association between two things. Amer. J. Psychol. 100, 3/4 (1987), 441471.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Tarihi Mojtaba, Asadi Hossein, and Sarbazi-Azad Hamid. 2015. DiskAccel: Accelerating disk-based experiments by representative sampling. In Proceedings of the ACM Special Interest Group for the Computer Performance Evaluation Community (SIGMETRICS’15). 297308.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Tencent. 2022. Tencent Block Storage. Retrieved from http://iotta.snia.org/traces/27917.Google ScholarGoogle Scholar
  41. [41] Verma Akshat, Koller Ricardo, Useche Luis, and Rangaswami Raju. 2010. SRCMap: Energy proportional storage using dynamic consolidation. In Proceedings of the 8th USENIX Conference on File and Storage Technologies (FAST’10). 267280.Google ScholarGoogle Scholar
  42. [42] Wajahat Muhammad, Yele Aditya, Estro Tyler, Gandhi Anshul, and Zadok Erez. 2019. Distribution fitting and performance modeling for storage traces. In Proceedings of the 27th IEEE International Symposium on the Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS’19). IEEE, 138151.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Waldspurger Carl A., Park Nohhyun, Garthwaite Alexander, and Ahmad Irfan. 2015. Efficient MRC construction with SHARDS. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST’15). 95110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. [44] Wang Hua, Yi Xinbo, Huang Ping, Cheng Bin, and Zhou Ke. 2018. Efficient SSD caching by avoiding unnecessary writes using machine learning. In Proceedings of the 47th ACM International Conference on Parallel Processing (ICPP’18). 110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. [45] Wang Qiuping, Li Jinhong, Lee Patrick P. C., Ouyang Tao, Shi Chao, and Huang Lilong. 2022. Separating data via block invalidation time inference for write amplification reduction in log-structured storage. In Proceedings of the 20th USENIX Conference on File and Storage Technologies (FAST’22). 429444.Google ScholarGoogle Scholar
  46. [46] Wang Shucheng, Lu Ziyi, Cao Qiang, Jiang Hong, Yao Jie, Dong Yuanyuan, and Yang Puyuan. 2020. BCW: Buffer-controlled writes to HDDs for SSD-HDD hybrid storage server. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST’20). 253266.Google ScholarGoogle Scholar
  47. [47] Wires Jake, Ingram Stephen, Drudi Zachary, Harvey Nicholas J. A., and Warfield Andrew. 2014. Characterizing storage workloads with counter stacks. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI’14). 335349.Google ScholarGoogle Scholar
  48. [48] Xu Erci, Zheng Mai, Qin Feng, Xu Yikang, and Wu Jiesheng. 2019. Lessons and actions: What we learned from 10K SSD-related storage system failures. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’19). 961976.Google ScholarGoogle Scholar
  49. [49] Yadgar Gala, Gabel Moshe, Jaffer Shehbaz, and Schroeder Bianca. 2021. SSD-based workload characteristics and their performance implications. ACM Trans. Storage 17, 1 (2021), 126.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. [50] Yang Jing, Pei Shuyi, and Yang Qing. 2019. WARCIP: Write amplification reduction by clustering I/O pages. In Proceedings of the 12th ACM International Systems and Storage Conference (SYSTOR’19). 155166.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Yang Juncheng, Yue Yao, and Rashmi K. V.. 2020. A large scale analysis of hundreds of in-memory cache clusters at Twitter. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI’20). 191208.Google ScholarGoogle Scholar
  52. [52] Zhang Yu, Huang Ping, Zhou Ke, Wang Hua, Hu Jianying, Ji Yongguang, and Cheng Bin. 2020. OSCA: An online-model based cache allocation scheme in cloud block storage systems. In Proceedings of USENIX Annual Technical Conference (USENIX ATC’20). 785798.Google ScholarGoogle Scholar
  53. [53] Zhang Yiming, Li Huiba, Liu Shengyun, Xu Jiawei, and Xue Guangtao. 2020. PBS: An efficient erasure-coded block storage system based on speculative partial writes. ACM Trans. Storage 16, 1 (2020), 125.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. [54] Zhou Deng, Pan Wen, Wang Wei, and Xie Tao. 2015. I/O characteristics of smartphone applications and their implications for eMMC design. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC’15). 1221.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. An In-depth Comparative Analysis of Cloud Block Storage Workloads: Findings and Implications

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Storage
        ACM Transactions on Storage  Volume 19, Issue 2
        May 2023
        269 pages
        ISSN:1553-3077
        EISSN:1553-3093
        DOI:10.1145/3585541
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 6 March 2023
        • Online AM: 22 November 2022
        • Accepted: 15 November 2022
        • Revised: 1 September 2022
        • Received: 10 February 2022
        Published in tos Volume 19, Issue 2

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!