skip to main content
research-article

Lightweight Dynamic Redundancy Control with Adaptive Encoding for Server-based Storage

Authors Info & Claims
Published:15 October 2021Publication History
Skip Abstract Section

Abstract

With the recent performance improvements in commodity hardware, low-cost commodity server-based storage has become a practical alternative to dedicated-storage appliances. Because of the high failure rate of commodity servers, data redundancy across multiple servers is required in a server-based storage system. However, the extra storage capacity for this redundancy significantly increases the system cost. Although erasure coding (EC) is a promising method to reduce the amount of redundant data, it requires distributing and encoding data among servers. There remains a need to reduce the performance impact of these processes involving much network traffic and processing overhead. Especially, the performance impact becomes significant for random-intensive applications. In this article, we propose a new lightweight redundancy control for server-based storage. Our proposed method uses a new local filesystem-based approach that avoids distributing data by adding data redundancy to locally stored user data. Our method switches the redundancy method of user data between replication and EC according to workloads to improve capacity efficiency while achieving higher performance. Our experiments show up to 230% better online-transaction-processing performance for our method compared with CephFS, a widely used alternative system. We also confirmed that our proposed method prevents unexpected performance degradation while achieving better capacity efficiency.

REFERENCES

  1. [1] Aghayev Abutalib, Weil Sage, Kuchnik Michael, Nelson Mark, Ganger Gregory R., and Amvrosiadis George. 2019. File systems unfit as distributed storage backends: Lessons from 10 years of Ceph evolution. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP'19). ACM, 353369. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Apache.org. 2019. Welcome to Apache ZooKeeper. Retrieved from https://zookeeper.apache.org/.Google ScholarGoogle Scholar
  3. [3] Barroso Luiz A., Holzle Urs, Ranganathan Parthasarathy, and Martonosi Margaret. 2018. The Datacenter as a Computer: Designing Warehouse-scale Machines (3rd ed.). Morgan & Claypool Publishers. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Beserra Patricia V., Camara Alessandro, Ximenes Rafael, Albuquerque Adriano B., and Mendonça Nabor C.. 2012. Cloudstep: A step-by-step decision process to support legacy application migration to the cloud. In Proceedings of the IEEE 6th International Workshop on the Maintenance and Evolution of Service-oriented and Cloud-based Systems (MESOCA'12). IEEE, 716.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Borthakur Dhruba. 2007. The Hadoop distributed file system: Architecture and design. Retrieved from http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.16.3/docs/hdfs_design.pdfGoogle ScholarGoogle Scholar
  6. [6] Bovet Daniel P. and Cesati Marco. 2005. Understanding the Linux Kernel: From I/O Ports to Process Management. O'Reilly Media, Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Boyer Eric B., Broomfield Matthew C., and Perrotti Terrell A.. 2012. GlusterFS One Storage Server to Rule Them All. Technical Report. Los Alamos National Lab (LANL).Google ScholarGoogle Scholar
  8. [8] Cano Ignacio, Aiyar Srinivas, Arora Varun, Bhattacharyya Manosiz, Chaganti Akhilesh, Cheah Chern, Chun Brent, Gupta Karan, Khot Vinayak, and Krishnamurthy Arvind. 2017. Curator: Self-managing storage for enterprise clusters. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI'17). 5166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Chan Jeremy C. W., Ding Qian, Lee Patrick P. C., and Chan Helen H. W.. 2014. Parity logging with reserved space: Towards efficient updates and recovery in erasure-coded clustered storage. In Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST'14). 163176. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Chen Yanpei, Srinivasan Kiran, Goodson Garth, and Katz Randy. 2011. Design implications for enterprise storage systems via multi-dimensional trace analysis. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP'15). 4356. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Corbet Johnathan. 2010. Open by handle. Retrieved from https://lwn.net/Articles/375888/.Google ScholarGoogle Scholar
  12. [12] Corbet Johnathan. 2010. Punching holes in files. Retrieved from https://lwn.net/Articles/415889/.Google ScholarGoogle Scholar
  13. [13] Corporation Intel. 2020. intel.com. Retrieved from https://www.intel.co.jp/.Google ScholarGoogle Scholar
  14. [14] Fan Bin, Tantisiriroj Wittawat, Xiao Lin, and Gibson Garth. 2011. DiskReduce: Replication as a Prelude to Erasure Coding in Data-intensive Scalable Computing. Technical Report CMU-PDL-11-112, Carnegie Mellon Univsersity, Parallel Data Laboratory.Google ScholarGoogle Scholar
  15. [15] Fikes Andrew. 2010. Storage architecture and challenges. Talk at the Google Faculty Summit.Google ScholarGoogle Scholar
  16. [16] Fukatani Takayuki, Le Hieu Hanh, and Yokota Haruo. 2019. Lightweight dynamic redundancy control for server-based storage. In Proceedings of the 38th International Symposium on Reliable Distributed Systems (SRDS'19). IEEE, 353369.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Fukatani Takayuki, Sutoh Atsushi, and Nakano Takahiro. 2018. A method to adapt storage protocol stack using custom file metadata to commodity Linux servers. Int. J. Smart Comput. Artif. Intell 2, 1 (2018), 2342.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Gibson Garth. 2010. DiskReduce v2.0 for HDFS. Retrieved from https://www.pdl.cmu.edu/DiskReduce/talks/2010-Jan-28-OpenCirrus-Gibson.pdf.Google ScholarGoogle Scholar
  19. [19] Huang Cheng, Simitci Huseyin, Xu Yikang, Ogus Aaron, Calder Brad, Gopalan Parikshit, Li Jin, and Yekhanin Sergey. 2012. Erasure coding in Windows Azure storage. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC'12). 1526. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] IEEE and Group The Open. 2018. The open group base specifications issue 7, 2018 edition (IEEE Std 1003.1™-2017).Google ScholarGoogle Scholar
  21. [21] Izraelevitz Joseph, Yang Jian, Zhang Lu, Kim Juno, Liu Xiao, Memaripour Amirsaman, Soh Yun Joon, Wang Zixuan, Xu Yi, Dulloor Subramanya R. et al. 2019. Basic performance measurements of the Intel Optane DC persistent memory module. arXiv preprint arXiv:1903.05714 (2019).Google ScholarGoogle Scholar
  22. [22] C Mohammed Rafi K.. 2016. Efficient data tiering in GlusterFS. In Proceedings of the Storage Developer Conference (SDC'16).Google ScholarGoogle Scholar
  23. [23] kernel.org. 2020. The Linux Kernel archives. Retrieved from https://www.kernel.org.Google ScholarGoogle Scholar
  24. [24] Khan Osama, Burns Randal C., Plank James S., Pierce William, and Huang Cheng. 2012. Rethinking erasure codes for cloud file systems: Minimizing I/O for recovery and degraded reads. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST'12). 20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Kim Won. 2009. Cloud computing: Today and tomorrow. J. Object Technol. 8, 1 (2009), 6572.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Koh Sungjoon, Zhang Jie, Kwon Miryeong, Yoon Jungyeon, Donofrio David, Kim Nam Sung, and Jung Myoungsoo. 2017. Understanding system characteristics of online erasure coding on scalable, distributed and large-scale SSD array systems. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC'17). IEEE, 7686.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Lee Chunghan, Kumano Tatsuo, Matsuki Tatsuma, Endo Hiroshi, Fukumoto Naoto, and Sugawara Mariko. 2017. Understanding storage traffic characteristics on enterprise virtual desktop infrastructure. In Proceedings of the 10th ACM International Systems and Storage Conference (SYSTOR'17). ACM, 111. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Lee Dong-Yun, Jeong Kisik, Han Sang-Hoon, Kim Jin-Soo, Hwang Joo-Young, and Cho Sangyeun. 2017. Understanding write behaviors of storage backends in Ceph object store. In Proceedings of the IEEE International Conference on Massive Storage Systems and Technology (MSST'17), Vol. 10.Google ScholarGoogle Scholar
  29. [29] Leung Andrew W., Pasupathy Shankar, Goodson Garth R., and Miller Ethan L.. 2008. Measurement and analysis of large-scale network file system workloads. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC'08), Vol. 1. 5–2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Li Runhui, Hu Yuchong, and Lee Patrick P. C.. 2017. Enabling efficient and reliable transition from replication to erasure coding for clustered file systems. IEEE Trans. Parallel Distrib. Syst. 28, 9 (2017), 25002513.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] libfuse. 2020. libfuse. Retrieved from https://github.com/libfuse.Google ScholarGoogle Scholar
  32. [32] Madhyastha Harsha V., McCullough John, Porter George, Kapoor Rishi, Savage Stefan, Snoeren Alex C., and Vahdat Amin. 2012. Scc: Cluster storage provisioning informed by application characteristics and SLAs. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST'12). Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Mathur Avantika, Cao Mingming, Bhattacharya Suparna, Dilger Andreas, Tomas Alex, and Vivier Laurent. 2007. The new ext4 filesystem: Current status and future plans. In Proceedings of the Linux Symposium, Vol. 2. 2133.Google ScholarGoogle Scholar
  34. [34] Matsuzawa Keiichi, Hayasaka Mitsuo, and Shinagawa Takahiro. 2020. Practical quick file server migration. ACM Trans. Stor. 16, 2 (2020), 130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Mattos Steve. 2012. Server-based storage. In Proceedings of the Flash Memory Summit.Google ScholarGoogle Scholar
  36. [36] Meyer Dutch T. and Bolosky William J.. 2012. A study of practical deduplication. ACM Trans. Stor. 7, 4 (2012). Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Mickens James, Nightingale Edmund B., Elson Jeremy, Gehring Darren, Fan Bin, Kadav Asim, Chidambaram Vijay, Khan Osama, and Nareddy Krishna. 2014. Blizzard: Fast, cloud-scale block storage for cloud-oblivious applications. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI'14). 257273. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. [38] Microsoft. 2020. Microsoft Azure Lsv2-series. Retrieved from https://docs.microsoft.com/en-us/azure/virtual-machines/lsv2-series.Google ScholarGoogle Scholar
  39. [39] Narayanamurthy Srinivasan. 2016. Modern erasure codes for distributed storage systems. In Proceedings of the Storage Developer Conference (SDC'16).Google ScholarGoogle Scholar
  40. [40] Narayanan Dushyanth, Donnelly Austin, and Rowstron Antony. 2008. Write off-loading: Practical power management for enterprise storage. ACM Trans. Stor. 4, 3 (2008), 123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. [41] Nemoto Jun, Sutoh Atsushi, and Iwasaki Masaaki. 2017. Directory-aware file system backup to object storage for fast on-demand restore. Int. J. Smart Comput. Artif. Intell. 1, 1 (2017), 119.Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Niu Junpeng, Xu Jun, and Xie Lihua. 2018. Hybrid storage systems: A survey of architectures and algorithms. IEEE Access 6 (2018), 1338513406.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Plank James S.. 2013. Erasure codes for storage systems: A brief primer. Login: The USENIX Mag. 38, 6 (2013), 4450.Google ScholarGoogle Scholar
  44. [44] Plank James S., Simmerman Scott, and Schuman Catherine D.. 2007. Jerasure: A Library in C/C++ Facilitating Erasure Coding for Storage Applications. Technical Report CS-07–603, University of Tennessee.Google ScholarGoogle Scholar
  45. [45] Pontes Rogério, Burihabwa Dorian, Maia Francisco, Paulo Joao, Schiavoni Valerio, Felber Pascal, Mercier Hugues, and Oliveira Rui. 2017. SafeFS: A modular architecture for secure user-space file systems: One FUSE to rule them all. In Proceedings of the 10th ACM International Systems and Storage Conference (SYSTOR'17). ACM, 9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. [46] Rashmi K. V., Nakkiran Preetum, Wang Jingyan, Shah Nihar B., and Ramchandran Kannan. 2015. Having your cake and eating it too: Jointly optimal erasure codes for I/O, storage, and network-bandwidth. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST'15). 8194. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. [47] Sanjay Ghemawat. 2003. The Google file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP'03). ACM, 2943. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Services Amazon Web. 2020. Amazon EC2 I3 Instances. Retrieved from https://aws.amazon.com/ec2/instance-types/i3/.Google ScholarGoogle Scholar
  49. [49] Sharon Shachar. 2020. PMEM file-system in user-space. In Proceedings of the Storage Developer Conference (SDC'20).Google ScholarGoogle Scholar
  50. [50] spec.org. 2017. SPEC SFS®2014. Retrieved from http://spec.org/sfs2014/.Google ScholarGoogle Scholar
  51. [51] Tarasov Vasily, Zadok Erez, and Shepler Spencer. 2016. Filebench: A flexible framework for file system benchmarking. login: The USENIX Magazine 41, 1 (2016), 612.Google ScholarGoogle Scholar
  52. [52] Umrao Vikhyat, Hackett Michael, and Singh Karan. 2017. Ceph Cookbook: Practical Recipes to Design, Implement, Operate, and Manage Ceph Storage Systems. Packt Publishing Ltd. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. [53] Vajha Myna, Ramkumar Vinayak, Puranik Bhagyashree, Kini Ganesh, Lobo Elita, Sasidharan Birenjith, Kumar P. Vijay, Barg Alexandar, Ye Min, Narayanamurthy Srinivasan, et al. 2018. Clay codes: Moulding MDS codes to yield an MSR code. In 16th USENIX Conference on File and Storage Technologies (FAST'18). 139154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. [54] Vangoor Bharath Kumar Reddy, Tarasov Vasily, and Zadok Erez. 2017. To FUSE or not to FUSE: Performance of user-space file systems. In 15th USENIX Conference on File and Storage Technologies (FAST'17). 5972. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. [55] vmware. 2018. VMware vSAN 6.7 Technical Overview. https://storagehub.vmware.com/static/media/65dde536-5a57-4226-87fd-1c08fd63516a.pdf.Google ScholarGoogle Scholar
  56. [56] Wallace Grant, Douglis Fred, Qian Hangwei, Shilane Philip, Smaldone Stephen, Chamness Mark, and Hsu Windsor. 2012. Characteristics of backup workloads in production systems.. In 10th USENIX Conference on File and Storage Technologies (FAST'12). 116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. [57] Wei Shuzhan, Li Yongkun, Xu Yinlong, and Wu Si. 2017. DSC: Dynamic stripe construction for asynchronous encoding in clustered file system. In Proceedings of IEEE INFOCOM 2017 International Conference on Computer Communications. IEEE, 19.Google ScholarGoogle ScholarCross RefCross Ref
  58. [58] Weil Sage A., Brandt Scott A., Miller Ethan L., Long Darrell DE, and Maltzahn Carlos. 2006. Ceph: A scalable, high-performance distributed file system. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI'06). USENIX Association, 307320. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. [59] Wires Jake and Warfield Andrew. 2017. Mirador: An active control plane for datacenter storage. In 15th USENIX Conference on File and Storage Technologies (FAST'17). 213228. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. [60] XFS.org. 2013. XFS.org. https://xfs.org/.Google ScholarGoogle Scholar
  61. [61] Xia Mingyuan, Saxena Mohit, Blaum Mario, and Pease David A.. 2015. A tale of two erasure codes in HDFS. In 13th USENIX Conference on File and Storage Technologies (FAST'15). 213226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. [62] Xu Qiumin, Siyamwala Huzefa, Ghosh Mrinmoy, Suri Tameesh, Awasthi Manu, Guz Zvika, Shayesteh Anahita, and Balakrishnan Vijay. 2015. Performance analysis of NVMe SSDs and their implication on realworld databases. In Proceedings of the 8th ACM International Systems and Storage Conference (SYSTOR 2015). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. [63] Xue Shuai, Zhao Shang, Chen Quan, Deng Gang, Liu Zheng, Zhang Jie, Song Zhuo, Ma Tao, Yang Yong, Zhou Yanbo, Niu Keqiang, Sun Sijie, and Guo Minyi. 2020. Spool: Reliable virtualized NVMe storage pool in public cloud infrastructure. In 2020 USENIX Annual Technical Conference (USENIX ATC'20). 97110. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. [64] Yang Yue and Zhu Jianwen. 2016. Write skew and zipf distribution: Evidence and implications. ACM Transactions on Storage (TOS) 12, 4 (2016), 21. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. [65] Zhang Yiming, Li Huiba, Liu Shengyun, Xu Jiawei, and Xue Guangtao. 2020. PBS: An efficient erasure-coded block storage system based on speculative partial writes. ACM Transactions on Storage (TOS) 16, 1 (2020), 125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. [66] Zhou Tianli and Tian Chao. 2019. Fast erasure coding for data storage: A comprehensive study of the acceleration techniques. In 17th USENIX Conference on File and Storage Technologies (FAST'19). 317329. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Lightweight Dynamic Redundancy Control with Adaptive Encoding for Server-based Storage

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on Storage
            ACM Transactions on Storage  Volume 17, Issue 4
            November 2021
            201 pages
            ISSN:1553-3077
            EISSN:1553-3093
            DOI:10.1145/3487989
            • Editor:
            • Sam H. Noh
            Issue’s Table of Contents

            Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 15 October 2021
            • Accepted: 1 March 2021
            • Revised: 1 January 2021
            • Received: 1 December 2019
            Published in tos Volume 17, Issue 4

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Refereed
          • Article Metrics

            • Downloads (Last 12 months)103
            • Downloads (Last 6 weeks)4

            Other Metrics

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          Full Text

          View this article in Full Text.

          View Full Text

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!