Abstract
Various techniques have been proposed in the literature to improve erasure code computation efficiency, including optimizing bitmatrix design and computation schedule, common XOR (exclusive-OR) operation reduction, caching management techniques, and vectorization techniques. These techniques were largely proposed individually, and, in this work, we seek to use them jointly. To accomplish this task, these techniques need to be thoroughly evaluated individually and their relation better understood. Building on extensive testing, we develop methods to systematically optimize the computation chain together with the underlying bitmatrix. This led to a simple design approach of optimizing the bitmatrix by minimizing a weighted computation cost function, and also a straightforward coding procedure—follow a computation schedule produced from the optimized bitmatrix to apply XOR-level vectorization. This procedure provides better performances than most existing techniques (e.g., those used in ISA-L and Jerasure libraries), and sometimes can even compete against well-known but less general codes such as EVENODD, RDP, and STAR codes. One particularly important observation is that vectorizing the XOR operations is a better choice than directly vectorizing finite field operations, not only because of the flexibility in choosing finite field size and the better encoding throughput, but also its minimal migration efforts onto newer CPUs.
- 2003. Library for efficient modeling and optimization in networks. Retreived from http://lemon.cs.elte.hu/trac/lemon.Google Scholar
- 2016. Intel® intelligent storage acceleration library. Retrieved from https://software.intel.com/en-us/articles/intel-isa-l-erasure-code-and-recovery.Google Scholar
- Mario Blaum, Jim Brady, Jehoshua Bruck, and Jai Menon. 1995. EVENODD: An efficient scheme for tolerating double disk failures in RAID architectures. IEEE Transactions on Computers 44, 2 (1995), 192--202.Google Scholar
Digital Library
- Johannes Blomer, Malik Kalfane, Richard Karp, Marek Karpinski, Michael Luby, and David Zuckerman. 1995. An XOR-based erasure-resilient coding scheme. Technical Report TR-95-048, University of California at Berkeley, 1995.Google Scholar
- Kevin D. Bowers, Ari Juels, and Alina Oprea. 2009. HAIL: A high-availability and integrity layer for cloud storage. In Proceedings of the 16th ACM Conference on Computer and Communications Security. ACM, 187--198.Google Scholar
Digital Library
- Peter M. Chen, Edward K. Lee, Garth A. Gibson, Randy H. Katz, and David A. Patterson. 1994. RAID: High-performance, reliable secondary storage. ACM Computing Surveys (CSUR) 26, 2 (1994), 145--185.Google Scholar
Digital Library
- Peter Corbett, Bob English, Atul Goel, Tomislav Grcanac, Steven Kleiman, James Leong, and Sunitha Sankar. 2004. Row-diagonal parity for double disk failure correction. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies. 1--14.Google Scholar
Digital Library
- Cheng Huang, Jin Li, and Minghua Chen. 2007. On optimizing XOR-based codes for fault-tolerant storage applications. In Proceedings of Information Theory Workshop. 218--223.Google Scholar
Cross Ref
- Cheng Huang and Lihao Xu. 2008. STAR: An efficient coding scheme for correcting triple storage node failures. IEEE Transactions on Computers 57, 7 (2008), 889--901.Google Scholar
Digital Library
- John Kubiatowicz, David Bindel, Yan Chen, Steven Czerwinski, Patrick Eaton, Dennis Geels, Ramakrishna Gummadi, Sean Rhea, Hakim Weatherspoon, Westley Weimer, et al. 2000. Oceanstore: An architecture for global-scale persistent storage. In ACM SIGARCH Computer Architecture News. ACM, volume 28, 190--201.Google Scholar
Digital Library
- Shu Lin and Daniel J. Costello. 2001. Error Control Coding. Pearson Education India.Google Scholar
- Witold Litwin and Thomas Schwarz. 2000. LH*RS: A high-availability scalable distributed data structure using Reed Solomon codes. In Proceedings of the ACM SIGMOD Record, volume 29, 237--248.Google Scholar
Digital Library
- Jianqiang Luo, Mochan Shrestha, Lihao Xu, and James S. Plank. 2014. Efficient encoding schedules for XOR-based erasure codes. IEEE Transactions on Computers 63, 9 (2014), 2259--2272.Google Scholar
Digital Library
- Michael Ovsiannikov, Silvius Rus, Damian Reeves, Paul Sutter, Sriram Rao, and Jim Kelly. 2013. The Quantcast file system. Proceedings of the VLDB Endowment 6, 11 (2013), 1092--1101.Google Scholar
Digital Library
- J. S. Plank and C. Huang. 2013. Tutorial: Erasure coding for storage applications. Slides presented at FAST-2013: 11th Usenix Conference on File and Storage Technologies, February 2013.Google Scholar
- James S. Plank. 2008. The RAID-6 liberation code. In Proceedings of the 6th Usenix Conference on File and Storage Technologies. 97--110.Google Scholar
Digital Library
- James S. Plank. 2011. XOR’s, lower bounds and MDS codes for storage. In Proceedings of the 2011 IEEE Information Theory Workshop (ITW). 503--507.Google Scholar
Cross Ref
- James S. Plank, Kevin M. Greenan, and Ethan L. Miller. 2013. Screaming fast Galois field arithmetic using intel SIMD instructions. In Proceedings of the 11th Usenix Conference on File and Storage Technologies. 299--306.Google Scholar
- James S. Plank, Jianqiang Luo, Catherine D. Schuman, Lihao Xu, and Zooko Wilcox-O’Hearn. 2009. A performance evaluation and examination of open-source erasure coding libraries for storage. In Fast, volume 9, 253--265.Google Scholar
- James S. Plank, Catherine D. Schuman, and B. Devin Robison. 2012. Heuristics for optimizing matrix-based erasure codes for fault-tolerant storage systems. In Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2012). 1--12.Google Scholar
Cross Ref
- James S. Plank, Scott Simmerman, and Catherine D. Schuman. 2008. Jerasure: A library in C/C++ facilitating erasure coding for storage applications-version 1.2. Technical Report CS-08-627, University of Tennessee, 2008.Google Scholar
- James S. Plank and Lihao Xu. 2006. Optimizing Cauchy Reed-Solomon codes for fault-tolerant network storage applications. In Proceedings of the 5th IEEE International Symposium on Network Computing and Applications. 173--180.Google Scholar
- Irving S. Reed and Gustave Solomon. 1960. Polynomial codes over certain finite fields. Journal of the Society for Industrial and Applied Mathematics 8, 2 (1960), 300--304.Google Scholar
Cross Ref
- Hakim Weatherspoon and John D. Kubiatowicz. 2002. Erasure coding vs. replication: A quantitative comparison. In Proceedings of the International Workshop on Peer-to-Peer Systems. 328--337.Google Scholar
- Zooko Wilcox-O’Hearn and Brian Warner. 2008. Tahoe—The least-authority filesystem. In Proceedings of the ACM Conference on Computer and Communications Security. 21--26.Google Scholar
- Tianli Zhou and Chao Tian. 2019. Fast erasure coding for data storage: A comprehensive study of the acceleration techniques. In Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST’19). 317--329.Google Scholar
Index Terms
Fast Erasure Coding for Data Storage: A Comprehensive Study of the Acceleration Techniques
Recommendations
Fast erasure coding for data storage: a comprehensive study of the acceleration techniques
FAST'19: Proceedings of the 17th USENIX Conference on File and Storage TechnologiesVarious techniques have been proposed in the literature to improve erasure code computation efficiency, including optimizing bitmatrix design, optimizing computation schedule, common XOR operation reduction, caching management techniques, and ...
On designing endurance aware erasure code for SSD-based storage systems
DPD-factor and GDP-pattern are proposed for comparing the endurance of erasure codes.EA-EO is designed as a modification of EVENODD with smaller DPD-factor.A code with smaller DPD-factor can provide higher endurance for systems.A code with sequential ...
Erasure coding in windows azure storage
USENIX ATC'12: Proceedings of the 2012 USENIX conference on Annual Technical ConferenceWindows Azure Storage (WAS) is a cloud storage system that provides customers the ability to store seemingly limitless amounts of data for any duration of time. WAS customers have access to their data from anywhere, at any time, and only pay for what ...






Comments