skip to main content
article

Improving file system reliability with I/O shepherding

Published:14 October 2007Publication History
Skip Abstract Section

Abstract

We introduce a new reliability infrastructure for file systems called I/O shepherding. I/O shepherding allows a file system developer to craft nuanced reliability policies to detect and recover from a wide range of storage system failures. We incorporate shepherding into the Linux ext3 file system through a set of changes to the consistency management subsystem, layout engine, disk scheduler, and buffer cache. The resulting file system, CrookFS, enables a broad class of policies to be easily and correctly specified. We implement numerous policies, incorporating data protection techniques such as retry, parity, mirrors, checksums, sanity checks, and data structure repairs; even complex policies can be implemented in less than 100 lines of code, confirming the power and simplicity of the shepherding framework. We also demonstrate that shepherding is properly integrated, adding less than 5% overhead to the I/O path.

Skip Supplemental Material Section

Supplemental Material

Video

References

  1. David G. Andersen, Deepak Bansal, Dorothy Curtis, Srinivasan Seshan, and Hari Balakrishnan. System Support for Bandwidth Management and Content Adaptation in Internet Applications. In OSDI'00, pages 213--226, San Diego, CA, October 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Dave Anderson, Jim Dykes, and Erik Riedel. More Than an Interface: SCSI vs. ATA. In FAST'03, San Francisco, CA, April 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Lakshmi Bairavasundaram. On the frequency of transient faults in modern disk drives. Personal Communication, 2007.Google ScholarGoogle Scholar
  4. Lakshmi Bairavasundaram, Garth Goodson, Shankar Pasupathy, and Jiri Schindler. An Analysis of Latent Sector Errors in Disk Drives. In SIGMETRICS'07, pages 289--300, San Diego, CA, June 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Hari Balakrishnan, Hariharan S. Rahul, and Srinivasan Seshan. An Integrated Congestion Management Architecture for Internet Hosts. In SIGCOMM'99, pages 175--187, Cambridge, MA, August 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Wendy Bartlett and Lisa Spainhower. Commercial Fault Tolerance: A Tale of Two Systems. IEEE Transactions on Dependable and Secure Computing, 1(1):87--96, January 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Aaron B. Brown and David A. Patterson. Undo for Operators: Building an Undoable E-mail Store. In USENIX'03, San Antonio, TX, June 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Andy Chou, Jun-Feng Yang, Benjamin Chelf, Seth Hallem, and Dawson Engler. An Empirical Study of Operating System Errors. In SOSP'01, pages 73--88, Banff, Canada, October 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Yvonne Coady, Gregor Kiczales, Mike Feeley, and Greg Smolyn. Using AspectC to Improve the Modularity of Path-Specific Customization in Operating System Code. In ESEC/FSE--9, September 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Peter Corbett, Bob English, Atul Goel, Tomislav Grcanac, Steven Kleiman, James Leong, and Sunitha Sankar. Row-Diagonal Parity for Double Disk Failure Correction. In FAST'04, pages 1--14, San Francisco, CA, April 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Timothy E. Denehy, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. Bridging the Information Gap in Storage Protocol Stacks. In USENIX'02, pages 177--190, Monterey, CA, June 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Dawson Engler, David Yu Chen, Seth Hallem, Andy Chou, and Benjamin Chelf. Bugs as Deviant Behavior: A General Approach to Inferring Errors in Systems Code. In SOSP'01, pages 57--72, Banff, Canada, October 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google File System. In SOSP'03, pages 29--43, Bolton Landing, NY, October 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Jim Gray. A Census of Tandem System Availability Between 1985 and 1990. Technical Report 90.1, Tandem Computers, 1990.Google ScholarGoogle Scholar
  15. Jim Gray and Andreas Reuter. Transaction Processing: Concepts and Techniques. Morgan Kaufmann, 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Roedy Green. EIDE Controller Flaws Version 24. http://mindprod.com/jgloss/eideflaw.html, February 2005.Google ScholarGoogle Scholar
  17. Robert Hagmann. Reimplementing the Cedar File System Using Logging and Group Commit. In SOSP'87, Austin, TX, November 1987. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. John S. Heidemann and Gerald J. Popek. File-system development with stackable layers. ACM Transactions on Computer Systems, 12(1):58--89, 1994. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Hai Huang, Wanda Hung, and Kang G. Shin. FS2: dynamic data replication in free disk space for improving disk performance and energy consumption. In SOSP'05, pages 263--276, Brighton, UK, October 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Gordon F. Hughes and Joseph F. Murray. Reliability and Security of RAID Storage Systems and D2D Archives Using SATA Disk Drives. ACM Transactions on Storage, 1(1):95--107, February 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Hannu H. Kari, H. Saikkonen, and F. Lombardi. Detection of Defective Media in Disks. In The IEEE International Workshop on Defect and Fault Tolerance in VLSI Systems, pages 49--55, Venice, Italy, October 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Jeffrey Katcher. PostMark: A New File System Benchmark. Technical Report TR-3022, Network Appliance Inc., October 1997.Google ScholarGoogle Scholar
  23. Gregor Kiczales, John Lamping, Anurag Mendhekar, Chris Maeda, Cristina Lopes, Jean-Marc Loingtier, and John Irwin. Aspect-Oriented Programming. In Proceedings of the European Conference on Object-Oriented Programming (ECOOP), pages 220--242, 1997.Google ScholarGoogle Scholar
  24. Steve R. Kleiman. Vnodes: An Architecture for Multiple File System Types in Sun UNIX. In USENIX Summer'86, pages 238--247, Atlanta, GA, June 1986.Google ScholarGoogle Scholar
  25. Henry F. Korth, Eliezer Levy, and Abraham Silberschatz. A Formal Approach to Recovery by Compensating Transactions. In VLDB 16, pages 95--106, Brisbane, Australia, August 1990. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Larry Lancaster and Alan Rowe. Measuring Real World Data Availability. In Proceedings of the LISA 2001 15th Systems Administration Conference, pages 93--100, San Diego, California, December 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Robert Morris, Eddie Kohler, John Jannotti, and M. Frans Kaashoek. The Click Modular Router. In SOSP'99, pages 217--231, Kiawah Island Resort, SC, December 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Kiran Nagaraja, Fabio Olivera, Ricardo Bianchini, Richard P. Martin, and Thu D. Nguyen. Understanding and Dealing with Operator Mistakes in Internet Services. In OSDI'04, San Francisco, CA, December 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. David Patterson, Garth Gibson, and Randy Katz. A Case for Redundant Arrays of Inexpensive Disks (RAID). In SIGMOD'88, pages 109--116, Chicago, IL, June 1988. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Vijayan Prabhakaran, Lakshmi N. Bairavasundaram, Nitin Agrawal, Haryadi S. Gunawi, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. IRON File Systems. In SOSP'05, pages 206--220, Brighton, UK, October 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Bianca Schroeder and Garth Gibson. Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? In FAST'07, pages 1--16, San Jose, CA, February 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Muthian Sivathanu, Vijayan Prabhakaran, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. Improving Storage System Availability with D-GRAID. In FAST'04, pages 15--30, San Francisco, CA, April 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Sun Microsystems. ZFS: The last word in file systems. www.sun.com/2004-0914/feature/, 2006.Google ScholarGoogle Scholar
  34. Rajesh Sundaram. The Private Lives of Disk Drives. http://www.netapp.com/go/techontap/matl/sample/0206tot_resiliency.html, February 2006.Google ScholarGoogle Scholar
  35. Michael M. Swift, Brian N. Bershad, and Henry M. Levy. Improving the Reliability of Commodity Operating Systems. In SOSP'03, Bolton Landing, NY, October 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Nisha Talagala and David Patterson. An Analysis of Error Behaviour in a Large Storage System. In The IEEE Workshop on Fault Tolerance in Parallel and Distributed Systems, San Juan, Puerto Rico, April 1999.Google ScholarGoogle Scholar
  37. Transaction Processing Council. TPC Benchmark B Standard Specification, Revision 3.2. Technical Report, 1990.Google ScholarGoogle Scholar
  38. Stephen C. Tweedie. Journaling the Linux ext2fs File System. In The Fourth Annual Linux Expo, Durham, North Carolina, May 1998.Google ScholarGoogle Scholar
  39. X. Yu, B. Gum, Y. Chen, R. Y. Wang, K. Li, A. Krishnamurthy, and T. E. Anderson. Trading Capacity for Performance in a Disk Array. In OSDI'00, San Diego, CA, October 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Erez Zadok and Jason Nieh. FiST: A Language for Stackable File Systems. In USENIX'00, pages 55--70, San Diego, CA, June 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Improving file system reliability with I/O shepherding

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM SIGOPS Operating Systems Review
          ACM SIGOPS Operating Systems Review  Volume 41, Issue 6
          SOSP '07
          December 2007
          363 pages
          ISSN:0163-5980
          DOI:10.1145/1323293
          Issue’s Table of Contents
          • cover image ACM Conferences
            SOSP '07: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles
            October 2007
            378 pages
            ISBN:9781595935915
            DOI:10.1145/1294261

          Copyright © 2007 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 14 October 2007

          Check for updates

          Qualifiers

          • article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!