skip to main content
research-article
Public Access

Finding Bugs in File Systems with an Extensible Fuzzing Framework

Authors Info & Claims
Published:18 May 2020Publication History
Skip Abstract Section

Abstract

File systems are too large to be bug free. Although handwritten test suites have been widely used to stress file systems, they can hardly keep up with the rapid increase in file system size and complexity, leading to new bugs being introduced. These bugs come in various flavors: buffer overflows to complicated semantic bugs. Although bug-specific checkers exist, they generally lack a way to explore file system states thoroughly. More importantly, no turnkey solution exists that unifies the checking effort of various aspects of a file system under one umbrella.

In this article, to highlight the potential of applying fuzzing to find any type of file system bugs in a generic way, we propose Hydra, an extensible fuzzing framework. Hydra provides building blocks for file system fuzzing, including input mutators, feedback engines, test executors, and bug post-processors. As a result, developers only need to focus on building the core logic for finding bugs of their interests. We showcase the effectiveness of Hydra with four checkers that hunt crash inconsistency, POSIX violations, logic assertion failures, and memory errors. So far, Hydra has discovered 157 new bugs in Linux file systems, including three in verified file systems (FSCQ and Yxv6).

References

  1. Josef Bacik. 2017. Btrfs: Add a Extent Ref Verify Tool. Retrieved April 10, 2020 from https://patchwork.kernel.org/patch/9978579/.Google ScholarGoogle Scholar
  2. Wendy Bartlett and Lisa Spainhower. 2004. Commercial fault tolerance: A tale of two systems. IEEE Transactions on Dependable and Secure Computing 1, 1 (2004), 87--96.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. 2017. Directed greybox fuzzing. In Proceedings of the 24th ACM Conference on Computer and Communications Security (CCS’17).Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coverage-based greybox fuzzing as Markov chain. In Proceedings of the 23rd ACM Conference on Computer and Communications Security (CCS’16).Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. James Bornholt, Antoine Kaufmann, Jialin Li, Arvind Krishnamurthy, Emina Torlak, and Xi Wang. 2016. Specifying and checking file system crash-consistency models. In Proceedings of the 21st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’16). 83--98.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Mingming Cao, Suparna Bhattacharya, and Ted Ts’o. 2007. Ext4: The next generation of Ext2/3 filesystem. In Proceedings of the USENIX Linux Storage and Filesystem Workshop.Google ScholarGoogle Scholar
  7. Tej Chajed. 2018. FSCQ Developer’s Comment on Logged Writes (Git Commit). Retrieved April 10, 2020 from https://github.com/mit-pdos/fscq/commit/97b50eceedf15a2c82ce1a5cf83c231eb3184760.Google ScholarGoogle Scholar
  8. Tej Chajed. 2019. FSCQ Developer’s Comment on Fdatasync (GitHub Issue). Retrieved April 10, 2020 from https://github.com/mit-pdos/fscq/issues/14#issuecomment-485482506.Google ScholarGoogle Scholar
  9. Haogang Chen, Tej Chajed, Alex Konradi, Stephanie Wang, Atalay İleri, Adam Chlipala, M. Frans Kaashoek, and Nickolai Zeldovich. 2017. Verifying a high-performance crash-safe file system using a tree specification. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP’17).Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Haogang Chen, Daniel Ziegler, Tej Chajed, Adam Chlipala, M. Frans Kaashoek, and Nickolai Zeldovich. 2015. Using crash Hoare logic for certifying the FSCQ file system. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP’15).Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Peter Corbett, Bob English, Atul Goel, Tomislav Grcanac, Steven Kleiman, James Leong, and Sunitha Sankar. 2004. Row-diagonal parity for double disk failure correction. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies (FAST’04).Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Pedro Fonseca, Rodrigo Rodrigues, and Björn B. Brandenburg. 2014. SKI: Exposing kernel concurrency bugs through systematic schedule exploration. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI’14).Google ScholarGoogle Scholar
  13. Daniel Fryer, Kuei Sun, Rahat Mahmood, TingHao Cheng, Shaun Benjamin, Ashvin Goel, and Angela Demke Brown. 2012. Recon: Verifying file system consistency at runtime. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12).Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and Zuoning Chen. 2018. CollAFL: Path sensitive fuzzing. In Proceedings of the 39th IEEE Symposium on Security and Privacy (Oakland).Google ScholarGoogle ScholarCross RefCross Ref
  15. Google. 2016. KernelAddressSanitizer, a Fast Memory Error Detector for the Linux Kernel. Retrieved April 10, 2020 from https://github.com/google/kasan.Google ScholarGoogle Scholar
  16. Google. 2018. KernelMemorySanitizer, a Detector of Uses of Uninitialized Memory in the Linux Kernel. Retrieved April 10, 2020 from https://github.com/google/kmsan.Google ScholarGoogle Scholar
  17. Google. 2018. Syzbot. Retrieved April 10, 2020 from https://syzkaller.appspot.com.Google ScholarGoogle Scholar
  18. Google. 2015. KernelThreadSanitizer, a Fast Data Race Detector for the Linux Kernel. Retrieved April 10, 2020 from https://github.com/google/ktsan.Google ScholarGoogle Scholar
  19. Google. 2019. Honggfuzz. Retrieved April 10, 2020 from http://honggfuzz.com/.Google ScholarGoogle Scholar
  20. Google. 2019. Syzkaller Is an Unsupervised, Coverage-Guided Kernel Fuzzer. Retrieved April 10, 2020 from https://github.com/google/syzkaller.Google ScholarGoogle Scholar
  21. Bogdan Gribincea. 2009. Ext4 Data Loss. Retrieved April 10, 2020 from https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781?comments=all.Google ScholarGoogle Scholar
  22. Alex Groce, Gerard Holzmann, and Rajeev Joshi. 2007. Randomized differential testing as a prelude to formal verification. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07).Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. HyungSeok Han and Sang Kil Cha. 2017. IMF: Inferred model-based fuzzer. In Proceedings of the 24th ACM Conference on Computer and Communications Security (CCS’17).Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Atalay Ileri, Tej Chajed, Adam Chlipala, Frans Kaashoek, and Nickolai Zeldovich. 2018. Proving confidentiality in a file system using DiskSec. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18).Google ScholarGoogle Scholar
  25. Dae R. Jeong, Kyungtae Kim, Basavesh Ammanaghatta Shivakumar, Byoungyoung Lee, and Insik Shin. 2019. Razzer: Finding kernel race bugs through fuzzing. In Proceedings of the 40th IEEE Symposium on Security and Privacy (Oakland).Google ScholarGoogle ScholarCross RefCross Ref
  26. Dave Jones. 2018. Linux System Call Fuzzer. Retrieved April 10, 2020 from https://github.com/kernelslacker/trinity.Google ScholarGoogle Scholar
  27. Jan Kara. 2014. ext4: Forbid Journal_async_commit in Data=ordered Mode. Retrieved April 10, 2020 from https://patchwork.ozlabs.org/patch/414750/.Google ScholarGoogle Scholar
  28. Kernel.org Bugzilla. 2018. Btrfs Bug Entries. Retrieved April 10, 2020 from https://bugzilla.kernel.org/buglist.cgi?component=btrfs.Google ScholarGoogle Scholar
  29. Kernel.org Bugzilla. 2018. Ext4 Bug Entries. Retrieved April 10, 2020 from https://bugzilla.kernel.org/buglist.cgi?component=ext4.Google ScholarGoogle Scholar
  30. Michael Kerrisk. 2019. Fsync, Fdatasync—Synchronize a File’s In-Core State with Storage Device. Retrieved April 10, 2020 from http://man7.org/linux/man-pages/man2/fdatasync.2.html.Google ScholarGoogle Scholar
  31. Seulbae Kim, Meng Xu, Sanidhya Kashyap, Jungyeon Yoon, Wen Xu, and Taesoo Kim. 2019. Finding semantic bugs in file systems with an extensible fuzzing framework. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP’19).Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Eric Koskinen and Junfeng Yang. 2016. Reducing crash recoverability to reachability. In Proceedings of the 43rd ACM Symposium on Principles of Programming Languages (POPL’16).Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. LLVM Dev Team. 2019. LibFuzzer—A Library for Coverage-Guided Fuzz Testing. Retrieved April 10, 2020 from https://llvm.org/docs/LibFuzzer.html.Google ScholarGoogle Scholar
  34. Kangjie Lu, Marie-Therese Walter, David Pfaff, Stefan Nümberger, Wenke Lee, and Michael Backes. 2017. Unleashing use-before-initialization vulnerabilities in the Linux kernel using targeted stack spraying. In Proceedings of the 2017 Annual Network and Distributed System Security Symposium (NDSS’17).Google ScholarGoogle ScholarCross RefCross Ref
  35. Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Shan Lu. 2014. A study of Linux file system evolution. ACM Transactions on Storage 10, 1 (Jan. 2014), Article 3, 32 pages. DOI:https://doi.org/10.1145/2560012Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Shan Lu, Zhenmin Li, Feng Qin, Lin Tan, Pin Zhou, and Yuanyuan Zhou. 2005. BugBench: Benchmarks for evaluating bug detection tools. In Proceedings of the Workshop on the Evaluation of Software Defect Detection Tools, Vol. 5.Google ScholarGoogle Scholar
  37. Changwoo Min, Sanidhya Kashyap, Byoungyoung Lee, Chengyu Song, and Taesoo Kim. 2015. Cross-checking semantic correctness: The case of finding file system bugs. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP’15).Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. MITRE Corporation. 2009. CVE-2009-1235. Retrieved April 10, 2020 from https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1235.Google ScholarGoogle Scholar
  39. Jayashree Mohan, Ashlie Martinez, Soujanya Ponnapalli, Pandian Raju, and Vijay Chidambaram. 2018. Finding crash-consistency bugs with bounded black-box crash testing. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18).Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Ingo Molnar and Arjan van de Ven. 2019. Runtime Locking Correctness Validator. Retrieved April 10, 2020 from https://www.kernel.org/doc/Documentation/locking/lockdep-design.txt.Google ScholarGoogle Scholar
  41. NCC Group. 2017. AFL/QEMU Fuzzing with Full-System Emulation. Retrieved April 10, 2020 from https://github.com/nccgroup/TriforceAFL.Google ScholarGoogle Scholar
  42. Shankara Pailoor, Andrew Aday, and Suman Jana. 2018. MoonShine: Optimizing OS fuzzer seed selection with trace distillation. In Proceedings of the 27th USENIX Security Symposium.Google ScholarGoogle Scholar
  43. Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP’17).Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Hui Peng, Yan Shoshitaishvili, and Mathias Payer. 2018. T-Fuzz: Fuzzing by program transformation. In Proceedings of the 39th IEEE Symposium on Security and Privacy (Oakland).Google ScholarGoogle ScholarCross RefCross Ref
  45. Vijayan Prabhakaran, Lakshmi N. Bairavasundaram, Nitin Agrawal, Haryadi S. Gunawi, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2005. IRON file systems. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP’05).Google ScholarGoogle Scholar
  46. Octavian Purdila, Lucian Adrian Grijincu, and Nicolae Tapus. 2010. LKL: The Linux kernel library. In Proceedings of the 9th Roedunet International Conference (RoEduNet’10). IEEE, Los Alamitos, CA.Google ScholarGoogle Scholar
  47. Sanjay Rawat, Vivek Jain, Ashish Kumar, Lucian Cojocar, Cristiano Giuffrida, and Herbert Bos. 2017. VUzzer: Application-aware evolutionary fuzzing. In Proceedings of the 24th ACM Conference on Computer and Communications Security (CCS’17).Google ScholarGoogle ScholarCross RefCross Ref
  48. Red Hat Inc. 2018. Utilities for Managing the XFS Filesystem. Retrieved April 10, 2020 from https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git.Google ScholarGoogle Scholar
  49. Tom Ridge, David Sheets, Thomas Tuerk, Andrea Giugliano, Anil Madhavapeddy, and Peter Sewell. 2015. SibylFS: Formal specification and oracle-based testing for POSIX and real-world file systems. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP’15).Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Ohad Rodeh, Josef Bacik, and Chris Mason. 2013. BTRFS: The Linux B-Tree Filesystem. ACM Transactions on Storage 9, 3 (2013), Article 9.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Andrey Ryabinin. 2014. UBSan: Run-Time Undefined Behavior Sanity Checker. Retrieved April 10, 2020 from https://lwn.net/Articles/617364/.Google ScholarGoogle Scholar
  52. Sergej Schumilo, Cornelius Aschermann, Robert Gawlik, Sebastian Schinzel, and Thorsten Holz. 2017. kAFL: Hardware-assisted feedback fuzzing for OS kernels. In Proceedings of the 26th USENIX Security Symposium.Google ScholarGoogle Scholar
  53. GitHub. 2018. Linux Test Project. Retrieved April 10, 2020 from https://github.com/linux-test-project/ltp.Google ScholarGoogle Scholar
  54. Helgi Sigurbjarnarson, James Bornholt, Emina Torlak, and Xi Wang. 2016. Push-button verification of file systems via crash refinement. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16).Google ScholarGoogle Scholar
  55. Silicon Graphics Inc. (SGI). 2018. (X)fstests Is a Filesystem Testing Suite. Retrieved April 10, 2020 from https://github.com/kdave/xfstests.Google ScholarGoogle Scholar
  56. Theodore Ts’o. 2018. Ext2/3/4 File System Utilities. Retrieved April 10, 2020 from https://github.com/tytso/e2fsprogs.Google ScholarGoogle Scholar
  57. Theodore Ts’o. 2019. Ext4 Developer’s Comment on Fsync and Special File. Retrieved April 10, 2020 from https://bugzilla.kernel.org/show_bug.cgi?id=202485#c3.Google ScholarGoogle Scholar
  58. Wen Xu, Hyungon Moon, Sanidhya Kashyap, Po-Ning Tseng, and Taesoo Kim. 2019. Fuzzing file systems via two-dimensional input space exploration. In Proceedings of the 40th IEEE Symposium on Security and Privacy (Oakland).Google ScholarGoogle ScholarCross RefCross Ref
  59. Junfeng Yang, Can Sar, and Dawson Engler. 2006. Explode: A lightweight, general system for finding serious storage system errors. In Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI’06).Google ScholarGoogle Scholar
  60. Junfeng Yang, Can Sar, Paul Twohey, Cristian Cadar, and Dawson Engler. 2006. Automatically generating malicious disks using symbolic execution. In Proceedings of the 27th IEEE Symposium on Security and Privacy (Oakland).Google ScholarGoogle Scholar
  61. Junfeng Yang, Paul Twohey, Dawson Engler, and Madanlal Musuvathi. 2004. Using model checking to find serious file system errors. In Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI’04).Google ScholarGoogle Scholar
  62. Chao Yu. 2018. F2fs: Disable F2fs_check_rb_tree_consistence. Retrieved April 10, 2020 from https://lore.kernel.org/patchwork/patch/953794/.Google ScholarGoogle Scholar
  63. Michal Zalewski. 2014. Bash bug: The Other Two RCEs, or How We Chipped Away at the Original Fix (CVE-2014-6277 and’78). Retrieved April 10, 2020 from https://lcamtuf.blogspot.com/2014/10/bash-bug-how-we-finally-cracked.html.Google ScholarGoogle Scholar
  64. Michal Zalewski. 2019. American Fuzzy Lop (2.52b). Retrieved April 10, 2020 from https://lcamtuf.coredump.cx/afl.Google ScholarGoogle Scholar
  65. Andreas Zeller, Holger Cleve, and Stephan Neuhaus. 2019. Delta Debugging: From Automated Testing to Automated Debugging. Retrieved April 10, 2020 from https://www.st.cs.uni-saarland.de/dd/.Google ScholarGoogle Scholar

Index Terms

  1. Finding Bugs in File Systems with an Extensible Fuzzing Framework

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Storage
        ACM Transactions on Storage  Volume 16, Issue 2
        SOSP 2019 Special Section and Regular Papers
        May 2020
        194 pages
        ISSN:1553-3077
        EISSN:1553-3093
        DOI:10.1145/3399155
        • Editor:
        • Sam H. Noh
        Issue’s Table of Contents

        Copyright © 2020 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 18 May 2020
        • Online AM: 7 May 2020
        • Revised: 1 March 2020
        • Accepted: 1 March 2020
        • Received: 1 January 2020
        Published in tos Volume 16, Issue 2

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!