Abstract
File systems are too large to be bug free. Although handwritten test suites have been widely used to stress file systems, they can hardly keep up with the rapid increase in file system size and complexity, leading to new bugs being introduced. These bugs come in various flavors: buffer overflows to complicated semantic bugs. Although bug-specific checkers exist, they generally lack a way to explore file system states thoroughly. More importantly, no turnkey solution exists that unifies the checking effort of various aspects of a file system under one umbrella.
In this article, to highlight the potential of applying fuzzing to find any type of file system bugs in a generic way, we propose Hydra, an extensible fuzzing framework. Hydra provides building blocks for file system fuzzing, including input mutators, feedback engines, test executors, and bug post-processors. As a result, developers only need to focus on building the core logic for finding bugs of their interests. We showcase the effectiveness of Hydra with four checkers that hunt crash inconsistency, POSIX violations, logic assertion failures, and memory errors. So far, Hydra has discovered 157 new bugs in Linux file systems, including three in verified file systems (FSCQ and Yxv6).
- Josef Bacik. 2017. Btrfs: Add a Extent Ref Verify Tool. Retrieved April 10, 2020 from https://patchwork.kernel.org/patch/9978579/.Google Scholar
- Wendy Bartlett and Lisa Spainhower. 2004. Commercial fault tolerance: A tale of two systems. IEEE Transactions on Dependable and Secure Computing 1, 1 (2004), 87--96.Google Scholar
Digital Library
- Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. 2017. Directed greybox fuzzing. In Proceedings of the 24th ACM Conference on Computer and Communications Security (CCS’17).Google Scholar
Digital Library
- Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2016. Coverage-based greybox fuzzing as Markov chain. In Proceedings of the 23rd ACM Conference on Computer and Communications Security (CCS’16).Google Scholar
Digital Library
- James Bornholt, Antoine Kaufmann, Jialin Li, Arvind Krishnamurthy, Emina Torlak, and Xi Wang. 2016. Specifying and checking file system crash-consistency models. In Proceedings of the 21st ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’16). 83--98.Google Scholar
Digital Library
- Mingming Cao, Suparna Bhattacharya, and Ted Ts’o. 2007. Ext4: The next generation of Ext2/3 filesystem. In Proceedings of the USENIX Linux Storage and Filesystem Workshop.Google Scholar
- Tej Chajed. 2018. FSCQ Developer’s Comment on Logged Writes (Git Commit). Retrieved April 10, 2020 from https://github.com/mit-pdos/fscq/commit/97b50eceedf15a2c82ce1a5cf83c231eb3184760.Google Scholar
- Tej Chajed. 2019. FSCQ Developer’s Comment on Fdatasync (GitHub Issue). Retrieved April 10, 2020 from https://github.com/mit-pdos/fscq/issues/14#issuecomment-485482506.Google Scholar
- Haogang Chen, Tej Chajed, Alex Konradi, Stephanie Wang, Atalay İleri, Adam Chlipala, M. Frans Kaashoek, and Nickolai Zeldovich. 2017. Verifying a high-performance crash-safe file system using a tree specification. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP’17).Google Scholar
Digital Library
- Haogang Chen, Daniel Ziegler, Tej Chajed, Adam Chlipala, M. Frans Kaashoek, and Nickolai Zeldovich. 2015. Using crash Hoare logic for certifying the FSCQ file system. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP’15).Google Scholar
Digital Library
- Peter Corbett, Bob English, Atul Goel, Tomislav Grcanac, Steven Kleiman, James Leong, and Sunitha Sankar. 2004. Row-diagonal parity for double disk failure correction. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies (FAST’04).Google Scholar
Digital Library
- Pedro Fonseca, Rodrigo Rodrigues, and Björn B. Brandenburg. 2014. SKI: Exposing kernel concurrency bugs through systematic schedule exploration. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI’14).Google Scholar
- Daniel Fryer, Kuei Sun, Rahat Mahmood, TingHao Cheng, Shaun Benjamin, Ashvin Goel, and Angela Demke Brown. 2012. Recon: Verifying file system consistency at runtime. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12).Google Scholar
Digital Library
- Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and Zuoning Chen. 2018. CollAFL: Path sensitive fuzzing. In Proceedings of the 39th IEEE Symposium on Security and Privacy (Oakland).Google Scholar
Cross Ref
- Google. 2016. KernelAddressSanitizer, a Fast Memory Error Detector for the Linux Kernel. Retrieved April 10, 2020 from https://github.com/google/kasan.Google Scholar
- Google. 2018. KernelMemorySanitizer, a Detector of Uses of Uninitialized Memory in the Linux Kernel. Retrieved April 10, 2020 from https://github.com/google/kmsan.Google Scholar
- Google. 2018. Syzbot. Retrieved April 10, 2020 from https://syzkaller.appspot.com.Google Scholar
- Google. 2015. KernelThreadSanitizer, a Fast Data Race Detector for the Linux Kernel. Retrieved April 10, 2020 from https://github.com/google/ktsan.Google Scholar
- Google. 2019. Honggfuzz. Retrieved April 10, 2020 from http://honggfuzz.com/.Google Scholar
- Google. 2019. Syzkaller Is an Unsupervised, Coverage-Guided Kernel Fuzzer. Retrieved April 10, 2020 from https://github.com/google/syzkaller.Google Scholar
- Bogdan Gribincea. 2009. Ext4 Data Loss. Retrieved April 10, 2020 from https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781?comments=all.Google Scholar
- Alex Groce, Gerard Holzmann, and Rajeev Joshi. 2007. Randomized differential testing as a prelude to formal verification. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07).Google Scholar
Digital Library
- HyungSeok Han and Sang Kil Cha. 2017. IMF: Inferred model-based fuzzer. In Proceedings of the 24th ACM Conference on Computer and Communications Security (CCS’17).Google Scholar
Digital Library
- Atalay Ileri, Tej Chajed, Adam Chlipala, Frans Kaashoek, and Nickolai Zeldovich. 2018. Proving confidentiality in a file system using DiskSec. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18).Google Scholar
- Dae R. Jeong, Kyungtae Kim, Basavesh Ammanaghatta Shivakumar, Byoungyoung Lee, and Insik Shin. 2019. Razzer: Finding kernel race bugs through fuzzing. In Proceedings of the 40th IEEE Symposium on Security and Privacy (Oakland).Google Scholar
Cross Ref
- Dave Jones. 2018. Linux System Call Fuzzer. Retrieved April 10, 2020 from https://github.com/kernelslacker/trinity.Google Scholar
- Jan Kara. 2014. ext4: Forbid Journal_async_commit in Data=ordered Mode. Retrieved April 10, 2020 from https://patchwork.ozlabs.org/patch/414750/.Google Scholar
- Kernel.org Bugzilla. 2018. Btrfs Bug Entries. Retrieved April 10, 2020 from https://bugzilla.kernel.org/buglist.cgi?component=btrfs.Google Scholar
- Kernel.org Bugzilla. 2018. Ext4 Bug Entries. Retrieved April 10, 2020 from https://bugzilla.kernel.org/buglist.cgi?component=ext4.Google Scholar
- Michael Kerrisk. 2019. Fsync, Fdatasync—Synchronize a File’s In-Core State with Storage Device. Retrieved April 10, 2020 from http://man7.org/linux/man-pages/man2/fdatasync.2.html.Google Scholar
- Seulbae Kim, Meng Xu, Sanidhya Kashyap, Jungyeon Yoon, Wen Xu, and Taesoo Kim. 2019. Finding semantic bugs in file systems with an extensible fuzzing framework. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP’19).Google Scholar
Digital Library
- Eric Koskinen and Junfeng Yang. 2016. Reducing crash recoverability to reachability. In Proceedings of the 43rd ACM Symposium on Principles of Programming Languages (POPL’16).Google Scholar
Digital Library
- LLVM Dev Team. 2019. LibFuzzer—A Library for Coverage-Guided Fuzz Testing. Retrieved April 10, 2020 from https://llvm.org/docs/LibFuzzer.html.Google Scholar
- Kangjie Lu, Marie-Therese Walter, David Pfaff, Stefan Nümberger, Wenke Lee, and Michael Backes. 2017. Unleashing use-before-initialization vulnerabilities in the Linux kernel using targeted stack spraying. In Proceedings of the 2017 Annual Network and Distributed System Security Symposium (NDSS’17).Google Scholar
Cross Ref
- Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Shan Lu. 2014. A study of Linux file system evolution. ACM Transactions on Storage 10, 1 (Jan. 2014), Article 3, 32 pages. DOI:https://doi.org/10.1145/2560012Google Scholar
Digital Library
- Shan Lu, Zhenmin Li, Feng Qin, Lin Tan, Pin Zhou, and Yuanyuan Zhou. 2005. BugBench: Benchmarks for evaluating bug detection tools. In Proceedings of the Workshop on the Evaluation of Software Defect Detection Tools, Vol. 5.Google Scholar
- Changwoo Min, Sanidhya Kashyap, Byoungyoung Lee, Chengyu Song, and Taesoo Kim. 2015. Cross-checking semantic correctness: The case of finding file system bugs. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP’15).Google Scholar
Digital Library
- MITRE Corporation. 2009. CVE-2009-1235. Retrieved April 10, 2020 from https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1235.Google Scholar
- Jayashree Mohan, Ashlie Martinez, Soujanya Ponnapalli, Pandian Raju, and Vijay Chidambaram. 2018. Finding crash-consistency bugs with bounded black-box crash testing. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18).Google Scholar
Digital Library
- Ingo Molnar and Arjan van de Ven. 2019. Runtime Locking Correctness Validator. Retrieved April 10, 2020 from https://www.kernel.org/doc/Documentation/locking/lockdep-design.txt.Google Scholar
- NCC Group. 2017. AFL/QEMU Fuzzing with Full-System Emulation. Retrieved April 10, 2020 from https://github.com/nccgroup/TriforceAFL.Google Scholar
- Shankara Pailoor, Andrew Aday, and Suman Jana. 2018. MoonShine: Optimizing OS fuzzer seed selection with trace distillation. In Proceedings of the 27th USENIX Security Symposium.Google Scholar
- Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP’17).Google Scholar
Digital Library
- Hui Peng, Yan Shoshitaishvili, and Mathias Payer. 2018. T-Fuzz: Fuzzing by program transformation. In Proceedings of the 39th IEEE Symposium on Security and Privacy (Oakland).Google Scholar
Cross Ref
- Vijayan Prabhakaran, Lakshmi N. Bairavasundaram, Nitin Agrawal, Haryadi S. Gunawi, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2005. IRON file systems. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP’05).Google Scholar
- Octavian Purdila, Lucian Adrian Grijincu, and Nicolae Tapus. 2010. LKL: The Linux kernel library. In Proceedings of the 9th Roedunet International Conference (RoEduNet’10). IEEE, Los Alamitos, CA.Google Scholar
- Sanjay Rawat, Vivek Jain, Ashish Kumar, Lucian Cojocar, Cristiano Giuffrida, and Herbert Bos. 2017. VUzzer: Application-aware evolutionary fuzzing. In Proceedings of the 24th ACM Conference on Computer and Communications Security (CCS’17).Google Scholar
Cross Ref
- Red Hat Inc. 2018. Utilities for Managing the XFS Filesystem. Retrieved April 10, 2020 from https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git.Google Scholar
- Tom Ridge, David Sheets, Thomas Tuerk, Andrea Giugliano, Anil Madhavapeddy, and Peter Sewell. 2015. SibylFS: Formal specification and oracle-based testing for POSIX and real-world file systems. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP’15).Google Scholar
Digital Library
- Ohad Rodeh, Josef Bacik, and Chris Mason. 2013. BTRFS: The Linux B-Tree Filesystem. ACM Transactions on Storage 9, 3 (2013), Article 9.Google Scholar
Digital Library
- Andrey Ryabinin. 2014. UBSan: Run-Time Undefined Behavior Sanity Checker. Retrieved April 10, 2020 from https://lwn.net/Articles/617364/.Google Scholar
- Sergej Schumilo, Cornelius Aschermann, Robert Gawlik, Sebastian Schinzel, and Thorsten Holz. 2017. kAFL: Hardware-assisted feedback fuzzing for OS kernels. In Proceedings of the 26th USENIX Security Symposium.Google Scholar
- GitHub. 2018. Linux Test Project. Retrieved April 10, 2020 from https://github.com/linux-test-project/ltp.Google Scholar
- Helgi Sigurbjarnarson, James Bornholt, Emina Torlak, and Xi Wang. 2016. Push-button verification of file systems via crash refinement. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16).Google Scholar
- Silicon Graphics Inc. (SGI). 2018. (X)fstests Is a Filesystem Testing Suite. Retrieved April 10, 2020 from https://github.com/kdave/xfstests.Google Scholar
- Theodore Ts’o. 2018. Ext2/3/4 File System Utilities. Retrieved April 10, 2020 from https://github.com/tytso/e2fsprogs.Google Scholar
- Theodore Ts’o. 2019. Ext4 Developer’s Comment on Fsync and Special File. Retrieved April 10, 2020 from https://bugzilla.kernel.org/show_bug.cgi?id=202485#c3.Google Scholar
- Wen Xu, Hyungon Moon, Sanidhya Kashyap, Po-Ning Tseng, and Taesoo Kim. 2019. Fuzzing file systems via two-dimensional input space exploration. In Proceedings of the 40th IEEE Symposium on Security and Privacy (Oakland).Google Scholar
Cross Ref
- Junfeng Yang, Can Sar, and Dawson Engler. 2006. Explode: A lightweight, general system for finding serious storage system errors. In Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI’06).Google Scholar
- Junfeng Yang, Can Sar, Paul Twohey, Cristian Cadar, and Dawson Engler. 2006. Automatically generating malicious disks using symbolic execution. In Proceedings of the 27th IEEE Symposium on Security and Privacy (Oakland).Google Scholar
- Junfeng Yang, Paul Twohey, Dawson Engler, and Madanlal Musuvathi. 2004. Using model checking to find serious file system errors. In Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI’04).Google Scholar
- Chao Yu. 2018. F2fs: Disable F2fs_check_rb_tree_consistence. Retrieved April 10, 2020 from https://lore.kernel.org/patchwork/patch/953794/.Google Scholar
- Michal Zalewski. 2014. Bash bug: The Other Two RCEs, or How We Chipped Away at the Original Fix (CVE-2014-6277 and’78). Retrieved April 10, 2020 from https://lcamtuf.blogspot.com/2014/10/bash-bug-how-we-finally-cracked.html.Google Scholar
- Michal Zalewski. 2019. American Fuzzy Lop (2.52b). Retrieved April 10, 2020 from https://lcamtuf.coredump.cx/afl.Google Scholar
- Andreas Zeller, Holger Cleve, and Stephan Neuhaus. 2019. Delta Debugging: From Automated Testing to Automated Debugging. Retrieved April 10, 2020 from https://www.st.cs.uni-saarland.de/dd/.Google Scholar
Index Terms
Finding Bugs in File Systems with an Extensible Fuzzing Framework
Recommendations
Finding semantic bugs in file systems with an extensible fuzzing framework
SOSP '19: Proceedings of the 27th ACM Symposium on Operating Systems PrinciplesFile systems are too large to be bug free. Although handwritten test suites have been widely used to stress file systems, they can hardly keep up with the rapid increase in file system size and complexity, leading to new bugs being introduced and ...
CrashMonkey and ACE: Systematically Testing File-System Crash Consistency
Systor 2018 Special Section on ATC 2018, Special Section on OSDI 2018 and Regular PapersWe present CrashMonkey and Ace, a set of tools to systematically find crash-consistency bugs in Linux file systems. CrashMonkey is a record-and-replay framework which tests a given workload on the target file system by simulating power-loss crashes ...






Comments