Abstract
Traditionally, file systems were implemented as part of operating systems kernels, which provide a limited set of tools and facilities to a programmer. As the complexity of file systems grew, many new file systems began being developed in user space. Low performance is considered the main disadvantage of user-space file systems but the extent of this problem has never been explored systematically. As a result, the topic of user-space file systems remains rather controversial: while some consider user-space file systems a “toy” not to be used in production, others develop full-fledged production file systems in user space. In this article, we analyze the design and implementation of a well-known user-space file system framework, FUSE, for Linux. We characterize its performance and resource utilization for a wide range of workloads. We present FUSE performance and also resource utilization with various mount and configuration options, using 45 different workloads that were generated using Filebench on two different hardware configurations. We instrumented FUSE to extract useful statistics and traces, which helped us analyze its performance bottlenecks and present our analysis results. Our experiments indicate that depending on the workload and hardware used, performance degradation (throughput) caused by FUSE can be completely imperceptible or as high as −83%, even when optimized; and latencies of FUSE file system operations can be increased from none to 4× when compared to Ext4. On the resource utilization side, FUSE can increase relative CPU utilization by up to 31% and underutilize disk bandwidth by as much as −80% compared to Ext4, though for many data-intensive workloads the impact was statistically indistinguishable. Our conclusion is that user-space file systems can indeed be used in production (non-“toy”) settings, but their applicability depends on the expected workloads.
- M. Accetta, R. Baron, W. Bolosky, D. Golub, R. Rashid, A. Tevanian, and M. Young. 1986. Mach: A new kernel foundation for UNIX development. In Proceedings of the Summer USENIX Technical Conference. USENIX Association, 93--112.Google Scholar
- Magnus Ahltorp, Love Hornquist-Astrand, and Assar Westerlund. 2000. Porting the arla file system to windows NT (2000). In Proceedings of Management and Administration of Distributed Environments Conference (MADE).Google Scholar
- A. D. Alexandrov, M. Ibel, K. E. Schauser, and C. J. Scheiman. 1997. Extending the operating system at the user level: The ufo global file system. In Proceedings of the Annual USENIX Technical Conference. USENIX Association, 77--90. Google Scholar
Digital Library
- Apache Foundation. 2010. Hadoop. http://hadoop.apache.org.Google Scholar
- AT8T Bell Laboratories 1995. Plan 9 -- Programmer’s Manual. AT8T Bell Laboratories.Google Scholar
- Avfs. 2019. AVFS: A Virtual Filesystem. http://avf.sourceforge.net/.Google Scholar
- Jens Axboe. 2009. Per Backing Device Write Back. https://linuxplumbersconf.org/2009/slides/Jens-Axboe-lpc2009-slides-axboe.pdf.Google Scholar
- J. Bent, G. Gibson, G. Grider, B. McClelland, P. Nowoczynski, J. Nunez, M. Polte, and M. Wingate. 2009. PLFS: A Checkpoint Filesystem for Parallel Applications. Technical Report LA-UR 09-02117. LANL. http://institute.lanl.gov/plfs/.Google Scholar
Digital Library
- B. Callaghan, B. Pawlowski, and P. Staubach. 1995. NFS Version 3 Protocol Specification. RFC 1813. Network Working Group. Google Scholar
Digital Library
- Claudio Calvelli. 2019. PerlFS. http://perlfs.sourceforge.net/.Google Scholar
- Zhen Cao, Vasily Tarasov, Hari Raman, Dean Hildebrand, and Erez Zadok. 2017. On the performance variation in modern storage stacks. In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST'17). USENIX Association, 329--343. Google Scholar
Digital Library
- R. Card, T. Ts’o, and S. Tweedie. 1994. Design and implementation of the second extended filesystem. In Proceedings to the 1st Dutch International Symposium on Linux.Google Scholar
- Mark Carlson, Alan Yoder, Leah Schoeb, Don Deel, Carlos Pratt, Chris Lionetti, and Doug Voigt. 2015. SNIA - Advanced Storage and Information Technology: Software Defined Storage. http://datastorageasean.com/sites/default/files/snia_software_defined_storage_white_paper_v1.pdf.Google Scholar
- Ming Chen, Dean Hildebrand, Henry Nelson, Jasmit Saluja, Ashok Subramony, and Erez Zadok. 2017. vNFS: Maximizing NFS performance with compounds and vectorized I/O. In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST'17). USENIX Association, 301--314. Google Scholar
Digital Library
- Michael Condict, Don Bolinger, Dave Mitchell, and Eamonn McManus. 1994. Microkernel modularity with integrated kernel performance. In Proceedings of the 1st Symposium on Operating Systems Design and Implementation (OSDI’94).Google Scholar
- Jonathan Corbet. 2009. In defense of per-BDI writeback. http://lwn.net/Articles/354851/.Google Scholar
- B. Cornell, P. A. Dinda, and F. E. Bustamante. 2004. Wayback: A user-level versioning file system for Linux. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track. USENIX Association, 19--28. Google Scholar
Digital Library
- Mathieu Desnoyers. 2016. Using the Linux Kernel Tracepoints. https://www.kernel.org/doc/Documentation/trace/tracepoints.txt.Google Scholar
- ext4-doc. 2019. Ext4 Documentation. https://www.kernel.org/doc/Documentation/filesystems/ext4.txt.Google Scholar
- Filebench. 2016. Filebench. https://github.com/filebench/filebench/wiki.Google Scholar
- Jeremy Fitzhardinge. 2019. Userfs. www.goop.org/∼jeremy/userfs/.Google Scholar
- fuse-appendix. 2018. FUSE Library Options and APIs. http://www.fsl.cs.stonybrook.edu/docs/fuse/fuse-article-appendices.html.Google Scholar
- FUSE BDI Max Ratio. 2019. FUSE - BDI Max Ratio Discussion. https://sourceforge.net/p/fuse/mailman/message/35192764/.Google Scholar
- FUSE Destroy message. 2019. FUSE - Destroy message discussion. https://sourceforge.net/p/fuse/mailman/message/27787354/.Google Scholar
- FUSE forgets queue. 2019. FUSE - Forgets queue. https://sourceforge.net/p/fuse/mailman/message/26659508/.Google Scholar
- S. Ghemawat, H. Gobioff, and S. T. Leung. 2003. The Google file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP’03). ACM SIGOPS, 29--43. Google Scholar
Digital Library
- Hermann Hartig, Michael Hohmuth, Jochen Liedtke, Jean Wolter, and Sebastian Schonberg. 1997. The performance of microkernel-based systems. In Proceedings of the 16th Symposium on Operating Systems Principles (SOSP’97). ACM. Google Scholar
Digital Library
- V. Henson, A. Ven, A. Gud, and Z. Brown. 2006. Chunkfs: Using divide-and-conquer to improve file system reliability and repair. In Proceedings of the 2nd Workshop on Hot Topics in System Dependability (HotDep’06). ACM SIGOPS. Google Scholar
Digital Library
- Hurd. 2019. GNU Hurd. www.gnu.org/software/hurd/hurd.html.Google Scholar
- Shun Ishiguro, Jun Murakami, Yoshihiro Oyama, and Osamu Tatebe. 2012. Optimizing local file accesses for FUSE-based distributed storage. In High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion. IEEE, 760--765. Google Scholar
Digital Library
- N. Joukov, A. Traeger, R. Iyer, C. P. Wright, and E. Zadok. 2006. Operating system profiling via latency analysis. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI’06). ACM SIGOPS, 89--102. Google Scholar
Digital Library
- lessfs. 2012. Lessfs. www.lessfs.com.Google Scholar
- Linus Fuse. 2019. Linus Torvalds doesn’t understand user-space filesystems. http://redhatstorage.redhat.com/2011/06/28/linus-torvalds-doesnt-understand-user-space-storage/.Google Scholar
- macfuse. 2013. FUSE for macOS. https://osxfuse.github.io/.Google Scholar
- Pavel Machek. 2019. PODFUK—POrtable Dodgy Filesystems in Userland (hacK). http://atrey.karlin.mff.cuni.cz/machek/podfuk/podfuk-old.html.Google Scholar
- Pavel Machek. 2019. UserVFS. http://sourceforge.net/projects/uservfs/.Google Scholar
- M. K. McKusick, W. N. Joy, S. J. Leffler, and R. S. Fabry. 1984. A fast file system for UNIX. ACM Transactions on Computer Systems 2, 3 (August 1984), 181--197. Google Scholar
Digital Library
- Sebastian Messmer. 2015. CryFS: Secure encrypted cloud file system. https://www.cryfs.org/howitworks.Google Scholar
- Sumit Narayan, Rohit K. Mehta, and John A. Chandy. 2010. User space storage system stack modules with file level control. In Proceedings of the 12th Annual Linux Symposium in Ottawa. 189--196.Google Scholar
- NetBSD Foundation. 2013. The Anykernel and Rump Kernels. http://wiki.netbsd.org/rumpkernel/.Google Scholar
- nimble_casl. 2019. Nimble’s Hybrid Storage Architecture. http://info.nimblestorage.com/rs/nimblestorage/images/nimblestorage_technology_overview.pdf.Google Scholar
- ntfs3g. 2019. NTFS-3G. www.tuxera.com.Google Scholar
- David Pease, Arnon Amir, Lucas Villa Real, Brian Biskeborn, Michael Richmond, and Atsushi Abe. 2010. The linear tape file system. In Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST’10). Google Scholar
Digital Library
- Aditya Rajgarhia and Ashish Gehani. 2010. Performance and extension of user space file systems. In Proceedings of the 25th Symposium on Applied Computing. ACM. Google Scholar
Digital Library
- Nikolaus Rath. 2013. S3QL: File system that stores all its data online using storage services like Amazon S3, Google Storage or Open Stack. https://github.com/s3ql/s3ql.Google Scholar
- RedHat’s GlusterFS. 2019. GlusterFS. http://www.gluster.org/.Google Scholar
- F. Schmuck and R. Haskin. 2002. GPFS: A shared-disk file system for large computing clusters. In Proceedings of the 1st USENIX Conference on File and Storage Technologies (FAST’02). USENIX Association, 231--244. Google Scholar
Digital Library
- sdfs. 2012. Opendedup. www.opendedup.org.Google Scholar
- D. Steere, J. Kistler, and M. Satyanarayanan. 1990. Efficient user-level file cache management on the sun vnode interface. In Proceedings of the Summer USENIX Technical Conference. IEEE.Google Scholar
- Swaminathan Sundararaman, Laxman Visampalli, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2011. Refuse to crash with Re-FUSE. In Proceedings of the 6th Conference on Computer Systems. ACM, 77--90. Google Scholar
Digital Library
- A. Sweeney, D. Doucette, W. Hu, C. Anderson, M. Nishimoto, and G. Peck. 1996. Scalability in the XFS file system. In Proceedings of the Annual USENIX Technical Conference. 1--14. Google Scholar
Digital Library
- M. Szeredi. 2005. Filesystem in Userspace. (February 2005). http://fuse.sourceforge.net.Google Scholar
- Vasily Tarasov, Abhishek Gupta, Kumar Sourav, Sagar Trehan, and Erez Zadok. 2015. Terra incognita: On the practicality of user-space file systems. In HotStorage’15: Proceedings of the 7th USENIX Workshop on Hot Topics in Storage. USENIX. Google Scholar
Digital Library
- Vasily Tarasov, Erez Zadok, and Spencer Shepler. 2016. Filebench: A flexible framework for file system benchmarking. The USENIX Magazine 41, 1 (March 2016), 6--12.Google Scholar
- The Linux Kernel Page Cache. 2019. The Linux Kernel Page Cache. http://sylab-srv.cs.fiu.edu/lib/exe/fetch.php?media=paperclub:lkd3ch16.pdf.Google Scholar
- L. Torvalds. 2007. splice(). Kernel Trap. http://kerneltrap.org/node/6505.Google Scholar
- Linux Torwalds. 2019. Re: {PATCH 0/7} overlay filesystem: request for inclusion. https://lkml.org/lkml/2011/6/9/462.Google Scholar
- C. Ungureanu, B. Atkin, A. Aranya, S. Gokhale, S. Rago, G. Calkowski, C. Dubnicki, and A. Bohra. 2010. HydraFS: A high-throughput file system for the HYDRAstor content-addressable storage system. In Proceedings of the FAST Conference. Google Scholar
Digital Library
- Sage Weil. 2019. Linus vs FUSE. http://ceph.com/dev-notes/linus-vs-fuse/.Google Scholar
- Assar Westerlund and Johan Danielsson. 1998. Arla-a free AFS client. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track. USENIX Association. Google Scholar
Digital Library
- E. Zadok and J. Nieh. 2000. FiST: A language for stackable file systems. In Proceedings of the Annual USENIX Technical Conference. USENIX Association, 55--70. Google Scholar
Digital Library
- zfslinux. 2016. ZFS for Linux. www.zfs-fuse.net.Google Scholar
- B. Zhu, K. Li, and H. Patterson. 2008. Avoiding the disk bottleneck in the data domain deduplication file system. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST’08). Google Scholar
Digital Library
Index Terms
Performance and Resource Utilization of FUSE User-Space File Systems
Recommendations
Direct-FUSE: Removing the Middleman for High-Performance FUSE File System Support
ROSS'18: Proceedings of the 8th International Workshop on Runtime and Operating Systems for SupercomputersDeveloping a file system is a challenging task, especially a kernel-level file system. User-level file systems alleviate the burden and development complexity associated with kernel-level implementations. The Filesystem in Userspace (FUSE) is a widely ...
Performance and protection in the ZoFS user-space NVM file system
SOSP '19: Proceedings of the 27th ACM Symposium on Operating Systems PrinciplesNon-volatile memory (NVM) can be directly accessed in user space without going through the kernel. This encourages several recent studies on building user-space NVM file systems. However, for the sake of file system protection, none of the existing file ...






Comments