skip to main content
research-article
Public Access

Performance and Resource Utilization of FUSE User-Space File Systems

Published:08 May 2019Publication History
Skip Abstract Section

Abstract

Traditionally, file systems were implemented as part of operating systems kernels, which provide a limited set of tools and facilities to a programmer. As the complexity of file systems grew, many new file systems began being developed in user space. Low performance is considered the main disadvantage of user-space file systems but the extent of this problem has never been explored systematically. As a result, the topic of user-space file systems remains rather controversial: while some consider user-space file systems a “toy” not to be used in production, others develop full-fledged production file systems in user space. In this article, we analyze the design and implementation of a well-known user-space file system framework, FUSE, for Linux. We characterize its performance and resource utilization for a wide range of workloads. We present FUSE performance and also resource utilization with various mount and configuration options, using 45 different workloads that were generated using Filebench on two different hardware configurations. We instrumented FUSE to extract useful statistics and traces, which helped us analyze its performance bottlenecks and present our analysis results. Our experiments indicate that depending on the workload and hardware used, performance degradation (throughput) caused by FUSE can be completely imperceptible or as high as −83%, even when optimized; and latencies of FUSE file system operations can be increased from none to 4× when compared to Ext4. On the resource utilization side, FUSE can increase relative CPU utilization by up to 31% and underutilize disk bandwidth by as much as −80% compared to Ext4, though for many data-intensive workloads the impact was statistically indistinguishable. Our conclusion is that user-space file systems can indeed be used in production (non-“toy”) settings, but their applicability depends on the expected workloads.

References

  1. M. Accetta, R. Baron, W. Bolosky, D. Golub, R. Rashid, A. Tevanian, and M. Young. 1986. Mach: A new kernel foundation for UNIX development. In Proceedings of the Summer USENIX Technical Conference. USENIX Association, 93--112.Google ScholarGoogle Scholar
  2. Magnus Ahltorp, Love Hornquist-Astrand, and Assar Westerlund. 2000. Porting the arla file system to windows NT (2000). In Proceedings of Management and Administration of Distributed Environments Conference (MADE).Google ScholarGoogle Scholar
  3. A. D. Alexandrov, M. Ibel, K. E. Schauser, and C. J. Scheiman. 1997. Extending the operating system at the user level: The ufo global file system. In Proceedings of the Annual USENIX Technical Conference. USENIX Association, 77--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Apache Foundation. 2010. Hadoop. http://hadoop.apache.org.Google ScholarGoogle Scholar
  5. AT8T Bell Laboratories 1995. Plan 9 -- Programmer’s Manual. AT8T Bell Laboratories.Google ScholarGoogle Scholar
  6. Avfs. 2019. AVFS: A Virtual Filesystem. http://avf.sourceforge.net/.Google ScholarGoogle Scholar
  7. Jens Axboe. 2009. Per Backing Device Write Back. https://linuxplumbersconf.org/2009/slides/Jens-Axboe-lpc2009-slides-axboe.pdf.Google ScholarGoogle Scholar
  8. J. Bent, G. Gibson, G. Grider, B. McClelland, P. Nowoczynski, J. Nunez, M. Polte, and M. Wingate. 2009. PLFS: A Checkpoint Filesystem for Parallel Applications. Technical Report LA-UR 09-02117. LANL. http://institute.lanl.gov/plfs/.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. B. Callaghan, B. Pawlowski, and P. Staubach. 1995. NFS Version 3 Protocol Specification. RFC 1813. Network Working Group. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Claudio Calvelli. 2019. PerlFS. http://perlfs.sourceforge.net/.Google ScholarGoogle Scholar
  11. Zhen Cao, Vasily Tarasov, Hari Raman, Dean Hildebrand, and Erez Zadok. 2017. On the performance variation in modern storage stacks. In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST'17). USENIX Association, 329--343. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. R. Card, T. Ts’o, and S. Tweedie. 1994. Design and implementation of the second extended filesystem. In Proceedings to the 1st Dutch International Symposium on Linux.Google ScholarGoogle Scholar
  13. Mark Carlson, Alan Yoder, Leah Schoeb, Don Deel, Carlos Pratt, Chris Lionetti, and Doug Voigt. 2015. SNIA - Advanced Storage and Information Technology: Software Defined Storage. http://datastorageasean.com/sites/default/files/snia_software_defined_storage_white_paper_v1.pdf.Google ScholarGoogle Scholar
  14. Ming Chen, Dean Hildebrand, Henry Nelson, Jasmit Saluja, Ashok Subramony, and Erez Zadok. 2017. vNFS: Maximizing NFS performance with compounds and vectorized I/O. In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST'17). USENIX Association, 301--314. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Michael Condict, Don Bolinger, Dave Mitchell, and Eamonn McManus. 1994. Microkernel modularity with integrated kernel performance. In Proceedings of the 1st Symposium on Operating Systems Design and Implementation (OSDI’94).Google ScholarGoogle Scholar
  16. Jonathan Corbet. 2009. In defense of per-BDI writeback. http://lwn.net/Articles/354851/.Google ScholarGoogle Scholar
  17. B. Cornell, P. A. Dinda, and F. E. Bustamante. 2004. Wayback: A user-level versioning file system for Linux. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track. USENIX Association, 19--28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Mathieu Desnoyers. 2016. Using the Linux Kernel Tracepoints. https://www.kernel.org/doc/Documentation/trace/tracepoints.txt.Google ScholarGoogle Scholar
  19. ext4-doc. 2019. Ext4 Documentation. https://www.kernel.org/doc/Documentation/filesystems/ext4.txt.Google ScholarGoogle Scholar
  20. Filebench. 2016. Filebench. https://github.com/filebench/filebench/wiki.Google ScholarGoogle Scholar
  21. Jeremy Fitzhardinge. 2019. Userfs. www.goop.org/∼jeremy/userfs/.Google ScholarGoogle Scholar
  22. fuse-appendix. 2018. FUSE Library Options and APIs. http://www.fsl.cs.stonybrook.edu/docs/fuse/fuse-article-appendices.html.Google ScholarGoogle Scholar
  23. FUSE BDI Max Ratio. 2019. FUSE - BDI Max Ratio Discussion. https://sourceforge.net/p/fuse/mailman/message/35192764/.Google ScholarGoogle Scholar
  24. FUSE Destroy message. 2019. FUSE - Destroy message discussion. https://sourceforge.net/p/fuse/mailman/message/27787354/.Google ScholarGoogle Scholar
  25. FUSE forgets queue. 2019. FUSE - Forgets queue. https://sourceforge.net/p/fuse/mailman/message/26659508/.Google ScholarGoogle Scholar
  26. S. Ghemawat, H. Gobioff, and S. T. Leung. 2003. The Google file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP’03). ACM SIGOPS, 29--43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Hermann Hartig, Michael Hohmuth, Jochen Liedtke, Jean Wolter, and Sebastian Schonberg. 1997. The performance of microkernel-based systems. In Proceedings of the 16th Symposium on Operating Systems Principles (SOSP’97). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. V. Henson, A. Ven, A. Gud, and Z. Brown. 2006. Chunkfs: Using divide-and-conquer to improve file system reliability and repair. In Proceedings of the 2nd Workshop on Hot Topics in System Dependability (HotDep’06). ACM SIGOPS. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Hurd. 2019. GNU Hurd. www.gnu.org/software/hurd/hurd.html.Google ScholarGoogle Scholar
  30. Shun Ishiguro, Jun Murakami, Yoshihiro Oyama, and Osamu Tatebe. 2012. Optimizing local file accesses for FUSE-based distributed storage. In High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion. IEEE, 760--765. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. N. Joukov, A. Traeger, R. Iyer, C. P. Wright, and E. Zadok. 2006. Operating system profiling via latency analysis. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI’06). ACM SIGOPS, 89--102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. lessfs. 2012. Lessfs. www.lessfs.com.Google ScholarGoogle Scholar
  33. Linus Fuse. 2019. Linus Torvalds doesn’t understand user-space filesystems. http://redhatstorage.redhat.com/2011/06/28/linus-torvalds-doesnt-understand-user-space-storage/.Google ScholarGoogle Scholar
  34. macfuse. 2013. FUSE for macOS. https://osxfuse.github.io/.Google ScholarGoogle Scholar
  35. Pavel Machek. 2019. PODFUK—POrtable Dodgy Filesystems in Userland (hacK). http://atrey.karlin.mff.cuni.cz/machek/podfuk/podfuk-old.html.Google ScholarGoogle Scholar
  36. Pavel Machek. 2019. UserVFS. http://sourceforge.net/projects/uservfs/.Google ScholarGoogle Scholar
  37. M. K. McKusick, W. N. Joy, S. J. Leffler, and R. S. Fabry. 1984. A fast file system for UNIX. ACM Transactions on Computer Systems 2, 3 (August 1984), 181--197. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Sebastian Messmer. 2015. CryFS: Secure encrypted cloud file system. https://www.cryfs.org/howitworks.Google ScholarGoogle Scholar
  39. Sumit Narayan, Rohit K. Mehta, and John A. Chandy. 2010. User space storage system stack modules with file level control. In Proceedings of the 12th Annual Linux Symposium in Ottawa. 189--196.Google ScholarGoogle Scholar
  40. NetBSD Foundation. 2013. The Anykernel and Rump Kernels. http://wiki.netbsd.org/rumpkernel/.Google ScholarGoogle Scholar
  41. nimble_casl. 2019. Nimble’s Hybrid Storage Architecture. http://info.nimblestorage.com/rs/nimblestorage/images/nimblestorage_technology_overview.pdf.Google ScholarGoogle Scholar
  42. ntfs3g. 2019. NTFS-3G. www.tuxera.com.Google ScholarGoogle Scholar
  43. David Pease, Arnon Amir, Lucas Villa Real, Brian Biskeborn, Michael Richmond, and Atsushi Abe. 2010. The linear tape file system. In Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST’10). Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Aditya Rajgarhia and Ashish Gehani. 2010. Performance and extension of user space file systems. In Proceedings of the 25th Symposium on Applied Computing. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Nikolaus Rath. 2013. S3QL: File system that stores all its data online using storage services like Amazon S3, Google Storage or Open Stack. https://github.com/s3ql/s3ql.Google ScholarGoogle Scholar
  46. RedHat’s GlusterFS. 2019. GlusterFS. http://www.gluster.org/.Google ScholarGoogle Scholar
  47. F. Schmuck and R. Haskin. 2002. GPFS: A shared-disk file system for large computing clusters. In Proceedings of the 1st USENIX Conference on File and Storage Technologies (FAST’02). USENIX Association, 231--244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. sdfs. 2012. Opendedup. www.opendedup.org.Google ScholarGoogle Scholar
  49. D. Steere, J. Kistler, and M. Satyanarayanan. 1990. Efficient user-level file cache management on the sun vnode interface. In Proceedings of the Summer USENIX Technical Conference. IEEE.Google ScholarGoogle Scholar
  50. Swaminathan Sundararaman, Laxman Visampalli, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2011. Refuse to crash with Re-FUSE. In Proceedings of the 6th Conference on Computer Systems. ACM, 77--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. A. Sweeney, D. Doucette, W. Hu, C. Anderson, M. Nishimoto, and G. Peck. 1996. Scalability in the XFS file system. In Proceedings of the Annual USENIX Technical Conference. 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. M. Szeredi. 2005. Filesystem in Userspace. (February 2005). http://fuse.sourceforge.net.Google ScholarGoogle Scholar
  53. Vasily Tarasov, Abhishek Gupta, Kumar Sourav, Sagar Trehan, and Erez Zadok. 2015. Terra incognita: On the practicality of user-space file systems. In HotStorage’15: Proceedings of the 7th USENIX Workshop on Hot Topics in Storage. USENIX. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Vasily Tarasov, Erez Zadok, and Spencer Shepler. 2016. Filebench: A flexible framework for file system benchmarking. The USENIX Magazine 41, 1 (March 2016), 6--12.Google ScholarGoogle Scholar
  55. The Linux Kernel Page Cache. 2019. The Linux Kernel Page Cache. http://sylab-srv.cs.fiu.edu/lib/exe/fetch.php?media=paperclub:lkd3ch16.pdf.Google ScholarGoogle Scholar
  56. L. Torvalds. 2007. splice(). Kernel Trap. http://kerneltrap.org/node/6505.Google ScholarGoogle Scholar
  57. Linux Torwalds. 2019. Re: {PATCH 0/7} overlay filesystem: request for inclusion. https://lkml.org/lkml/2011/6/9/462.Google ScholarGoogle Scholar
  58. C. Ungureanu, B. Atkin, A. Aranya, S. Gokhale, S. Rago, G. Calkowski, C. Dubnicki, and A. Bohra. 2010. HydraFS: A high-throughput file system for the HYDRAstor content-addressable storage system. In Proceedings of the FAST Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Sage Weil. 2019. Linus vs FUSE. http://ceph.com/dev-notes/linus-vs-fuse/.Google ScholarGoogle Scholar
  60. Assar Westerlund and Johan Danielsson. 1998. Arla-a free AFS client. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. E. Zadok and J. Nieh. 2000. FiST: A language for stackable file systems. In Proceedings of the Annual USENIX Technical Conference. USENIX Association, 55--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. zfslinux. 2016. ZFS for Linux. www.zfs-fuse.net.Google ScholarGoogle Scholar
  63. B. Zhu, K. Li, and H. Patterson. 2008. Avoiding the disk bottleneck in the data domain deduplication file system. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST’08). Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Performance and Resource Utilization of FUSE User-Space File Systems

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on Storage
            ACM Transactions on Storage  Volume 15, Issue 2
            Systor 2018 Special Section on ATC 2018, Special Section on OSDI 2018 and Regular Papers
            May 2019
            187 pages
            ISSN:1553-3077
            EISSN:1553-3093
            DOI:10.1145/3326597
            • Editor:
            • Sam H. Noh
            Issue’s Table of Contents

            Copyright © 2019 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 8 May 2019
            • Accepted: 1 January 2019
            • Revised: 1 December 2018
            • Received: 1 January 2018
            Published in tos Volume 15, Issue 2

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Research
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!