skip to main content
research-article
Public Access

vNFS: Maximizing NFS Performance with Compounds and Vectorized I/O

Published:28 September 2017Publication History
Skip Abstract Section

Abstract

Modern systems use networks extensively, accessing both services and storage across local and remote networks. Latency is a key performance challenge, and packing multiple small operations into fewer large ones is an effective way to amortize that cost, especially after years of significant improvement in bandwidth but not latency. To this end, the NFSv4 protocol supports a compounding feature to combine multiple operations. Yet compounding has been underused since its conception because the synchronous POSIX file-system API issues only one (small) request at a time.

We propose vNFS, an NFSv4.1-compliant client that exposes a vectorized high-level API and leverages NFS compound procedures to maximize performance. We designed and implemented vNFS as a user-space RPC library that supports an assortment of bulk operations on multiple files and directories. We found it easy to modify several UNIX utilities, an HTTP/2 server, and Filebench to use vNFS. We evaluated vNFS under a wide range of workloads and network latency conditions, showing that vNFS improves performance even for low-latency networks. On high-latency networks, vNFS can improve performance by as much as two orders of magnitude.

References

  1. HTTP Archive. 2016. URL Statistics. Retrieved from http://httparchive.org/trends.php.Google ScholarGoogle Scholar
  2. Vaggelis Atlidakis, Jeremy Andrus, Roxana Geambasu, Dimitris Mitropoulos, and Jason Nieh. 2016. POSIX abstractions in modern operating systems: The old, the new, and the missing. In Proceedings of the 11th European Conference on Computer Systems. ACM, 19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Jens Axboe. 2007. CFQ IO Scheduler. Retrieved from http://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks/123.ogg.Google ScholarGoogle Scholar
  4. Alexandros Batsakis, Randal Burns, Arkady Kanevsky, James Lentini, and Thomas Talpey. 2009. CA-NFS: A congestion-aware network file system. ACM Transactions on Storage 5, 4 (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. M. Belshe, R. Peon, and M. Thomson. 2015. Hypertext Transfer Protocol Version 2 (HTTP/2). RFC 7540. Internet Engineering Task Force. Google ScholarGoogle ScholarCross RefCross Ref
  6. John Bent, Douglas Thain, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Miron Livny. 2004. Explicit control in the batch-aware distributed file system. In The 1st USENIX Symposium on Networked Systems Design and Implementation (NSDI’04). USENIX Association, 365--378.Google ScholarGoogle Scholar
  7. Werner Beroux. 2016. Rename-It! Retrieved from https://github.com/wernight/renameit.Google ScholarGoogle Scholar
  8. B. Callaghan, B. Pawlowski, and P. Staubach. 1995. NFS Version 3 Protocol Specification. RFC 1813. Network Working Group.Google ScholarGoogle Scholar
  9. Ming Chen, Dean Hildebrand, Geoff Kuenning, Soujanya Shankaranarayana, Bharat Singh, and Erez Zadok. 2015. Newer is sometimes better: An evaluation of NFSv4.1. In Proceedings of the 2015 ACM International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS 2015). ACM, Portland, OR. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Ming Chen, Dean Hildebrand, Henry Nelson, Jasmit Saluja, Ashok Subramony, and Erez Zadok. 2017. vNFS: Maximizing NFS performance with compounds and vectorized I/O. In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST). USENIX Association, Santa Clara, CA, 301--314. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Ming Chen, Arun Vasudevan, Kelong Wang, and Erez Zadok. 2016. SeMiNAS: A secure middleware for wide-area network-attached storage. In Proceedings of the 9th ACM International Systems and Storage Conference (ACM SYSTOR’16). ACM, Haifa, Israel. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Stuart Cheshire. 2005. TCP Performance problems caused by interaction between Nagle’s Algorithm and Delayed ACK. Retrieved from http://www.stuartcheshire.org/papers/nagledelayedack.Google ScholarGoogle Scholar
  13. Philippe Deniel, Thomas Leibovici, and Jacques-Charles Lafoucrière. 2007. GANESHA, a multi-usage with large cache NFSv4 server. In Linux Symposium. USENIX Association, 113.Google ScholarGoogle Scholar
  14. Dan Duchamp. 1994. Optimistic lookup of whole NFS paths in a single operation. In Proceedings of the Summer 1994 USENIX Technical Conference. Boston, MA, 143--170.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Daniel Ellard and Margo Seltzer. 2003. NFS tricks and benchmarking traps. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track. USENIX Association, San Antonio, TX, 101--114.Google ScholarGoogle Scholar
  16. Filebench. 2016. Filebench. Retrieved from https://github.com/filebench/filebench/wiki.Google ScholarGoogle Scholar
  17. Jason Fitzpatrick. 2016. Bulk Rename Utility. Retrieved from http://www.bulkrenameutility.co.uk/Main_Intro.php.Google ScholarGoogle Scholar
  18. M. Haardt and M. Coleman. 1999. ftw(3). Linux Programmer’s Manual, Section 3. http://man7.org/linux/man-pages/man3/ftw.3.html.Google ScholarGoogle Scholar
  19. Sangjin Han, Scott Marshall, Byung-Gon Chun, and Sylvia Ratnasamy. 2012. MegaPipe: A new programming interface for scalable network I/O. In Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI 12).Google ScholarGoogle Scholar
  20. T. Haynes. 2015. NFS Version 4 Minor Version 2 Protocol. RFC Draft. Network Working Group. Retrieved from https://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-39.Google ScholarGoogle Scholar
  21. Dean Hildebrand and Peter Honeyman. 2005. Exporting storage systems in a scalable manner with pNFS. In Proceedings of the 22nd IEEE/13th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST). IEEE, Monterey, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Kim Jensen. 2016. AdvancedRenamer. Retrieved from https://www.advancedrenamer.com/.Google ScholarGoogle Scholar
  23. Chet Juszczak. 1994. Improving the write performance of an NFS server. In Proceedings of the USENIX Winter 1994 Technical Conference (WTEC’94). USENIX Association, San Francisco, California, 1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Chuck Lever. Close-To-Open Cache Consistency in the Linux NFS Client. Retrieved from http://goo.gl/o9i0MM.Google ScholarGoogle Scholar
  25. Linux Programmer’s Manual, Section 2. Open(2) - Open and Possibly Create a File or Device. Linux Programmer’s Manual, Section 2. Retrieved from http://linux.die.net/man/2/open.Google ScholarGoogle Scholar
  26. Linux Programmer’s Manual, Section 3 2016. Lio_listio(3) - Initiate a List of I/O Requests. Linux Programmer’s Manual, Section 3. Retrieved from http://man7.org/linux/man-pages/man3/lio_listio.3.html.Google ScholarGoogle Scholar
  27. Alex McDonald. 2012. The background to NFSv4.1. Login: The USENIX Magazine 37, 1 (February 2012), 28--35.Google ScholarGoogle Scholar
  28. Sun Microsystems. 1989. NFS: Network File System Protocol Specification. RFC 1094. Network Working Group.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. J. Nagle. 1984. Congestion Control in IP/TCP Internetworks. RFC 896. Network Working Group.Google ScholarGoogle Scholar
  30. NFS-Ganesha. 2016. NFS-Ganesha. Retrieved from http://nfs-ganesha.github.io/.Google ScholarGoogle Scholar
  31. nghttp2. 2016. nghttp2: HTTP/2 C Library. Retrieved from http://nghttp2.org.Google ScholarGoogle Scholar
  32. Poco. 2017. Poco C++ Libraries: The Cache Framework. Retrieved from https://pocoproject.org/slides/140-Cache.pdf.Google ScholarGoogle Scholar
  33. Antoine Potten. 2016. Ant Renamer. Retrieved from http://www.antp.be/software/renamer.Google ScholarGoogle Scholar
  34. Amit Purohit, Charles P. Wright, Joseph Spadavecchia, and Erez Zadok. 2003. Cosy: Develop in user-land, run in kernel-mode. In Proceedings of the 2003 ACM Workshop on Hot Topics in Operating Systems (HotOS IX). USENIX Association, Lihue, Hawaii, 109--114.Google ScholarGoogle Scholar
  35. Stephen M. Rumble, Diego Ongaro, Ryan Stutsman, Mendel Rosenblum, and John K. Ousterhout. 2011. It’s time for low latency. In Proceedings of the 13th USENIX Conference on Hot Topics in Operating Systems.Google ScholarGoogle Scholar
  36. S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, and D. Noveck. 2003. NFS Version 4 Protocol. RFC 3530. Network Working Group.Google ScholarGoogle Scholar
  37. S. Shepler, M. Eisler, and D. Noveck. 2010. NFS Version 4 Minor Version 1 Protocol. RFC 5661. Network Working Group.Google ScholarGoogle Scholar
  38. SPEC. 2001. SPEC SFS97_R1 V3.0. Retrieved from www.spec.org/sfs97r1.Google ScholarGoogle Scholar
  39. David C. Steere. 1997. Exploiting the non-determinism and asynchrony of set iterators to reduce aggregate file I/O latency. In Proceedings of the 9th ACM Symposium on Operating Systems Principles (SOSP’97).Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Vasily Tarasov, Abhishek Gupta, Kumar Sourav, Sagar Trehan, and Erez Zadok. 2015. Terra incognita: On the practicality of user-space file systems. In HotStorage’15: Proceedings of the 7th USENIX Workshop on Hot Topics in Storage. USENIX, USENIX, Santa Clara, CA.Google ScholarGoogle Scholar
  41. Vasily Tarasov, Erez Zadok, and Spencer Shepler. 2016. Filebench: A flexible framework for file system benchmarking. Login: The USENIX Magazine 41, 1 (March 2016), 6--12.Google ScholarGoogle Scholar
  42. Chia-Che Tsai, Yang Zhan, Jayashree Reddy, Yizheng Jiao, Tao Zhang, and Donald E. Porter. 2015. How to get more value from your file system directory cache. In Proceedings of the 25th Symposium on Operating Systems Principles (SOSP’15). ACM, New York, 441--456. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Vijay Vasudevan, David G. Andersen, and Michael Kaminsky. 2011. The case for VOS: The vector operating system. In Proceedings of the 13th USENIX Conference on Hot Topics in Operating Systems (HotOS’13). USENIX Association, Berkeley, CA, 31--31.Google ScholarGoogle Scholar
  44. Vijay Vasudevan, Michael Kaminsky, and David G. Andersen. 2012. Using vector interfaces to deliver millions of IOPS from a networked key-value storage server. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC’12). Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. M. Vilayannur, S. Lang, R. Ross, R. Klundt, and L. Ward. 2008. Extending the POSIX I/O Interface: A Parallel File System Perspective. Technical Report ANL/MCS-TM-302. Argonne National Laboratory.Google ScholarGoogle Scholar
  46. M. Mitchell Waldrop. 2016. The chips are down for Moore’s law. Nature 530, 7589 (2016), 144--147. Google ScholarGoogle ScholarCross RefCross Ref
  47. WML. 2016. Filebench Workload Model Language (WML). Retrieved from https://github.com/filebench/filebench/wiki/Workload-Model-Language.Google ScholarGoogle Scholar
  48. Shuanglong Zhang, Helen Catanese, and An-I. Andy Wang. 2016. The composite-file file system: Decoupling the one-to-one mapping of files and metadata for better performance. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST). USENIX Association, Santa Clara, CA.Google ScholarGoogle Scholar

Index Terms

  1. vNFS: Maximizing NFS Performance with Compounds and Vectorized I/O

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Storage
        ACM Transactions on Storage  Volume 13, Issue 3
        Special Issue on FAST 2017 and Regular Papers
        August 2017
        265 pages
        ISSN:1553-3077
        EISSN:1553-3093
        DOI:10.1145/3141876
        • Editor:
        • Sam H. Noh
        Issue’s Table of Contents

        Copyright © 2017 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 28 September 2017
        • Received: 1 June 2017
        • Accepted: 1 June 2017
        Published in tos Volume 13, Issue 3

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!