Abstract
Modern systems use networks extensively, accessing both services and storage across local and remote networks. Latency is a key performance challenge, and packing multiple small operations into fewer large ones is an effective way to amortize that cost, especially after years of significant improvement in bandwidth but not latency. To this end, the NFSv4 protocol supports a compounding feature to combine multiple operations. Yet compounding has been underused since its conception because the synchronous POSIX file-system API issues only one (small) request at a time.
We propose vNFS, an NFSv4.1-compliant client that exposes a vectorized high-level API and leverages NFS compound procedures to maximize performance. We designed and implemented vNFS as a user-space RPC library that supports an assortment of bulk operations on multiple files and directories. We found it easy to modify several UNIX utilities, an HTTP/2 server, and Filebench to use vNFS. We evaluated vNFS under a wide range of workloads and network latency conditions, showing that vNFS improves performance even for low-latency networks. On high-latency networks, vNFS can improve performance by as much as two orders of magnitude.
- HTTP Archive. 2016. URL Statistics. Retrieved from http://httparchive.org/trends.php.Google Scholar
- Vaggelis Atlidakis, Jeremy Andrus, Roxana Geambasu, Dimitris Mitropoulos, and Jason Nieh. 2016. POSIX abstractions in modern operating systems: The old, the new, and the missing. In Proceedings of the 11th European Conference on Computer Systems. ACM, 19. Google Scholar
Digital Library
- Jens Axboe. 2007. CFQ IO Scheduler. Retrieved from http://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks/123.ogg.Google Scholar
- Alexandros Batsakis, Randal Burns, Arkady Kanevsky, James Lentini, and Thomas Talpey. 2009. CA-NFS: A congestion-aware network file system. ACM Transactions on Storage 5, 4 (2009). Google Scholar
Digital Library
- M. Belshe, R. Peon, and M. Thomson. 2015. Hypertext Transfer Protocol Version 2 (HTTP/2). RFC 7540. Internet Engineering Task Force. Google Scholar
Cross Ref
- John Bent, Douglas Thain, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Miron Livny. 2004. Explicit control in the batch-aware distributed file system. In The 1st USENIX Symposium on Networked Systems Design and Implementation (NSDI’04). USENIX Association, 365--378.Google Scholar
- Werner Beroux. 2016. Rename-It! Retrieved from https://github.com/wernight/renameit.Google Scholar
- B. Callaghan, B. Pawlowski, and P. Staubach. 1995. NFS Version 3 Protocol Specification. RFC 1813. Network Working Group.Google Scholar
- Ming Chen, Dean Hildebrand, Geoff Kuenning, Soujanya Shankaranarayana, Bharat Singh, and Erez Zadok. 2015. Newer is sometimes better: An evaluation of NFSv4.1. In Proceedings of the 2015 ACM International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS 2015). ACM, Portland, OR. Google Scholar
Digital Library
- Ming Chen, Dean Hildebrand, Henry Nelson, Jasmit Saluja, Ashok Subramony, and Erez Zadok. 2017. vNFS: Maximizing NFS performance with compounds and vectorized I/O. In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST). USENIX Association, Santa Clara, CA, 301--314. Google Scholar
Digital Library
- Ming Chen, Arun Vasudevan, Kelong Wang, and Erez Zadok. 2016. SeMiNAS: A secure middleware for wide-area network-attached storage. In Proceedings of the 9th ACM International Systems and Storage Conference (ACM SYSTOR’16). ACM, Haifa, Israel. Google Scholar
Digital Library
- Stuart Cheshire. 2005. TCP Performance problems caused by interaction between Nagle’s Algorithm and Delayed ACK. Retrieved from http://www.stuartcheshire.org/papers/nagledelayedack.Google Scholar
- Philippe Deniel, Thomas Leibovici, and Jacques-Charles Lafoucrière. 2007. GANESHA, a multi-usage with large cache NFSv4 server. In Linux Symposium. USENIX Association, 113.Google Scholar
- Dan Duchamp. 1994. Optimistic lookup of whole NFS paths in a single operation. In Proceedings of the Summer 1994 USENIX Technical Conference. Boston, MA, 143--170.Google Scholar
Digital Library
- Daniel Ellard and Margo Seltzer. 2003. NFS tricks and benchmarking traps. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track. USENIX Association, San Antonio, TX, 101--114.Google Scholar
- Filebench. 2016. Filebench. Retrieved from https://github.com/filebench/filebench/wiki.Google Scholar
- Jason Fitzpatrick. 2016. Bulk Rename Utility. Retrieved from http://www.bulkrenameutility.co.uk/Main_Intro.php.Google Scholar
- M. Haardt and M. Coleman. 1999. ftw(3). Linux Programmer’s Manual, Section 3. http://man7.org/linux/man-pages/man3/ftw.3.html.Google Scholar
- Sangjin Han, Scott Marshall, Byung-Gon Chun, and Sylvia Ratnasamy. 2012. MegaPipe: A new programming interface for scalable network I/O. In Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI 12).Google Scholar
- T. Haynes. 2015. NFS Version 4 Minor Version 2 Protocol. RFC Draft. Network Working Group. Retrieved from https://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-39.Google Scholar
- Dean Hildebrand and Peter Honeyman. 2005. Exporting storage systems in a scalable manner with pNFS. In Proceedings of the 22nd IEEE/13th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST). IEEE, Monterey, CA. Google Scholar
Digital Library
- Kim Jensen. 2016. AdvancedRenamer. Retrieved from https://www.advancedrenamer.com/.Google Scholar
- Chet Juszczak. 1994. Improving the write performance of an NFS server. In Proceedings of the USENIX Winter 1994 Technical Conference (WTEC’94). USENIX Association, San Francisco, California, 1.Google Scholar
Digital Library
- Chuck Lever. Close-To-Open Cache Consistency in the Linux NFS Client. Retrieved from http://goo.gl/o9i0MM.Google Scholar
- Linux Programmer’s Manual, Section 2. Open(2) - Open and Possibly Create a File or Device. Linux Programmer’s Manual, Section 2. Retrieved from http://linux.die.net/man/2/open.Google Scholar
- Linux Programmer’s Manual, Section 3 2016. Lio_listio(3) - Initiate a List of I/O Requests. Linux Programmer’s Manual, Section 3. Retrieved from http://man7.org/linux/man-pages/man3/lio_listio.3.html.Google Scholar
- Alex McDonald. 2012. The background to NFSv4.1. Login: The USENIX Magazine 37, 1 (February 2012), 28--35.Google Scholar
- Sun Microsystems. 1989. NFS: Network File System Protocol Specification. RFC 1094. Network Working Group.Google Scholar
Digital Library
- J. Nagle. 1984. Congestion Control in IP/TCP Internetworks. RFC 896. Network Working Group.Google Scholar
- NFS-Ganesha. 2016. NFS-Ganesha. Retrieved from http://nfs-ganesha.github.io/.Google Scholar
- nghttp2. 2016. nghttp2: HTTP/2 C Library. Retrieved from http://nghttp2.org.Google Scholar
- Poco. 2017. Poco C++ Libraries: The Cache Framework. Retrieved from https://pocoproject.org/slides/140-Cache.pdf.Google Scholar
- Antoine Potten. 2016. Ant Renamer. Retrieved from http://www.antp.be/software/renamer.Google Scholar
- Amit Purohit, Charles P. Wright, Joseph Spadavecchia, and Erez Zadok. 2003. Cosy: Develop in user-land, run in kernel-mode. In Proceedings of the 2003 ACM Workshop on Hot Topics in Operating Systems (HotOS IX). USENIX Association, Lihue, Hawaii, 109--114.Google Scholar
- Stephen M. Rumble, Diego Ongaro, Ryan Stutsman, Mendel Rosenblum, and John K. Ousterhout. 2011. It’s time for low latency. In Proceedings of the 13th USENIX Conference on Hot Topics in Operating Systems.Google Scholar
- S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, and D. Noveck. 2003. NFS Version 4 Protocol. RFC 3530. Network Working Group.Google Scholar
- S. Shepler, M. Eisler, and D. Noveck. 2010. NFS Version 4 Minor Version 1 Protocol. RFC 5661. Network Working Group.Google Scholar
- SPEC. 2001. SPEC SFS97_R1 V3.0. Retrieved from www.spec.org/sfs97r1.Google Scholar
- David C. Steere. 1997. Exploiting the non-determinism and asynchrony of set iterators to reduce aggregate file I/O latency. In Proceedings of the 9th ACM Symposium on Operating Systems Principles (SOSP’97).Google Scholar
Digital Library
- Vasily Tarasov, Abhishek Gupta, Kumar Sourav, Sagar Trehan, and Erez Zadok. 2015. Terra incognita: On the practicality of user-space file systems. In HotStorage’15: Proceedings of the 7th USENIX Workshop on Hot Topics in Storage. USENIX, USENIX, Santa Clara, CA.Google Scholar
- Vasily Tarasov, Erez Zadok, and Spencer Shepler. 2016. Filebench: A flexible framework for file system benchmarking. Login: The USENIX Magazine 41, 1 (March 2016), 6--12.Google Scholar
- Chia-Che Tsai, Yang Zhan, Jayashree Reddy, Yizheng Jiao, Tao Zhang, and Donald E. Porter. 2015. How to get more value from your file system directory cache. In Proceedings of the 25th Symposium on Operating Systems Principles (SOSP’15). ACM, New York, 441--456. Google Scholar
Digital Library
- Vijay Vasudevan, David G. Andersen, and Michael Kaminsky. 2011. The case for VOS: The vector operating system. In Proceedings of the 13th USENIX Conference on Hot Topics in Operating Systems (HotOS’13). USENIX Association, Berkeley, CA, 31--31.Google Scholar
- Vijay Vasudevan, Michael Kaminsky, and David G. Andersen. 2012. Using vector interfaces to deliver millions of IOPS from a networked key-value storage server. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC’12). Google Scholar
Digital Library
- M. Vilayannur, S. Lang, R. Ross, R. Klundt, and L. Ward. 2008. Extending the POSIX I/O Interface: A Parallel File System Perspective. Technical Report ANL/MCS-TM-302. Argonne National Laboratory.Google Scholar
- M. Mitchell Waldrop. 2016. The chips are down for Moore’s law. Nature 530, 7589 (2016), 144--147. Google Scholar
Cross Ref
- WML. 2016. Filebench Workload Model Language (WML). Retrieved from https://github.com/filebench/filebench/wiki/Workload-Model-Language.Google Scholar
- Shuanglong Zhang, Helen Catanese, and An-I. Andy Wang. 2016. The composite-file file system: Decoupling the one-to-one mapping of files and metadata for better performance. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST). USENIX Association, Santa Clara, CA.Google Scholar
Index Terms
vNFS: Maximizing NFS Performance with Compounds and Vectorized I/O
Recommendations
Implementation of a stackable file system for real-time network backup
We propose a backup system based on a stackable mirroring file system, general-purpose mirroring file system (GMFS). This file system mirrors data in real-time on the file system layer. It uses the typical network file system (NFS) and backs up data to ...
vNFS: maximizing NFS performance with compounds and vectorized I/O
FAST'17: Proceedings of the 15th Usenix Conference on File and Storage TechnologiesModern systems use networks extensively, accessing both services and storage across local and remote networks. Latency is a key performance challenge, and packing multiple small operations into fewer large ones is an effective way to amortize that cost, ...
pNFS/PVFS2 over InfiniBand: early experiences
PDSW '07: Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing '07The computing power of clusters has been rapidly growing up towards petascale capability, which requires petascale I/O systems to provide data in a sustained high-throughput manner. Network File System (NFS), a ubiquitous standard used in most existing ...






Comments