skip to main content
research-article

HiNFS: A Persistent Memory File System with Both Buffering and Direct-Access

Published:04 April 2018Publication History
Skip Abstract Section

Abstract

Persistent memory provides data persistence at main memory with emerging non-volatile main memories (NVMMs). Recent persistent memory file systems aggressively use direct access, which directly copy data between user buffer and the storage layer, to avoid the double-copy overheads through the OS page cache. However, we observe they all suffer from slow writes due to NVMMs’ asymmetric read-write performance and much slower performance than DRAM.

In this article, we propose HiNFS, a high-performance file system for non-volatile main memory, to combine both buffering and direct access for fine-grained file system operations. HiNFS uses an NVMM-aware Write Buffer to buffer the lazy-persistent file writes in DRAM, while performing direct access to NVMM for eager-persistent file writes. It directly reads file data from both DRAM and NVMM, by ensuring read consistency with a combination of the DRAM Block Index and Cacheline Bitmap to track the latest data between DRAM and NVMM. HiNFS also employs a Buffer Benefit Model to identify the eager-persistent file writes before issuing I/Os. Evaluations show that HiNFS significantly improves throughput by up to 184% and reduces execution time by up to 64%comparing with state-of-the-art persistent memory file systems PMFS and EXT4-DAX.

References

  1. Dbt2 test suite. Retrieved from http://sourceforge.net/apps/mediawiki/osdldbt.Google ScholarGoogle Scholar
  2. Filebench 1.4.9.1. Retrieved from http://sourceforge.net/projects/filebench/.Google ScholarGoogle Scholar
  3. FIU system call io trace. Retrieved from http://sylab-srv.cs.fiu.edu/dokuwiki/doku.php?id=projects:nbw:start.Google ScholarGoogle Scholar
  4. Flexible IO (fio) Tester. Retrieved from http://freecode.com/projects/fio.Google ScholarGoogle Scholar
  5. Lasr system call io trace. Retrieved from http://iotta.snia.org/historical_section?tracetype_id=1.Google ScholarGoogle Scholar
  6. 2014. Support ext4 on NV-DIMMs. Retrieved from http://lwn.net/Articles/588218/.Google ScholarGoogle Scholar
  7. 2014. Supporting file systems in persistent memory. Retrieved from https://lwn.net/Articles/610174/.Google ScholarGoogle Scholar
  8. Geoffrey W. Burr, Matthew J. Breitwisch, Michele Franceschini, Davide Garetto, Kailash Gopalakrishnan, Bryan Jackson, Bülent Kurdi, Chung Lam, Luis A. Lastras, Alvaro Padilla, Bipin Rajendran, Simone Raoux, and Rohit S. Shenoy. 2010. Phase change memory technology. J. Vacuum Sci. Technol. B 28, 2 (2010), 223--262.Google ScholarGoogle ScholarCross RefCross Ref
  9. Daniel Campello, Hector Lopez, Ricardo Koller, Raju Rangaswami, and Luis Useche. 2015. Non-blocking writes to files. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST’15). 151--165. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Shimin Chen, Phillip B. Gibbons, and Suman Nath. 2011. Rethinking database algorithms for phase change memory. In Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR’11). 21--31.Google ScholarGoogle Scholar
  11. Joel Coburn, Adrian M. Caulfield, Ameen Akel, Laura M. Grupp, Rajesh K. Gupta, Ranjit Jhala, and Steven Swanson. 2011. NV-heaps: Making persistent objects fast and safe with next-generation, non-volatile memories. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’11). 105--118. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Edward Grady Coffman and Peter J. Denning. 1973. Operating Systems Theory. Vol. 973.Google ScholarGoogle Scholar
  13. Jeremy Condit, Edmund B. Nightingale, Christopher Frost, Engin Ipek, Benjamin Lee, Doug Burger, and Derrick Coetzee. 2009. Better I/O through byte-addressable, persistent memory. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP’09). 133--146. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Intel Cooperation. 2015. NVDIMM Namespace Specification. Retrieved from http://pmem.io/documents/NVDIMM_ Namespace_Spec.pdf.Google ScholarGoogle Scholar
  15. Intel Cooperation. 2016. Intel Architecture Instruction Set Extensions Programming Reference. Retrieved from https://software.intel.com/sites/default/files/managed/b4/3a/319433-024.pdf.Google ScholarGoogle Scholar
  16. Peter J. Denning. 1968. The working set model for program behavior. Commun. ACM 11, 5 (1968), 323--333. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Edward Doller. 2009. Phase change memory and its impacts on memory hierarchy. Retrieved from http://www.pdl.cmu.edu/SDI/2009/slides/Numonyx.pdf.Google ScholarGoogle Scholar
  18. Subramanya R. Dulloor, Sanjay Kumar, Anil Keshavamurthy, Philip Lantz, Dheeraj Reddy, Rajesh Sankaran, and Jeff Jackson. 2014. System software for persistent memory. In Proceedings of the 9th European Conference on Computer Systems (EuroSys’14). 15:1--15:15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Tyler Harter, Chris Dragga, Michael Vaughn, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2011. A file is not a file: Understanding the I/O behavior of apple desktop applications. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP’11). 71--83. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Jian Huang, Karsten Schwan, and Moinuddin K. Qureshi. 2014. NVRAM-aware logging in transaction systems. Proc. VLDB Endow. 8, 4 (Dec. 2014), 389--400. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Taeho Hwang, Jaemin Jung, and Youjip Won. 2015. Heapo: Heap-based persistent object store. ACM Trans. Storage (TOS) 11, 1 (2015), 3. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Lei Jiang, Bo Zhao, Youtao Zhang, Jun Yang, and B. R. Childers. 2012. Improving write operations in MLC phase change memory. In Proceedings of the 18th International Symposium on High Performance Computer Architecture (HPCA’12). 1--10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Heeseung Jo, Jeong-Uk Kang, Seon-Yeong Park, Jin-Soo Kim, and Joonwon Lee. 2006. FAB: Flash-aware buffer management policy for portable media players. IEEE Trans. Consum. Electron. 52, 2 (May 2006), 485--493. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Theodore Johnson and Dennis Shasha. 1994. 2Q: A low-overhead high-performance buffer management replacement algorithm. In Proceedings of the 20th International Conference on Very Large Data Bases (VLDB’94). 439--450. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. William K. Josephson, Lars A. Bongo, David Flynn, and Kai Li. 2010. DFS: A file system for virtualized flash storage. In Proceedings of the 8th USENIX Conference on File and Storage Technologies (FAST’10). USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Myoungsoo Jung, John Shalf, and Mahmut Kandemir. 2013. Design of a large-scale storage-class RRAM system. In Proceedings of the 27th International ACM Conference on International Conference on Supercomputing (ICS’13). 103--114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Sooyong Kang, Sungmin Park, Hoyoung Jung, Hyoki Shim, and Jaehyuk Cha. 2009. Performance trade-offs in using NVRAM write buffer for flash memory-based storage devices. IEEE Trans. Comput. (TC) 58, 6 (June 2009), 744--758. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Jeffrey Katcher. 1997. Postmark: A New File System Benchmark. Technical Report TR3022, Network Appliance.Google ScholarGoogle Scholar
  29. Hyojun Kim and Seongjun Ahn. 2008. BPLRU: A buffer management scheme for improving random writes in flash storage. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST’08). 16:1--16:14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. ESOS LABORATORY. 2013. Mobibench benchmark tool. Retrieved from http://www.mobibench.co.kr/.Google ScholarGoogle Scholar
  31. B. C. Lee, Ping Zhou, Jun Yang, Youtao Zhang, Bo Zhao, E. Ipek, O. Mutlu, and D. Burger. 2010. Phase-change technology and the future of main memory. Micro, IEEE 30, 1 (Jan 2010), 143--143. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Benjamin C. Lee, Engin Ipek, Onur Mutlu, and Doug Burger. 2009. Architecting phase change memory as a scalable dram alternative. In Proceedings of the 36th Annual International Symposium on Computer Architecture (ISCA’09). 2--13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Changman Lee, Dongho Sim, Jooyoung Hwang, and Sangyeun Cho. 2015. F2FS: A new file system for flash storage. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST’15). USENIX, Santa Clara, CA. Retrieved from https://www.usenix.org/conference/fast15/technical-sessions/presentation/lee. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Eunji Lee and Hyokyung Bahn. 2014. Caching strategies for high-performance storage media. ACM Trans. Storage (TOS) 10, 3 (Aug. 2014), 11:1--11:22. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Youyou Lu, Jiwu Shu, Youmin Chen, and Tao Li. 2017. Octopus: An RDMA-enabled distributed persistent memory file system. In Proceedings of the 2017 USENIX Annual Technical Conference (USENIX ATC’17). USENIX Association, 773--785. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Y. Lu, J. Shu, and L. Sun. 2015. Blurred persistence in transactional persistent memory. In Proceedings of the 31st Symposium on Mass Storage Systems and Technologies (MSST’15). 1--13.Google ScholarGoogle Scholar
  37. Youyou Lu, Jiwu Shu, and Long Sun. 2016. Blurred persistence: Efficient transactions in persistent memory. ACM Trans. Storage (TOS) 12, 1 (2016), 3. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Y. Lu, J. Shu, L. Sun, and O. Mutlu. 2014. Loose-ordering consistency for persistent memory. In Proceedings of the 32nd IEEE International Conference on Computer Design (ICCD’14). 216--223.Google ScholarGoogle Scholar
  39. Youyou Lu, Jiwu Shu, Long Sun, and Onur Mutlu. 2017. Improving performance and endurance of persistent memory with loose-ordering consistency. IEEE Trans. Parall. Distrib. Syst. PP, 99 (2017), 1--1.Google ScholarGoogle Scholar
  40. Youyou Lu, Jiwu Shu, and Wei Wang. 2014. ReconFS: A reconstructable file system on flash storage. In Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST’14). USENIX, Berkeley, CA, 75--88. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Youyou Lu, Jiwu Shu, and Weimin Zheng. 2013. Extending the lifetime of flash-based storage through reducing write amplification from file systems. In Proceedings of the 11th USENIX Conference on File and Storage Technologies (FAST’13). USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Nimrod Megiddo and Dharmendra S. Modha. 2003. ARC: A self-tuning, low overhead replacement cache. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies (FAST’03). 115--130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Changwoo Min, Kangnyeon Kim, Hyunjin Cho, Sang-Won Lee, and Young Ik Eom. 2012. SFS: Random write considered harmful in solid state drives. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12). San Jose, CA, 139--154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Iulian Moraru, David G. Andersen, Michael Kaminsky, Niraj Tolia, Parthasarathy Ranganathan, and Nathan Binkert. 2013. Consistent, durable, and safe memory management for byte-addressable nonvolatile main memory. In Proceedings of the First ACM SIGOPS Conference on Timely Results in Operating Systems (TRIOS’13). 1:1--1:17. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Edmund B. Nightingale, Kaushik Veeraraghavan, Peter M. Chen, and Jason Flinn. 2006. Rethink the sync. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI’06). 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Jiaxin Ou and Jiwu Shu. 2016. Fast and failure-consistent updates of application data in non-volatile main memory file system. In Proceedings of the 32st Symposium on Mass Storage Systems and Technologies (MSST’16).Google ScholarGoogle Scholar
  47. Jiaxin Ou, Jiwu Shu, and Youyou Lu. 2016. A high-performance file system for non-volatile main memory. In Proceedings of the 11th European Conference on Computer Systems. ACM, 12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Jiaxin Ou, Jiwu Shu, Youyou Lu, Letian Yi, and Wei Wang. 2014. EDM: An endurance-aware data migration scheme for load balancing in SSD storage clusters. In Proceedings of the 28th International Parallel and Distributed Processing Symposium (IPDPS’14). 787--796. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Steven Pelley, Peter M. Chen, and Thomas F. Wenisch. 2014. Memory persistency. In Proceedings of the 41st Annual International Symposium on Computer Architecuture (ISCA’14). 265--276. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Moinuddin K. Qureshi, Vijayalakshmi Srinivasan, and Jude A. Rivers. 2009. Scalable high-performance main memory system using phase-change memory technology. In Proceedings of the 36th Annual International Symposium on Computer Architecture (ISCA’09). 24--33. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Luiz E. Ramos, Eugene Gorbatov, and Ricardo Bianchini. 2011. Page placement in hybrid memory systems. In Proceedings of the International Conference on Supercomputing (ICS’11). 85--95. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Drew Roselli, Jacob R. Lorch, and Thomas E. Anderson. 2000. A comparison of file system workloads. In Proceedings of the Annual Conference on USENIX Annual Technical Conference (ATEC’00). 41--54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Chris Ruemmler and John Wilkes. 1993. UNIX disk access patterns. In Proceedings of the USENIX Winter Conference, Vol. 93. 405--420.Google ScholarGoogle Scholar
  54. Long Sun, Youyou Lu, and Jiwu Shu. 2015. DP2: Reducing transaction overhead with differential and dual persistency in persistent memory. In Proceedings of the ACM International Conference on Computing Frontiers (CF’15). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Kosuke Suzuki and Steven Swanson. 2015. The Non-Volatile Memory Technology Database (NVMDB). Technical Report CS2015-1011. Department of Computer Science 8 Engineering, University of California, San Diego.Google ScholarGoogle Scholar
  56. Shivaram Venkataraman, Niraj Tolia, Parthasarathy Ranganathan, and Roy H. Campbell. 2011. Consistent and durable data structures for non-volatile byte-addressable memory. In Proceedings of the 9th USENIX Conference on File and Storage Technologies (FAST’11). San Jose, CA, 61--75. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Haris Volos, Guilherme Magalhaes, Ludmila Cherkasova, and Jun Li. 2015. Quartz: A lightweight performance emulator for persistent memory software. In Proceedings of the 16th Annual Middleware Conference. ACM, 37--49. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Haris Volos, Sanketh Nalli, Sankarlingam Panneerselvam, Venkatanathan Varadarajan, Prashant Saxena, and Michael M. Swift. 2014. Aerie: Flexible file-system interfaces to storage-class memory. In Proceedings of the 9th European Conference on Computer Systems (EuroSys’14). 14:1--14:14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Haris Volos, Andres Jaan Tack, and Michael M. Swift. 2011. Mnemosyne: Lightweight persistent memory. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’11). 91--104. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. D. L. Willick, D. L. Eager, and R. B. Bunt. 1993. Disk cache replacement policies for network fileservers. In Proceedings the 13th International Conference on Distributed Computing Systems (ICDCS’93). 2--11.Google ScholarGoogle Scholar
  61. Xiaojian Wu and A. L. Narasimha Reddy. 2011. SCMFS: A file system for storage class memory. In Proceedings of the 2011 International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’11). 39:1--39:11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Jian Xu and Steven Swanson. 2016. NOVA: A log-structured file system for hybrid volatile/non-volatile main memories. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 323--338. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Jun Yang, Qingsong Wei, Cheng Chen, Chundong Wang, Khai Leong Yong, and Bingsheng He. 2015. NV-tree: Reducing consistency cost for NVM-based single level systems. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST’15). 167--181. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. J. Joshua Yang and R. Stanley Williams. 2013. Memristive devices in computing system: Promises and challenges. J. Emerg. Technol. Comput. Syst. (JETC’13) 9, 2 (May 2013), 11:1--11:20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Jiacheng Zhang, Jiwu Shu, and Youyou Lu. 2016. ParaFS: A log-structured file system to exploit the internal parallelism of flash devices. In Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC’16). Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Yiying Zhang and Steven Swanson. 2015. A study of application performance with non-volatile main memory. In Proceedings of the 31st Symposium on Mass Storage Systems and Technologies (MSST’15). 1--10.Google ScholarGoogle ScholarCross RefCross Ref
  67. Yiying Zhang, Jian Yang, Amirsaman Memaripour, and Steven Swanson. 2015. Mojim: A reliable and highly-available non-volatile memory system. In Proceedings of the 20th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’15). 3--18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. Jishen Zhao, Sheng Li, Doe Hyun Yoon, Yuan Xie, and Norman P. Jouppi. 2013. Kiln: Closing the performance gap between systems with and without persistence support. In Proceedings of the 46th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO’13). ACM, 421--432. Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Ping Zhou, Bo Zhao, Jun Yang, and Youtao Zhang. 2009. A durable and energy efficient main memory using phase change memory technology. In Proceedings of the 36th Annual International Symposium on Computer Architecture (ISCA’09). 14--23. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. HiNFS: A Persistent Memory File System with Both Buffering and Direct-Access

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Storage
          ACM Transactions on Storage  Volume 14, Issue 1
          Special Issue on NVM and Storage
          February 2018
          237 pages
          ISSN:1553-3077
          EISSN:1553-3093
          DOI:10.1145/3190860
          • Editor:
          • Sam H. Noh
          Issue’s Table of Contents

          Copyright © 2018 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 4 April 2018
          • Accepted: 1 July 2017
          • Received: 1 February 2017
          Published in tos Volume 14, Issue 1

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!