skip to main content
research-article

RACE: One-sided RDMA-conscious Extendible Hashing

Published:28 April 2022Publication History
Skip Abstract Section

Abstract

Memory disaggregation is a promising technique in datacenters with the benefit of improving resource utilization, failure isolation, and elasticity. Hashing indexes have been widely used to provide fast lookup services in distributed memory systems. However, traditional hashing indexes become inefficient for disaggregated memory, since the computing power in the memory pool is too weak to execute complex index requests. To provide efficient indexing services in disaggregated memory scenarios, this article proposes RACE hashing, a one-sided RDMA-Conscious Extendible hashing index with lock-free remote concurrency control and efficient remote resizing. RACE hashing enables all index operations to be efficiently executed by using only one-sided RDMA verbs without involving any compute resource in the memory pool. To support remote concurrent access with high performance, RACE hashing leverages a lock-free remote concurrency control scheme to enable different clients to concurrently operate the same hashing index in the memory pool in a lock-free manner. To resize the hash table with low overheads, RACE hashing leverages an extendible remote resizing scheme to reduce extra RDMA accesses caused by extendible resizing and allow concurrent request execution during resizing. Extensive experimental results demonstrate that RACE hashing outperforms state-of-the-art distributed in-memory hashing indexes by 1.4–13.7× in YCSB hybrid workloads.

REFERENCES

  1. [1] 2022. Gen-Z Technology. Retrieved from https://genzconsortium.org/.Google ScholarGoogle Scholar
  2. [2] 2022. Memcached—A Distributed Memory Object Caching System. Retrieved from https://memcached.org/.Google ScholarGoogle Scholar
  3. [3] 2022. Redis. Retrieved from https://redis.io/.Google ScholarGoogle Scholar
  4. [4] Aguilera Marcos K., Amit Nadav, Calciu Irina, Deguillard Xavier, Gandhi Jayneel, Novakovic Stanko, Ramanathan Arun, Subrahmanyam Pratap, Suresh Lalith, Tati Kiran, Venkatasubramanian Rajesh, and Wei Michael. 2018. Remote regions: A simple abstraction for remote memory. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’18). 775787.Google ScholarGoogle Scholar
  5. [5] Aguilera Marcos K., Keeton Kimberly, Novakovic Stanko, and Singhal Sharad. 2019. Designing far memory data structures: Think outside the box. In Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS’19). 120126.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Asanović Krste and Patterson David. 2014. FireBox: A hardware building block for 2020 warehouse-scale computers. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST’14).Google ScholarGoogle Scholar
  7. [7] Atikoglu Berk, Xu Yuehai, Frachtenberg Eitan, Jiang Song, and Paleczny Mike. 2012. Workload analysis of a large-scale key-value store. In Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS’12). 5364.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Birrittella Mark S., Debbage Mark, Huggahalli Ram, Kunz James, Lovett Tom, Rimmer Todd, Underwood Keith D., and Zak Robert C.. 2015. Intel Omni-path architecture: Enabling scalable, high performance fabrics. In Proceedings of the IEEE 23rd Annual Symposium on High-Performance Interconnects (HOTI’15). IEEE, 19.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Carbonari Amanda and Beschasnikh Ivan. 2017. Tolerating faults in disaggregated datacenters. In Proceedings of the 16th ACM Workshop on Hot Topics in Networks (HotNets’17). 164170.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Chen Zhangyu, Hua Yu, Ding Bo, and Zuo Pengfei. 2020. Lock-free concurrent level hashing for persistent memory. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’20). 799812.Google ScholarGoogle Scholar
  11. [11] Cooper Brian F., Silberstein Adam, Tam Erwin, Ramakrishnan Raghu, and Sears Russell. 2010. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC’10). 143154.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Costa Paolo, Ballani Hitesh, Razavi Kaveh, and Kash Ian. 2015. R2C2: A network stack for rack-scale computers. In Proceedings of the ACM Conference on Special Interest Group on Data Communication (SIGCOMM’15). 551564.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Ana Lúcia De Moura and Roberto Ierusalimschy. 2009. Revisiting coroutines. ACM Trans. Program. Lang. Syst. 31, 2, Article 6 (February 2009), 31 pages.Google ScholarGoogle Scholar
  14. [14] Dragojević Aleksandar, Narayanan Dushyanth, Castro Miguel, and Hodson Orion. 2014. FaRM: Fast remote memory. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI’14). 401414.Google ScholarGoogle Scholar
  15. [15] Fagin Ronald, Nievergelt Jurg, Pippenger Nicholas, and Strong H Raymond. 1979. Extendible hashing-a fast access method for dynamic files. ACM Trans. Database Syst. 4, 3 (1979), 315344.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Fan Bin, Andersen David G., and Kaminsky Michael. 2013. Memc3: Compact and concurrent memcache with dumber caching and smarter hashing. In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI’13). 371384.Google ScholarGoogle Scholar
  17. [17] Faraboschi Paolo, Keeton Kimberly, Marsland Tim, and Milojicic Dejan. 2015. Beyond processor-centric operating systems. In Proceedings of the 15th Workshop on Hot Topics in Operating Systems (HotOS’15).Google ScholarGoogle Scholar
  18. [18] Franklin Michael J., Carey Michael J., and Livny Miron. 1997. Transactional client-server cache consistency: Alternatives and performance. ACM Trans. Database Syst. 22, 3 (1997), 315363.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Gao Peter X., Narayan Akshay, Karandikar Sagar, Carreira Joao, Han Sangjin, Agarwal Rachit, Ratnasamy Sylvia, and Shenker Scott. 2016. Network requirements for resource disaggregation. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16). 249264.Google ScholarGoogle Scholar
  20. [20] He Yongjun, Lu Jiacheng, and Wang Tianzheng. 2020. CoroBase: Coroutine-oriented main-memory database engine. arXiv: 2010.15981. Retrieved from https://arxiv.org/abs/2010.15981.Google ScholarGoogle Scholar
  21. [21] Herlihy Maurice, Shavit Nir, and Tzafrir Moran. 2008. Hopscotch hashing. In Proceedings of the International Symposium on Distributed Computing (DISC’08). Springer, 350364.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Corporation Hewlett Packard. 2015. The Machine: A New Kind of Computer. Retrieved from https://www.hpl.hp.com/research/systems-research/themachine/.Google ScholarGoogle Scholar
  23. [23] Corporation Intel. 2022. Intel Rack Scale Design Architecture. Retrieved from https://www.intel.com/content/www/us/en/architecture-and-technology/rack-scale-design-overview.html.Google ScholarGoogle Scholar
  24. [24] Kalia Anuj, Kaminsky Michael, and Andersen David G.. 2016. Design guidelines for high performance RDMA systems. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’16). 437450.Google ScholarGoogle Scholar
  25. [25] Kocberber Onur, Grot Boris, Picorel Javier, Falsafi Babak, Lim Kevin, and Ranganathan Parthasarathy. 2013. Meet the walkers: Accelerating index traversals for in-memory databases. In Proceedings of the 46th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO’15). 468479.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Li Bojie, Ruan Zhenyuan, Xiao Wencong, Lu Yuanwei, Xiong Yongqiang, Putnam Andrew, Chen Enhong, and Zhang Lintao. 2017. KV-Direct: High-performance in-memory key-value store with programmable NIC. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP’17). 137152.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Li Sheng, Lim Hyeontaek, Lee Victor W., Ahn Jung Ho, Kalia Anuj, Kaminsky Michael, Andersen David G., Seongil O., Lee Sukhan, and Dubey Pradeep. 2015. Architecting to achieve a billion requests per second throughput on a single key-value store server platform. In Proceedings of the 42nd Annual International Symposium on Computer Architecture (ISCA’15). 476488.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Li Xiaozhou, Andersen David G., Kaminsky Michael, and Freedman Michael J.. 2014. Algorithmic improvements for fast concurrent cuckoo hashing. In Proceedings of the 9th European Conference on Computer Systems (EuroSys’14). 114.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Lim Hyeontaek, Kaminsky Michael, and Andersen David G.. 2017. Cicada: Dependably fast multi-core in-memory transactions. In Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD’17). 2135.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Lim Kevin, Chang Jichuan, Mudge Trevor, Ranganathan Parthasarathy, Reinhardt Steven K., and Wenisch Thomas F.. 2009. Disaggregated memory for expansion and sharing in blade servers. In Proceedings of the 36th Annual International Symposium on Computer Architecture (ISCA’09). 267278.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Lim Kevin, Turner Yoshio, Santos Jose Renato, AuYoung Alvin, Chang Jichuan, Ranganathan Parthasarathy, and Wenisch Thomas F.. 2012. System-level implications of disaggregated memory. In Proceedings of the IEEE International Symposium on High-Performance Comp Architecture (HPCA’12). IEEE, 112.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Mao Yandong, Kohler Eddie, and Morris Robert Tappan. 2012. Cache craftiness for fast multicore key-value storage. In Proceedings of the 7th ACM European Conference on Computer Systems (EuroSys’12). 183196.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. [33] Mitchell Christopher, Geng Yifeng, and Li Jinyang. 2013. Using one-sided RDMA reads to build a fast, CPU-efficient key-value store. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’15). 103114.Google ScholarGoogle Scholar
  34. [34] Mitzenmacher Michael. 2001. The power of two choices in randomized load balancing. IEEE Trans. Parallel Distrib. Syst. 12, 10 (2001), 10941104.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Nam Moohyeon, Cha Hokeun, Choi Young-ri, Noh Sam H., and Nam Beomseok. 2019. Write-optimized dynamic hashing for persistent memory. In Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST’19). 3144.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Nguyen Nhan and Tsigas Philippas. 2014. Lock-free cuckoo hashing. In Proceedings of the IEEE 34th International Conference on Distributed Computing Systems (ICDCS’14). IEEE, 627636.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Nishtala Rajesh, Fugal Hans, Grimm Steven, Kwiatkowski Marc, Lee Herman, Li Harry C., McElroy Ryan, Paleczny Mike, Peek Daniel, Saab Paul, et al. 2013. Scaling memcache at facebook. In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI’13). 385398.Google ScholarGoogle Scholar
  38. [38] Pagh Rasmus and Rodler Flemming Friche. 2004. Cuckoo hashing. J. Algor. 51, 2 (2004), 122144.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. [39] Ruan Zhenyuan, Schwarzkopf Malte, Aguilera Marcos K., and Belay Adam. 2020. AIFM: High-performance, application-integrated far memory. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI’20). 315332.Google ScholarGoogle Scholar
  40. [40] Shan Yizhou, Huang Yutong, Chen Yilun, and Zhang Yiying. 2018. LegoOS: A disseminated, distributed OS for hardware resource disaggregation. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18). 6987.Google ScholarGoogle Scholar
  41. [41] Shrivastav Vishal, Valadarsky Asaf, Ballani Hitesh, Costa Paolo, Lee Ki Suh, Wang Han, Agarwal Rachit, and Weatherspoon Hakim. 2019. Shoal: A network architecture for disaggregated racks. In Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI’19). 255270.Google ScholarGoogle Scholar
  42. [42] Tsai Shin-Yeh, Shan Yizhou, and Zhang Yiying. 2020. Disaggregating persistent memory and controlling them remotely: An exploration of passive disaggregated key-value stores. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’20). 3348.Google ScholarGoogle Scholar
  43. [43] Tsai Shin-Yeh and Zhang Yiying. 2017. Lite kernel rdma support for datacenter applications. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP’17). 306324.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. [44] Wang Chenxi, Ma Haoran, Liu Shi, Li Yuanqi, Ruan Zhenyuan, Nguyen Khanh, Bond Michael D, Netravali Ravi, Kim Miryung, and Xu Guoqing Harry. 2020. Semeru: A memory-disaggregated managed runtime. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI’20). 261280.Google ScholarGoogle Scholar
  45. [45] Wang Tianzheng, Levandoski Justin, and Larson Per-Ake. 2018. Easy lock-free indexing in non-volatile memory. In Proceedings of the IEEE 34th International Conference on Data Engineering (ICDE’18). IEEE, 461472.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Wei Xingda, Shi Jiaxin, Chen Yanzhe, Chen Rong, and Chen Haibo. 2015. Fast in-memory transaction processing using RDMA and HTM. In Proceedings of the 25th Symposium on Operating Systems Principles (SOSP’15). 87104.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. [47] Yu Xiangyao, Bezerra George, Pavlo Andrew, Devadas Srinivas, and Stonebraker Michael. 2014. Staring into the abyss: An evaluation of concurrency control with one thousand Cores. Proc. VLDB Endow. 8, 3 (Nov. 2014), 209220.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Zuo Pengfei, Hua Yu, and Wu Jie. 2018. Write-optimized and high-performance hashing index scheme for persistent memory. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18). 461476.Google ScholarGoogle Scholar

Index Terms

  1. RACE: One-sided RDMA-conscious Extendible Hashing

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Storage
        ACM Transactions on Storage  Volume 18, Issue 2
        May 2022
        248 pages
        ISSN:1553-3077
        EISSN:1553-3093
        DOI:10.1145/3522733
        • Editor:
        • Sam H. Noh
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 28 April 2022
        • Accepted: 1 January 2022
        • Revised: 1 November 2021
        • Received: 1 November 2021
        Published in tos Volume 18, Issue 2

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!