Abstract
We present XIndex, which is a concurrent index library and designed for fast queries. It includes a concurrent ordered index (XIndex-R) and a concurrent hash index (XIndex-H). Similar to a recent proposal of the learned index, the indexes in XIndex use learned models to optimize index efficiency. Compared with the learned index, for the ordered index, XIndex-R is able to handle concurrent writes effectively and adapts its structure according to runtime workload characteristics. For the hash index, XIndex-H is able to avoid the resize operation blocking concurrent writes. Furthermore, the indexes in XIndex can index string keys much more efficiently than the learned index. We demonstrate the advantages of XIndex with YCSB, TPC-C (KV), which is a TPC-C-inspired benchmark for key-value stores, and micro-benchmarks. Compared with ordered indexes of Masstree and Wormhole, XIndex-R achieves up to 3.2× and 4.4× performance improvement on a 24-core machine. Compared with hash indexes of Intel TBB HashMap, XIndex-H achieves up to 3.1× speedup. The performance further improves by 91% after adding the optimizations on indexing string keys. The library is open-sourced.1
- [1] . 2017. TRIAD: Creating synergies between memory, disk and log in log structured key-value stores. In 2017 USENIX Annual Technical Conference, USENIX ATC 2017, Santa Clara, CA, USA, July 12-14, 2017, and (Eds.). USENIX Association, 363–375. https://www.usenix.org/conference/atc17/technical-sessions/presentation/balmau. Google Scholar
Digital Library
- [2] . 1977. Concurrency of operations on B-trees. Acta Informatica 9 (1977), 1–21.
DOI: DOI: https://doi.org/10.1007/BF00263762 Google ScholarDigital Library
- [3] . 2013. STX B+ Tree C++ Template Classes. https://panthema.net/2007/stx-btree/. (2013).Google Scholar
- [4] . 2018. HOT: A height optimized trie index for main-memory database systems. In Proceedings of the 2018 International Conference on Management of Data (SIGMOD’18). Association for Computing Machinery, New York, NY, USA, 521–534.
DOI: DOI: https://doi.org/10.1145/3183713.3196896 Google ScholarDigital Library
- [5] . 1970. Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13, 7 (
July 1970), 422–426.DOI: DOI: https://doi.org/10.1145/362686.362692 Google ScholarDigital Library
- [6] . 2001. Main-memory index structures with fixed-size partial keys. In Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data (SIGMOD’01). Association for Computing Machinery, New York, NY, USA, 163–174.
DOI: DOI: https://doi.org/10.1145/375663.375681 Google ScholarCross Ref
- [7] . 2006. An improved construction for counting bloom filters. In Algorithms - ESA 2006, 14th Annual European Symposium, Zurich, Switzerland, September 11-13, 2006, Proceedings (Lecture Notes in Computer Science), and (Eds.), Vol. 4168. Springer, 684–695.
DOI: DOI: https://doi.org/10.1007/11841036_61 Google ScholarCross Ref
- [8] . 2010. A practical concurrent binary search tree. In Proceedings of the 15th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP’10). Association for Computing Machinery, New York, NY, USA, 257–268.
DOI: DOI: https://doi.org/10.1145/1693453.1693488 Google ScholarCross Ref
- [9] . 2010. A practical concurrent binary search tree. SIGPLAN Not. 45, 5 (
Jan. 2010), 257–268.DOI: DOI: https://doi.org/10.1145/1837853.1693488 Google ScholarCross Ref
- [10] . 2001. Cache-conscious concurrency control of main-memory indexes on shared-memory multiprocessor systems. In Proceedings of the 27th International Conference on Very Large Data Bases (VLDB’01). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 181–190. Google Scholar
Digital Library
- [11] . 1986. Fractional cascading: I. A data structuring technique. Algorithmica 1, 2 (1986), 133–162.
DOI: DOI: https://doi.org/10.1007/BF01840440Google ScholarDigital Library
- [12] . 1986. Fractional cascading: II. Applications. Algorithmica 1, 2 (1986), 163–191.
DOI: DOI: https://doi.org/10.1007/BF01840441Google ScholarCross Ref
- [13] . 2020. Lock-free concurrent level hashing for persistent memory. In 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15-17, 2020, and (Eds.). USENIX Association, 799–812. https://www.usenix.org/conference/atc20/presentation/chen. Google Scholar
Digital Library
- [14] . 2012. Scalable address spaces using RCU balanced trees. In Proceedings of the Seventeenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS XVII). Association for Computing Machinery, New York, NY, USA, 199–210.
DOI: DOI: https://doi.org/10.1145/2150976.2150998 Google ScholarCross Ref
- [15] . 2012. Scalable address spaces using RCU balanced trees. SIGPLAN Not. 47, 4 (
March 2012), 199–210.DOI: DOI: https://doi.org/10.1145/2248487.2150998 Google ScholarCross Ref
- [16] . 2020. From WiscKey to bourbon: A learned index for log-structured merge trees. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020. USENIX Association, 155–171. https://www.usenix.org/conference/osdi20/presentation/dai. Google Scholar
Digital Library
- [17] . 2013. Anti-caching: A new approach to database management system architecture. Proc. VLDB Endow. 6, 14 (
Sept. 2013), 1942–1953.DOI: DOI: https://doi.org/10.14778/2556549.2556575 Google ScholarDigital Library
- [18] . 2013. Hekaton: SQL server’s memory-optimized OLTP engine. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data (SIGMOD’13). Association for Computing Machinery, New York, NY, USA, 1243–1254.
DOI: DOI: https://doi.org/10.1145/2463676.2463710 Google ScholarCross Ref
- [19] . 2020. ALEX: An updatable adaptive learned index. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (SIGMOD’20). Association for Computing Machinery, New York, NY, USA, 969–984.
DOI: DOI: https://doi.org/10.1145/3318464.3389711 Google ScholarCross Ref
- [20] . 1971. Process synchronization without long-term interlock. In Proceedings of the Third ACM Symposium on Operating Systems Principles (SOSP’71). Association for Computing Machinery, New York, NY, USA, 95–100.
DOI: DOI: https://doi.org/10.1145/800212.806505 Google ScholarCross Ref
- [21] . 1998. Summary cache: A scalable wide-area web cache sharing protocol. In Proceedings of the ACM SIGCOMM’98 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM’98). Association for Computing Machinery, New York, NY, USA, 254–265.
DOI: DOI: https://doi.org/10.1145/285237.285287 Google ScholarCross Ref
- [22] . 2020. The PGM-index: A fully-dynamic compressed learned index with provable worst-case bounds. Proc. VLDB Endow. 13, 8 (
April 2020), 1162–1175.DOI: DOI: https://doi.org/10.14778/3389133.3389135 Google ScholarCross Ref
- [23] . 2019. OpenStreetMap on AWS. https://aws.amazon.com/public-datasets/osm.
Accessed: 2019-4-24. Google Scholar - [24] . 1960. Trie memory. Commun. ACM 3, 9 (
Sept. 1960), 490–499.DOI: DOI: https://doi.org/10.1145/367390.367400 Google ScholarDigital Library
- [25] . 2019. FITing-tree: A data-aware index structure. In Proceedings of the 2019 International Conference on Management of Data (SIGMOD’19). Association for Computing Machinery, New York, NY, USA, 1189–1206.
DOI: DOI: https://doi.org/10.1145/3299869.3319860 Google ScholarCross Ref
- [26] . 2012. Foster B-trees. ACM Trans. Database Syst. 37, 3, Article
17 (Sept. 2012), 29 pages.DOI: DOI: https://doi.org/10.1145/2338626.2338630 Google ScholarCross Ref
- [27] . 2015. Fast compressed tries through path decompositions. ACM J. Exp. Algorithmics 19, Article
3.4 (Jan. 2015), 20 pages.DOI: DOI: https://doi.org/10.1145/2656332 Google ScholarCross Ref
- [28] . 1990. Linearizability: A correctness condition for concurrent objects. ACM Trans. Program. Lang. Syst. 12, 3 (
July 1990), 463–492.DOI: DOI: https://doi.org/10.1145/78969.78972 Google ScholarDigital Library
- [29] . 2018. The periodic table of data structures. IEEE Data Eng. Bull. 41, 3 (2018), 64–75. http://sites.computer.org/debull/A18sept/p64.pdf.Google Scholar
- [30] . 2019. Learning data structure alchemy. IEEE Data Eng. Bull. 42, 2 (2019), 47–58. http://sites.computer.org/debull/A19june/p47.pdf.Google Scholar
- [31] . 2018. The data calculator: Data structure design and cost synthesis from first principles and learned cost models. In Proceedings of the 2018 International Conference on Management of Data (SIGMOD’18). Association for Computing Machinery, New York, NY, USA, 535–550.
DOI: DOI: https://doi.org/10.1145/3183713.3199671 Google ScholarCross Ref
- [32] . 2018. Redesigning LSMs for nonvolatile memory with NoveLSM. In 2018 USENIX Annual Technical Conference, USENIX ATC 2018, Boston, MA, USA, July 11-13, 2018, and (Eds.). USENIX Association, 993–1005. https://www.usenix.org/conference/atc18/presentation/kannan. Google Scholar
Digital Library
- [33] . 2020. RadixSpline: A single-pass learned index. In Proceedings of the Third International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (aiDM’20). Association for Computing Machinery, New York, NY, USA, Article
5 , 5 pages.DOI: DOI: https://doi.org/10.1145/3401071.3401659 Google ScholarCross Ref
- [34] . 2019. SageDB: A learned database system. In CIDR 2019, 9th Biennial Conference on Innovative Data Systems Research, Asilomar, CA, USA, January 13-16, 2019, Online Proceedings. www.cidrdb.org. http://cidrdb.org/cidr2019/papers/p117-kraska-cidr19.pdf.Google Scholar
- [35] . 2018. The case for learned index structures. In Proceedings of the 2018 International Conference on Management of Data (SIGMOD’18). Association for Computing Machinery, New York, NY, USA, 489–504.
DOI: DOI: https://doi.org/10.1145/3183713.3196909 Google ScholarCross Ref
- [36] . 1981. On optimistic methods for concurrency control. ACM Trans. Database Syst. 6, 2 (
June 1981), 213–226.DOI: DOI: https://doi.org/10.1145/319566.319567 Google ScholarDigital Library
- [37] . 1977. Concurrent reading and writing. Commun. ACM 20, 11 (
Nov. 1977), 806–811.DOI: DOI: https://doi.org/10.1145/359863.359878 Google ScholarCross Ref
- [38] . 2013. The adaptive radix tree: ARTful indexing for main-memory databases. In Proceedings of the 2013 IEEE 29th International Conference on Data Engineering (ICDE’13). IEEE Computer Society, 38–49.
DOI: DOI: https://doi.org/10.1109/ICDE.2013.6544812 Google ScholarCross Ref
- [39] . 2019. KVell: The design and implementation of a fast persistent key-value store. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP’19). Association for Computing Machinery, New York, NY, USA, 447–461.
DOI: DOI: https://doi.org/10.1145/3341301.3359628 Google ScholarDigital Library
- [40] . 2009. Meme-tracking and the dynamics of the news cycle. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’09). Association for Computing Machinery, New York, NY, USA, 497–506.
DOI: DOI: https://doi.org/10.1145/1557019.1557077 Google ScholarDigital Library
- [41] . 2013. The Bw-tree: A B-tree for new hardware platforms. In 29th IEEE International Conference on Data Engineering, ICDE 2013, Brisbane, Australia, April 8-12, 2013, , , and (Eds.). IEEE Computer Society, 302–313.
DOI: DOI: https://doi.org/10.1109/ICDE.2013.6544834 Google ScholarCross Ref
- [42] . 2019. A scalable learned index scheme in storage systems. CoRR abs/1905.06256 (2019). arXiv:1905.06256 http://arxiv.org/abs/1905.06256.Google Scholar
- [43] . 2020. LISA: A learned index structure for spatial data. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (SIGMOD’20). Association for Computing Machinery, New York, NY, USA, 2119–2133.
DOI: DOI: https://doi.org/10.1145/3318464.3389703 Google ScholarCross Ref
- [44] . 2016. WiscKey: Separating keys from values in SSD-conscious storage. In 14th USENIX Conference on File and Storage Technologies, FAST 2016, Santa Clara, CA, USA, February 22-25, 2016, and (Eds.). USENIX Association, 133–148. https://www.usenix.org/conference/fast16/technical-sessions/presentation/lu. Google Scholar
Digital Library
- [45] . 2012. Cache craftiness for fast multicore key-value storage. In Proceedings of the 7th ACM European Conference on Computer Systems (EuroSys’12). Association for Computing Machinery, New York, NY, USA, 183–196.
DOI: DOI: https://doi.org/10.1145/2168836.2168855 Google ScholarCross Ref
- [46] . 1998. Read-copy update: Using execution history to solve concurrency problems. In Parallel and Distributed Computing and Systems. 509–518.Google Scholar
- [47] . 1992. ARIES: A transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM Trans. Database Syst. 17, 1 (
March 1992), 94–162.DOI: DOI: https://doi.org/10.1145/128765.128770 Google ScholarDigital Library
- [48] . 2020. Learning multi-dimensional indexes. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (SIGMOD’20). Association for Computing Machinery, New York, NY, USA, 985–1000.
DOI: DOI: https://doi.org/10.1145/3318464.3380579 Google ScholarCross Ref
- [49] . 1996. The log-structured merge-tree (LSM-Tree). Acta Informatica 33, 4 (1996), 351–385.
DOI: DOI: https://doi.org/10.1007/s002360050048 Google ScholarCross Ref
- [50] . 2018. An efficient memory-mapped key-value store for flash storage. In Proceedings of the ACM Symposium on Cloud Computing (SoCC’18). Association for Computing Machinery, New York, NY, USA, 490–502.
DOI: DOI: https://doi.org/10.1145/3267809.3267824 Google ScholarCross Ref
- [51] . 2008. Intel® threading building blocks. Journal of Computing Sciences in Colleges 23, 4 (2008), 298–298. Google Scholar
Digital Library
- [52] . 2007. Cache-, hash- and space-efficient bloom filters. In Experimental Algorithms, 6th International Workshop, WEA 2007, Rome, Italy, June 6-8, 2007, Proceedings (Lecture Notes in Computer Science), (Ed.), Vol. 4525. Springer, 108–121.
DOI: DOI: https://doi.org/10.1007/978-3-540-72845-0_9 Google ScholarCross Ref
- [53] . 2007. On-line index selection for shifting workloads. In Proceedings of the 23rd International Conference on Data Engineering Workshops, ICDE 2007, 15-20 April 2007, Istanbul, Turkey. IEEE Computer Society, 459–468.
DOI: DOI: https://doi.org/10.1109/ICDEW.2007.4401029 Google ScholarCross Ref
- [54] . 2012. BLSM: A general purpose log structured merge tree. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD’12). Association for Computing Machinery, New York, NY, USA, 217–228.
DOI: DOI: https://doi.org/10.1145/2213836.2213862 Google ScholarCross Ref
- [55] . 2006. Split-ordered lists: Lock-free extensible hash tables. J. ACM 53, 3 (
May 2006), 379–405.DOI: DOI: https://doi.org/10.1145/1147954.1147958 Google ScholarCross Ref
- [56] . 2013. Speedy transactions in multicore in-memory databases. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (SOSP’13). Association for Computing Machinery, New York, NY, USA, 18–32.
DOI: DOI: https://doi.org/10.1145/2517349.2522713 Google ScholarCross Ref
- [57] . 2014. An efficient design and implementation of LSM-tree based key-value store on open-channel SSD. In Proceedings of the Ninth European Conference on Computer Systems (EuroSys’14). Association for Computing Machinery, New York, NY, USA, Article
16 , 14 pages.DOI: DOI: https://doi.org/10.1145/2592798.2592804 Google ScholarDigital Library
- [58] . 2019. Wormhole: A fast ordered index for in-memory data management. In Proceedings of the Fourteenth EuroSys Conference 2019 (EuroSys’19). Association for Computing Machinery, New York, NY, USA, Article
18 , 16 pages.DOI: DOI: https://doi.org/10.1145/3302424.3303955 Google ScholarCross Ref
- [59] . 2020. MatrixKV: Reducing write stalls and write amplification in LSM-tree based KV stores with matrix container in NVM. In 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15-17, 2020, and (Eds.). USENIX Association, 17–31. https://www.usenix.org/conference/atc20/presentation/yao. Google Scholar
Digital Library
- [60] . 2016. Reducing the storage overhead of main-memory OLTP databases with hybrid indexes. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD’16). Association for Computing Machinery, New York, NY, USA, 1567–1581.
DOI: DOI: https://doi.org/10.1145/2882903.2915222 Google ScholarCross Ref
- [61] . 2018. SuRF: Practical range query filtering with fast succinct tries. In Proceedings of the 2018 International Conference on Management of Data (SIGMOD’18). Association for Computing Machinery, New York, NY, USA, 323–336.
DOI: DOI: https://doi.org/10.1145/3183713.3196931 Google ScholarCross Ref
- [62] . 2014. Fast databases with fast durability and recovery through multicore parallelism. In 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI’14, Broomfield, CO, USA, October 6-8, 2014, and (Eds.). USENIX Association, 465–477. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/zheng_wenting.Google Scholar
Index Terms
The Concurrent Learned Indexes for Multicore Data Storage
Recommendations
ALEX: An Updatable Adaptive Learned Index
SIGMOD '20: Proceedings of the 2020 ACM SIGMOD International Conference on Management of DataRecent work on "learned indexes" has changed the way we look at the decades-old field of DBMS indexing. The key idea is that indexes can be thought of as "models" that predict the position of a key in a dataset. Indexes can, thus, be learned. The ...
A Tutorial on Learned Multi-dimensional Indexes
SIGSPATIAL '20: Proceedings of the 28th International Conference on Advances in Geographic Information SystemsRecently, Machine Learning (ML, for short) has been successfully applied to database indexing. Initial experimentation on Learned Indexes has demonstrated better search performance and lower space requirements than their traditional database ...
Updatable Learned Indexes Meet Disk-Resident DBMS - From Evaluations to Design Choices
PACMMODAlthough many updatable learned indexes have been proposed in recent years, whether they can outperform traditional approaches on disk remains unknown. In this study, we revisit and implement four state-of-the-art updatable learned indexes on disk, and ...






Comments