Abstract
In the past decades, advances in speed of commodity CPUs have far outpaced advances in RAM latency. Main-memory access has therefore become a performance bottleneck for many computer applications; a phenomenon that is widely known as the "memory wall." In this paper, we report how research around the MonetDB database system has led to a redesign of database architecture in order to take advantage of modern hardware, and in particular to avoid hitting the memory wall. This encompasses (i) a redesign of the query execution model to better exploit pipelined CPU architectures and CPU instruction caches; (ii) the use of columnar rather than row-wise data storage to better exploit CPU data caches; (iii) the design of new cache-conscious query processing algorithms; and (iv) the design and automatic calibration of memory cost models to choose and tune these cache-conscious algorithms in the query optimizer.
References
- Ailamaki, A.G., DeWitt, DJ., Hill, M.D., and Wood, D.A. DBMSs on a modern processor: Where does time go? In International Conference on Very Large Data Bases (VLDB), Sept. 1999, 266--277. Google Scholar
Digital Library
- Boncz, P.A., Manegold, S., and Kersten, M.L. Database architecture optimized for the new bottleneck: Memory access. In International Conference on Very Large Data Bases (VLDB), Sept. 1999, 54--65. Google Scholar
Digital Library
- Boncz, P.A., Zukowski, M., and Nes, N. MonetDB/X100: Hyper-pipelining query execution. In International Conference on Innovative Data Systems Research (CIDR), Jan. 2005, 225--237.Google Scholar
- Chaudhuri, S. and Weikum, G. Rethinking database system architecture: Towards a self-tuning RISC-style database system. In International Conference on Very Large Data Bases (VLDB), Sept. 2000, 1--10. Google Scholar
Digital Library
- Chen, S., Ailamaki, A., Gibbons, PB., and Mowry, T.C. Inspector joins. In International Conference on Very Large Data Bases (VLDB), Aug. 2005. Google Scholar
Digital Library
- Chen, S., Ailamaki, A., Gibbons, PB., and Mowry, T.C. Improving hash join performance through prefetching ACM Trans. Database Syst., 32, 3,2007). Google Scholar
Digital Library
- Chen, S., Gibbons, PB., and Mowry T.C. Improving index performance through prefetching. In ACM SIGMOD International Conference on Management of Data (SIGMOD), June 2001. Google Scholar
Digital Library
- Copeland, G.P. and Khoshafian, S. A decomposition storage model. In ACM SIGMOD International Conference on Management of Data (SIGMOD), May 1985, 268--279. Google Scholar
Digital Library
- Govindaraju, N., Gray, J., Kumar, R., and Manocha, D. GPUTeraSort: High performance graphics coprocessor sorting for large database management. In ACM SIGMOD International Conference on Management of Data (SIGMOD), June 2006. Google Scholar
Digital Library
- Harizopoulos, S. and Ailamaki, A. STEPS towards cache-resident transaction processing. In International Conference on Very Large Data Bases (VLDB), Aug. 2004. Google Scholar
Digital Library
- Manegold, S. Understanding, modeling, and improving main-memory database performance. PhD thesis, Universiteit van Amsterdam, Amsterdam, the Netherlands, Dec. 2002.Google Scholar
- Manegold, S., Boncz, P.A., and Kersten, M.L. Generic database cost models for hierarchical memory systems. In International Conference on Very Large Data Bases (VLDB), Aug. 2002, 191--202. Google Scholar
Digital Library
- Manegold, S., Boncz, P.A., and Kersten, M.L. Optimizing main-memory join on modern hardware. IEEE Trans, Knowl, Data Eng., 14, 4 (July 2002), 709--730. Google Scholar
Digital Library
- Manegold, S., Boncz, P.A., Nes, N., and Kersten, M.L. Cache-conscious radix-decluster projections. In International Conference on Very Large Data Bases (VLDB), Aug. 2004, 684--695. Google Scholar
Digital Library
- Maynard, A.M.G., Donnelly, CM., and Olszewski, B.R. Contrasting characteristics and cache performance of technical and multi-user commercial workloads. SIGOPS Oper. Syst. Rev., 28, 5 (Aug. 1994), 145--156. Google Scholar
Digital Library
- Patterson, D. Latency lags bandwidth. Commun. ACM 47,10 (Oct. 2004), 71--75. Google Scholar
Digital Library
- Rao, J. and Ross, K.A. Making B+ -Trees cache conscious in main memory. In ACM SIGMOD International Conference on Management of Data (SIGMOD), June 2000. Google Scholar
Digital Library
- Ross, K.A. Conjunctive selection conditions in main memory. In Proceedings of the ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS), June 2002. Google Scholar
Digital Library
- Shatdal, A., Kant, C., and Naughton, J. Cache conscious algorithms for relational query processing. In International Conference on Very Large Data Bases (VLDB), Sept. 1994. 510--512. Google Scholar
Digital Library
- Stonebraker, M., Abadi, DJ., Batkin, A., Chen, X., Cherniack, M., Ferreira, M., Lau, E., Lin, A., Madden, S.R., O'Neil, E.J., O'Neil, P.E., Rasin, A., Tran, N., and Zdonik, S.B. C-Store: A column-oriented DBMS. In International Conference on Very Large Data Bases (VLDB), Sept. 2005, 553--564. Google Scholar
Digital Library
- Stonebraker, M., Madden, S.R., Abadi, D.J., Harizopoulos, S., Hachem, N., and Heiland, P. The end of an architectural era (it's time for a complete rewrite). In International Conference on Very Large Data Bases (VLDB), Sept. 2007 1150--1160. Google Scholar
Digital Library
- Zhou, J. and Ross, K.A. Implementing database operations using SIMD instructions. In ACM SIGMOD International Conference on Management of Data (SIGMOD), June 2002. Google Scholar
Digital Library
- Zhou, J. and Ross, K.A. Buffering database operations for enhanced instruction cache performance. In ACM SIGMOD International Conference on Management of Data (SIGMOD), June 2004. Google Scholar
Digital Library
Index Terms
Breaking the memory wall in MonetDB





Comments