FMKe: a Real-World Benchmark for Key-Value Data Stores

ABSTRACT

Standard benchmarks are essential tools to enable developers to validate and evaluate their systems' design in terms of both relevant properties and performance. Benchmarks provide the means to evaluate a system with workloads that mimics real use cases. Although a large number of benchmarks exist for database system, there is a lack of standard benchmarks for an increasingly relevant class of storage systems: geo-replicated key-value stores providing weak consistency guarantees. This has led developers and researchers to rely on ad-hoc tools, whose results are both hard to reproduce and compare.

In this paper, we propose the first standardized benchmark specially tailored for weakly consistent key-value stores. The benchmark, named FMKe, is modeled after a real application: the Danish National Joint Medicine Card. The benchmark is scalable, it can be parameterized to emulate a large number of access patterns, and it is also highly flexible, enabling its application on systems that offer different consistency guarantees and mechanisms.

References

1. The Transaction Processing Performance Council, Benchmark C. http://www.tpc.org/tpcc/default.asp. Accessed: 2017-02-15.
2. The Transaction Processing Performance Council, Benchmark C. http://www.tpc.org/tpcw/. Accessed: 2017-02-15.
3. Fusion Ticket Solutions Limited. https://github.com/fusionticket. Accessed: 2017-02-16.
4. Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC '10, pages 143--154, New York, NY, USA, 2010. ACM.
5. AntidoteDB. http://antidotedb.eu. Accessed: 2017-02-15.
6. FMKe code repository. https://github.com/goncalotomas/fmke. Accessed: 2017-02-15.
7. Basho Bench. https://docs.basho.com/riak/kv/2.2.0/using/performance/benchmarking/. Accessed: 2017-02-15.