A recent set of benchmarks compares Aerospike, Cassandra, Couchbase and MongoDB to see how they fare when it comes to insert throughput, maximum throughput, latency and behavior during a failover.
Thumbtack Technology has released two benchmark whitepapers with results from the comparison of several key-value stores: Ultra-High Performance NoSQL Benchmarking: Analyzing Durability and Performance Tradeoffs (PDF) and NoSQL Failover Characteristics: Aerospike, Cassandra, Couchbase, MongoDB (PDF). Both benchmarks attempted to test “consumer-facing applications which require extremely high throughput and low latency, and whose information can be represented using a key-value schema.”
Thumbtack used an improved version of Yahoo! Cloud Serving Benchmark (YCSB) one that is supposed to overcome some limitations reached when using very high volumes and multiple clients. The YCSB changes have been documented in the first whitepaper and committed back to the community.
The NoSQL databases tested were Aerospike, Cassandra, Couchbase (1.8 and 2.0), and MongoDB. The first is a commercial product, and the last is a document data store not a key-value store, but it was included because “in our experience clients often consider it for similar kinds of applications.” All databases were optimized using recommendations from the vendors backing them. The test systems used SSD storage rather than rotating disks. The whitepapers contain detailed information regarding the methodology used, the client and workload configuration, the hardware configuration, etc.
Thumbtack acknowledged having “strategic and/or commercial relationships with Aerospike, Couchbase, and 10gen” and the hardware used was rented from Aerospike.
We are including some of the results of the benchmarks.
Insert Throughput
The databases were loaded with initial working sets using YCSB’s loading routing which performs a number of inserts. Couchbase had good results when the working sets were loaded into memory, but it had problems loading on SSD when Couchbase 1.8 did not finish the operation and a smaller set and asynchronous mode had to be used for Couchbase 2.0. That explains the gradient blue used for it. Aerospike came second.
Maximum Throughput
This test used a “a strong durability model, using a dataset that, when replicated, would be significantly larger than the server’s RAM. This test is intended to model usage for transactional data that requires strong durability guarantees.”
Couchbase does not appear on the graphic because it could not complete the test using synchronous replication.
When asynchronous replication was applied, the result in memory was:
Latency/Throughput
The benchmarks also measured read and update latency at different levels of traffic. The next graphic includes a full view and a zoomed one for each of those:
Failover
Thumbtack attempted to see what happens when a node goes down, a hardware failure being simulated:
The downtime was also measured, i.e. the time needed for the cluster the become responsive again after a failure, all databases showing reasonable values:
The Thumbtack benchmarks contains more results for different cases but were not included here.
Another NoSQL benchmark was published in October 2012 when Cassandra, HBase, MongoDB, and Riak were compared. MySQL was also included in those tests for a reference against SQL technologies.