Cockroach Labs recently released their annual cloud report which evaluates the performance of AWS, Microsoft Azure and Google Cloud for common OLTP workloads. Differently from the past, this year’s report does not indicate a best overall provider, but concludes that AMD instances outperform Intel ones. ARM instances were not covered in the tests.
According to the report, all the main three cloud providers offer similar price-competitive options for critical applications and workloads. Performing over 3000 benchmark runs on 56 different instance types and 107 discrete configurations, the price-for-performance leaders were AMD-based instances running Milan processors. Keith McClellan, director of partner solutions engineering at Cockroach Labs and the lead author of the Cloud Report, writes:
It's like the late 1990s again, with AMD beating out Intel on pure performance... I think the last time this happened AMD was still selling chips called "Athlon"!
The lack of coverage for ARM instances and the most recent instance types, leaves questions unanswered for use cases where for example Graviton instances on AWS could be a better fit. McClellan acknowledges it:
ARM is going to be making a comeback in the 2023 Cloud Report - unfortunately we didn't have supported ARM binaries for CockroachDB ready in time for the report this year.
Not every analysis agrees on the benefits of AMD instances for database workloads. The benchmark "Economical Comparison of AWS CPUs for MySQL (ARM vs Intel vs AMD)" performed by Percona, concludes that Graviton is cheaper in most cases and usually Intel performs better than AMD on AWS.
According to Cockroach Labs, small instances outperform large ones. In both the OLTP and CPU benchmarks, the report shows a per-vCPU performance advantage to running on smaller instances of a particular instance type, regardless of CPU platform, cloud, or instance type.
Processors and CPU benchmarks were not the only focus. The report stresses the importance of the storage and transfer costs and how they have a significant impact on total cost to operate:
For even relatively small amounts of persistent block storage, the cost of running a particular workload is much more influenced by the cost of the storage than it is the cost of the instance. For persistent workloads, it is extremely important to optimize cost calculations based on this consideration.
The benchmark highlights that it is generally not worth paying for high-performance storage. For the same processor, cloud providers offer different instance classes, with different vCPU to RAM ratio. In the report, the authors conclude:
Our tests suggest that while you may save a bit of money by choosing instance types with a lower vCPU to RAM ratio, you will likely see more consistent performance from instance types with more available memory. The impact of this decision is more apparent with larger, more complicated workloads. In our tests, we found the sweet spot to be a vCPU:RAM ratio of 1:4.
Source: https://www.cockroachlabs.com/guides/2022-cloud-report/
Compared to the previous edition, the 2022 Cloud Report added testing for different-sized instance types, testing cross-region latency, storage tests with fsync and variable OLTP workload complexity testing. As the framework with all the resources and configurations is available on GitHub, cloud architects can perform further benchmarks with newer processors and different storage options.
Access to the full report is free but requires registration.