Meta has recently released DCPerf, aiming to provide a representation of the diverse workloads found in data center cloud deployments. This collection of benchmarks is expected to be a useful resource for researchers, hardware developers, and internet companies, helping the design and evaluation of future products.
In a blog post announcing the release, Abhishek Dhanotia, Wei Su, Carlos Torres, Shobhit Kanaujia and Maxim Naumov from Meta highlighted the unique nature of hyperscale and cloud datacenter workloads. They emphasized that these workloads, which dominate the server market, differ significantly from those in high performance computing (HPC) or traditional enterprise settings. This differentiation requires tailored server design and evaluation approaches, implying the need for specialized benchmarks.
DCPerf is a benchmark suite designed to mimic real-world hyperscale cloud applications, offering hardware vendors, system software developers and researchers a tool to evaluate new products and conduct performance projection and modeling. This approach reflects actual production workloads developed by internet application companies and deployed in hyperscale cloud datacenters.
The team at Meta employed various techniques to ensure benchmark representativeness, ranging from examining low-level hardware microarchitecture features to analyzing application and library usage profiles. This approach allowed them to capture the critical characteristics of production workloads and incorporate them into DCPerf.
By utilizing these benchmarks, the design and optimization of hardware and software on future server platforms will more directly translate into improved efficiency in hyperscaler production deployments.
Source: DCPerf: An open source benchmark suite for hyperscale compute applications
Meta has ensured compatibility with various instruction set architectures (x86, ARM), validated benchmark effectiveness for emerging trends (e.g., chiplets), and added multi-tenancy support to leverage increasing core counts on modern servers.
When this announcement was shared on Hacker News, the tech community noted its similarity to Fleetbench, a benchmarking suite tailored for Google's workloads. Fleetbench's C++ code is designed to aid chip vendors, compiler researchers, and others looking to improve performance on workloads similar to those used by Google.
Meta has been utilizing DCPerf internally, alongside the SPEC CPU benchmark suite, to enhance product evaluations and guide configuration choices for their data centers. This approach enables early performance projections for capacity planning, aids in pinpointing performance issues in both hardware and software and fosters collaborative platform optimization with hardware partners.
DCPerf's collection of application software provides a more comprehensive understanding of platform performance compared to conventional benchmarks like SPEC CPU. Recognizing these benefits, Meta has integrated DCPerf into their platform selection process for data center deployments.
DCPerf demonstrates strong applicability in evaluating and optimizing CPU performance, IPC, memory latency, and, to a certain extent, memory bandwidth and power. However, its usefulness for network and storage evaluations is more limited and dependent on the specific workload. The blog post highlights specific areas where DCPerf might need further development or where users should exercise caution when interpreting results.
Meta has thanked its collaborators for their support and contributions in utilizing DCPerf. Interested readers can check out the project on GitHub for more details.