Uber has written about successfully implementing a new differential backup system for its distributed databases, which led to storage cost reductions of up to 70%.
Uber engineers developed this solution in response to growing storage costs and lengthy backup completion times that emerged after Uber migrated the storage for their Schemaless and Docstore services to MyRocks, a MySQL storage engine based on RocksDB.
At Uber, the Schemaless and Docstore distributed databases handle tens of petabytes of operational data and process tens of millions of requests per second. These databases are critical to Uber's global operations, supporting various business functions and ensuring business continuity.
Problems arose when Uber's Storage Platform team migrated to MyRocks, which, while optimised for write operations and storage efficiency, lacked support for incremental backups. This limitation meant that each database partition needed a full backup every time, which led to substantial duplicate data storage and increasing blob storage costs.
The new differential backup system leverages the immutable nature of MyRocks' SSTable files, which often remain unchanged between consecutive backups. Instead of copying all files during each backup, the system maintains a shared pool of SSTable files and only adds newly created files to this pool. A manifest file defines each backup by recording the list of included files, so restoration is efficient when needed.
According to Adithya Reddy, who detailed the solution in a technical blog post, the system works by first performing an initial full backup, which stores all metadata and SSTable files in a shared pool within the blob store. Subsequent differential backups only add new SSTable files to the pool while reusing existing files from previous backups.
The backup manifest file, implemented as a JSON document, tracks essential information, including the backup type, success status, timing details, and file checksums. This manifest serves as an index for the backup and provides the necessary information for restoration processes.
The new system is managed by a stateless service called Backup Scheduler. Backup Scheduler determines the timing and frequency of backups based on partition backup states. The actual backup process is handled by ephemeral backup containers that are activated when needed to execute the backup using the Percona XtraBackup tool.
While the system primarily performs differential backups, full backups are still required in certain situations, such as:
- When the backup is first run in production
- When the previous partition node becomes unsuitable
- Before version upgrades of MyRocks
- When the existing backup pool reaches its differential backup limit
The backup system continues to use the established Percona XtraBackup tool for both backup and restoration processes, maintaining compatibility with existing systems while adding the new differential backup capability. This approach allowed Uber to implement the improvement without requiring extensive changes to their broader database infrastructure.
Despite these constraints, the improvements have been significant. Uber reports an average 45% reduction in data storage across most instances, with some larger instances seeing 70% or more reductions in storage costs for each differential backup. The speed of the backup process has also improved significantly, with full backups completing twice as fast as before and differential backups showing a fivefold improvement in completion time.
MyRocks was initially developed at Facebook (now Meta) to add replication and an SQL layer to RocksDB. In 2016, Facebook moved its user database to MyRocks. RocksDB - the underlying storage engine behind MyRocks - has also been adopted by other organisations, with some users claiming three-fold performance increases, and Percona publishing benchmarks showing significant improvements over InnoDB. For some use cases, however, there have been reports of RocksDB underperforming compared to InnoDB for retrievals.