BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Uber Cuts Translation Cache Memory by up to 90% While Keeping P99 Latency Below 1 MS

Uber Cuts Translation Cache Memory by up to 90% While Keeping P99 Latency Below 1 MS

Listen to this article -  0:00

Uber has redesigned its Go-based translation cache to reduce its initial memory footprint by more than 90% while keeping p99 cache miss latency below one millisecond. The hybrid architecture retains translation keys and frequently accessed values in memory while storing the remaining values on disk. The system serves millions of translations per second and maintains a cache hit rate above 99.9%.

The optimization addresses two problems identified through memory profiling across Uber's Go services. Garbage collection (GC) consumed more than 20% of CPU utilization in some services, with translations accounting for approximately 70% of allocated objects.

Uber Engineering noted that

At Uber's scale, even small inefficiencies in memory management can add up.

The first optimization replaced Go's sync.Map with a conventional map[string]string protected by an RWMutex. The workload was predominantly read-heavy, with infrequent updates and no individual deletions. The previous implementation created multiple objects and pointers for each translation entry, increasing GC overhead. Moving to a value-based map reduced object allocations and improved CPU utilization by more than 30% in the affected service.

However, reducing object allocations did not solve the overall memory consumption problem. Translation datasets could contain two to three million keys, although individual services typically accessed only a small subset. Cache refreshes introduced another issue: the service built a replacement cache in the background before atomically swapping it with the existing one. Both copies therefore occupied memory simultaneously, causing some services to run out of memory during refreshes.

Uber considered external caching and generating service-specific translation subsets, but both introduced additional operational or maintenance costs. The team instead adopted a hybrid storage design that preserves the existing Get() and Put() API. All keys remain available in memory for fast lookups, while translation values are stored in disk-backed files.

Uber's hybrid translation cache (Source: Uber Blog Post)

The implementation uses 256 shards, each with its own read-write mutex, in-memory entries, and disk file. Six-byte hash-based keys reduce metadata overhead, with a fallback to the original key format to handle collisions. A time-based global least recently used (LRU) policy manages in-memory entries. During cache refreshes, the system uses access history to preload translations accessed within the previous 15 minutes, avoiding a cold cache after the swap.

In one example, a translation dataset that previously consumed approximately 800 MB in memory required around 40 MB for metadata after the redesign. With a configured limit of 50,000 in-memory entries, actual memory consumption typically ranged between 80 MB and 120 MB. In another reported workload, approximately 30,000 of more than four million entries were loaded in memory, while the cache hit rate remained above 99.9%. The p99 latency for disk-backed cache misses stayed below 100 microseconds.

Memory usage decreased from approximately 2.7 GB to 750 MB (Source: Uber Blog Post)

Uber's experience highlights the cost implications of in-memory caching at scale. The team recommends minimizing object references to reduce GC overhead and measuring how frequently cached values are accessed. When only a small proportion of cached data is actively used, a hybrid approach can retain frequently accessed values in memory while moving the rest to disk.

About the Author

Rate this Article

Adoption
Style

BT