Uber Engineering recently announced an upgrade to their search infrastructure, transitioning from Apache Lucene 8.0 to version 9.5. This upgrade improves Uber's search capabilities, performance and efficiency across their various services.
Anand Kotriwal, Aparajita Pandey, Charu Jain and Yupeng Fu from Uber's Search Platform and Data engineering team elaborated on the upgrade effort in a blog post. Uber’s search platform is built on a solid architecture that includes both a serving layer (read path) and an ingestion layer (write path), along with components for offline processing.
The serving layer handles user queries and retrieves information from Lucene indices. It consists of two main parts: the Routing Service, which directs incoming queries to the appropriate search nodes and manages load balancing, and the Search Service, which executes queries against Lucene indices to retrieve results in real-time.
The ingestion layer updates Lucene indices when there are changes in the data. The Ingestion Service, powered by Apache Flink, processes real-time updates to ensure the search indices remain up-to-date.
For offline processing, Uber uses Offline Jobs. These are Apache Spark jobs that handle bulk index creation and reconstruction, efficiently processing large volumes of data to build or rebuild Lucene indices.
Source: Lucene: Uber’s Search Platform Version Upgrade
The engineering team made changes in a separate feature branch, affecting over 400 files in the monorepo, which were incompatible with the current codebase. To manage this, the team opted for a phased rollout of the Lucene update. They began by deploying it to lower-priority internal use cases and gradually expanded to higher tiers.
The entire process took around six months, involving thorough code reviews, validations, collaboration with customer teams, and a tiered rollout before merging the branch.
Apache Lucene is a Java-based search engine library. It supports various search needs, including structured and full-text search, faceting, nearest-neighbor searches, spell correction, and query suggestions. They also have a sub-project PyLucene, which provides Python bindings for Lucene Core.
Recently they have released version 10, that adds a new IndexInput prefetch API, support for sparse indexing on doc values, and upgraded Snowball dictionaries resulting in improved tokenization.
The upgrade has improved Uber's search speed and efficiency. Searches run faster and use less resources, which means quicker results for app users. The team has reported some searches are now up to 30% faster than before. This speed boost affects both riders and drivers using the app.
Another benefit of this upgrade is its impact on Uber's infrastructure. Uber's search requests rely heavily on CPU power, so lowering CPU usage is critical. This reduction helps cut infrastructure costs and allows Uber to reduce the number of machines needed to serve multiple customers.
Uber's Lucene version upgrade underscores the importance of keeping core technologies up-to-date in large-scale, technology-driven operations.