Lyft shared its experiences using Protocol Buffers for inter-system integration, primarily focusing on collaborative protocol design for definitions shared between teams and systems. The company promotes approaches that improve knowledge sharing, consistency, and development process quality over raw efficiency optimizations.
Lyft has been a long-time adopter of Protocol Buffers, and the company explained the reasons for moving from HTTP+JSON in 2020, listing out the descriptiveness of ProtoBuf as an Interface Definition Language (IDL), the availability of open-source code generators for most languages, the optimized binary format, and the ability to be backward compatible with JSON APIs.
Having used Protocol Buffers for many years across three language stacks (Python, Swift, and Kotlin), engineers gained experience in protocol definition design, particularly when working across teams and systems, often with colleagues not versed in using Protobuf. Roman Kotenko, software engineer at Lyft, called out key principles for protocol design:
- Clarity: A well-designed protocol should define its messages in a way where it’s not only explicit about which fields must be set. This prevents missetting any of the messages during implementation. In other words, good protocols leave no ambiguity for its implementers.
- Extensibility: It is crucial that protocol structure is built with future vision and potential roadmap in mind. This way, some foreseeable additions and breaking changes can be accounted for in advance.
Furthermore, in his post, Kotenko highlighted a few best practices that Lyft considered crucial to successfully adopting Protocol Buffers as the Interface Definition Language (IDL) for an extensive, distributed system.
The author emphasizes the benefits of using Protocol Buffers' message validation functionality. The project protovalidate is a recommended way to validate messages at runtime based on user-defined validation rules. It is a successor to the protoc-gen-validate plugin (created and, for now, still used at Lyft) and is powered by Google’s Common Expression Language (CEL). The solution supports many validation rules that can be applied to messages and fields of different kinds, like unions (oneof), enums, collections (repeated), maps, and scalars, including wrapper types.
Lyft teams established an approach for unifying constant values used in protocol definitions across many entities. Kotenko demonstrates how custom options can be used for that purpose but warns about caveats in adopting the solution, including the requirement for complete control of protocol definition users (for instance, when the protocol is used internally).
The example of Protocol Buffers Message Definition (Source: Lyft Engineering Blog)
Other best practices reflect those included in the official documentation, such as using well-known data types for common use cases, including unknown enum values, and explicitly marking optional fields leveraging the new "optional" label introduced in Protobuf 3.15.
Lastly, the author pointed out that reviewing language-specific tutorials is vital to learn about the required setup and any nuanced details, like type mappings and the generated code features, as certain behaviors differ between languages.