Michael Bryzek, co-founder and ex-CTO at Gilt, discussed at QCon New York how ‘dependency hell’ could impact the delivery and maintenance of microservice platforms. Bryzek suggested that dependency hell may be mitigated by making API design ‘first class’, ensuring backward and forward compatibility, providing accurate documentation, and generating client libraries.
Quoting the Wikipedia entry for dependency hell, Bryzek introduced the phrase to the audience and suggested that almost every developer has witnessed this at some point in their career:
Dependency hell is a colloquial term for the frustration of some software users who have installed software packages which have dependencies on specific versions of other software packages
Bryzek discussed how Gilt.com, a luxury brand flash sales ecommerce site, moved from a traditional Ruby on Rails monolithic stack to a Scala-based microservice platform over the course of five years. This, combined with the growth of the technical team to over 150 people, lead the team to ask how dependencies should be managed, specifically dependencies that were used in libraries.
Four strategies for minimising the pain associated with dependency management were presented:
- API design must be first class
- Backward and forward compatibility
- Accurate documentation
- Generated client libraries
Bryzek proposed that several of these strategies can be implemented with appropriate tooling, and accordingly he has created a Scala-based tool named apidoc for this purpose. apidoc has no external dependencies on an existing software process nor runtime, and is open source and provided as a free to use software as a service (SaaS). Bryzek commented that over 100 services have already been built at Gilt with apidoc.
Diving into each of the strategies mentioned above, Bryzek proposed that the design of an API and the associated data structures are often the hardest things to change within a system. Therefore, they should be designed up-front, and the corresponding artefacts integrated into the design process to ensure accuracy between a service and associated clients. This is in essence, “schema first design”.
In regards to ensuring backwards compatibility of an API or client library, Bryzek suggested that developers should imagine processing historical records whenever a change to an API is made. Fields added to an API should either be optional or have sensible defaults, and renaming of fields should never occur - instead new models should be introduced, and a migration path be provided where appropriate.
Forwards compatibility is also an important concern, and for this scenario developers should imagine processing new messages arriving to the service with new data. The benefits of Postel’s law, ‘be conservative in what you do, be liberal in what you accept from others’ are relevant to this discussion. APIs (and corresponding client libraries) should be versioned to aid developers, and one such technique for versioning is semantic versioning.
Additional considerations for establishing forwards compatibility include being careful when using enums, for example when using binary interface definition languages such as Apache Avro or Apache Thrift, and being careful when throwing errors if an unknown field is encountered within an API’s data structure. Ideally tooling should clearly indicate when things have changed within an API or client library, and the apidoc tool provides this functionality.
Accurate documentation for an API is also essential, and this may be automatically generated, for example using a tool such as Swagger 2.0. Bryzek also discussed that although he was initially sceptical, he now believes that generating client libraries automatically from an API can be beneficial. For example, boilerplate API consumption code can be removed (or encapsulated within the client library), and consistent naming for fields and actions can be maintained across services. However, care should be taken to ensure that the client library has minimal external dependencies (so that the dependency hell is not pushed down into the stack), and generating a client that developers will enjoy using can be very challenging.
Additional details about Michael Bryzek’s “Microservices and the art of taming the Dependency Hell Monster” talk can be found on the QCon New York 2015 website, and more information about the apidoc tool can be found on the project’s Github website and the Gilt Tech blog.
