Jean Barmash is the director of engineering for Compass, a technology firm focussed real estate brokerage based out of New York City. Prior to Compass he was the VP of Engineering for Trade Shift where he was building third party developer platforms. Jean is also a co-founder of the CTO school meet up in NYC and is a programme committe member of QCon New York.
Today’s chat talks about binary communication protocols, such as Apache Thrift and Google’s gRPC, as well as code generation and design of APIs, concluding with lean and constraint theory based on software and people systems.
Key Takeaways
- Migrating from Apache Thrift to gRPC is possible if you extend gRPC to understand Thrift messages
- Adding languages into a stack needs to be done carefully but can bring developer productivity
- Schema changes and evolution needs to be built in from the start to avoid problems migrating later
- Designing public APIs is much like designing private APIs, but with a harder constraint of not being able to change them in future
- Constraint theory and lean development can be used to identify failures quicker and unblock development effectively
Subscribe on:
Show Notes
What is Compass?
02m:35s Compass is a real estate brokerage based on today’s technology, from marketing to implementation.
03m:20s One of the things we’ve been getting interest for recently is a collaboration tool that allows agent and customers to work on their search.
What does the stack look like?
04m:40s The back end is mostly Java and Python, with Go introduced a year and a half ago.
04m:50s On the front end we are using Angular, and Swift 3 on iOS devices.
05m:10s We added Go as another part of the stack, due to increased developer productivity.
05m:40s On the management side I was hesitant about moving to Go, because of adding a third language in the stack, and because Java is here to stay.
06m:00s Some of the newer stuff, particularly small services, are being written in Go.
How is the developer’s day to day affected by introducing a new language?
06m:25s You find that when debugging implementations in different languages that the system is integrated slightly differently, from logging to telemetry.
06m:50s One of the arguments for introducing Go is that it’s a bit simpler; you can give the code to your front-end developers, where Java may be harder to read.
What about the architecture of Compass?
07m:15s We are a microservices architecture, using RPC to communicate between services.
07m:30s We started out with Thrift and Finagle, primarily because of familiarity with the system from some of the founding engineers.
07m:35s Over the last six months we have been moving towards gRPC.
07m:50s We used the flexibility of gRPC to allow communication with Thrift, which doesn’t understand Protobuf (gRPC’s native transport mechanism).
08m:30s This allowed us to take advantage of the newer model to represent communication in gRPC while keeping backward compatible protocols for existing systems.
09m:00s It took around a month to integrate the back end, as well as open-sourcing a Thrift linter.
09m:20s The underlying programming model was similar between Thrift and gRPC: they provide a binary protocol across the wire along with error handling to call RPC services over a remote end.
09m:50s If we had been using REST based systems instead, it would have been much harder to move from a REST based model to an RPC based one.
You mentioned you were using Finagle and Thrift - what made you move?
10m:55s At one point we realised that we weren’t using the full functionality of Thrift, but were interested in exploring more about gRPC.
11m:25s We’re seeing gRPC as analogous to Thrift, but having some additional functionality around error handling and an increased momentum towards gRPC.
12m:00s Performance wasn’t a key driver: we didn’t measure and didn’t have any problems in the migration.
12m:20s It was driven more by developer experience rather than implementation.
What did the developers think?
12m:40s We’re still in the middle of rolling this out, and some of the developers haven’t moved because of training - we’re about 80% converted to gRPC but are still working on the remainder.
13m:15s There are about 40 to 50 back end microservices that we’ve got at the moment.
What about schema changes and evolution?
13m:50s There are a certain type of problems that you’d get using REST that go away if you’re using schema evolution.
14m:00s If you’re using REST without a framework for schema representation like RAML or code generation frameworks then you do need to think about versioning.
14m:20s With Thrift and Protobuf there are forwards compatible mechanisms which allow you to evolve the schema forwards.
What tool did you write to parse Thrift and why?
15m:15s Because we use Thrift internally, we have a lot of structured APIs already. We used Thrift’s IDL to generate code to be able to call these services.
15m:45s Another reason we moved to gRPC is to allow using our frameworks in other languages that we support, such as Python and Go (which aren’t well supported by Thrift, which is mainly a Java framework).
16m:15s The tool looks through the Thrift files, and annotations embedded within that cover the REST endpoints and use a template to generate code for languages like Java/Kotlin, JavaScript and Swift - as well as RAML for documentation.
17m:10s We don’t have public services, but the mobile platforms use RESTful services to interact with the servers - but the back end servers all speak gRPC between them.
How does designing public and private APIs differ?
17m:50s Ideally there wouldn’t be big differences between public and private APIs, because you’ll have developers consuming both of them.
18m:05s The focus is on how to simplify the APIs, how to make them feel natural and match the mental models that the developer’s have.
18m:20s In practice when an API is being opened up for public consumption, so you should focus on simplifying the model much more beforehand, because fixing the API after it goes public is much more difficult.
What kind of things do you think of when designing a public API?
18m:55s Starting from an internal API gives a good basis to build upon.
19m:15s Think of what concepts you want to offer to the users, and decide what the extensibility model should look like.
20m:05s It’s always good to start with the client’s feedback, as well as thinking about versioning and evolving the API up front.
What’s the CTO school in New York?
20m:50s The goal is to educate and bring together technical leaders to improve their skills.
21m:20s From the management side - both product and engineering management - it gives you a framework to think about decisions, such as velocity and how to optimise it.
What’s your recommendations?
21m:40s I recommend Don Reinertsen’s “Principles of product development flow” - when I read that, I felt like I really understood some of the theory behind agile and some of the different flavours.
22m:10s When you set up cadences (e.g. weekly, monthly meetings) it removes overhead about scheduling that meeting.
22m:30s A lot is about breaking things up into smaller chunks - the asymmetry of innovation means that as long as you can cut your losses, that’s a finite loss you can measure and cut off. Keeping in mind that symmetry in mind for software is important.
23m:10s I’m reminded of a talk by Roy Raporport at QCon who went through ten different projects in the last 18 months, and ranked which ones were successful and which were failures.
23m:30s About 50% of them were failures, so he asked the question whether that was a good percentage or not?
23m:45s For the losses, they were contained (a month of development) but for the wins, they were continuing to bring value for months afterwards. So that highlights the asymmetry between them.
How do constraint theory and lean product development relate?
26m:00s There are a bunch of related disciplines around lean product development, constraints theory and systems development.
26m:15s With constraints theory, there is one key constraint which impedes throughput, which can apply to computer systems or people teams.
26m:35s For example, maybe your team doesn’t have enough front end developers - which means the team will be constrained from developing front ends.
26m:55s The first step is to identify the constraint, then the constraint can be managed and appropriately dealt with.
27m:10s For example, if you only have one front-end developer, how do you maximise their experience with working on front-end systems?
27m:20s Do you help other back-end developers work and take the load of the front-end developer in certain situations to free up their front-end time?
27m:30s If you have 30 engineers and only one front-end developer, perhaps it would be more effective to bring in a higher paid consultant to speed up the front-end work to unblock the progress of the team as a whole.
What can you tell us about the architectures talk at QCon New York?
29m:20s Some of the things we hope to have this year is to have a great talk on IoT.
29m:40s Another talk will be focussed on event driven architectures in a large scale production use cases.
30m:05s Some of the IoT content will be about sensor data, along with server-side processing, thinking about a particular domain such as kitchen restaurants.
30m:40s This all helps optimise the throughput of the system, making food appear quicker based on the data collected.
31m:15s There’s also a collection of failure conditions, so the system needs to be resilient in case of central failure.
Languages Mentioned
Frameworks mentioned
Angular
gRPC
Thrift
Finagle
Further reading
Don Reinertsen’s “Principles of product development flow”
