Netflix has developed an orchestration engine called “Conductor”, and has used it internally in production for the last year . During this time they executed some 2.6 million process workflows, starting with linear ones and ending with dynamic ones running over multiple days. Now they have open sourced Conductor, making it available to all those interested in workflow orchestration.
According to Netflix’ developer documentation on Conductor, the main characteristics of this engine are:
- The ability to create complex workflows
- The tasks are executed by microservices
- The workflow blueprint is written in a JSON DSL
- The engine provides visibility and traceability into process execution
- The ability to pause, resume, restart, stop tasks
- The task execution is generally asynchronous but it can be enforced to run synchronously
- The ability to scale to millions of process flows
The Conductor’s architectural diagram is shown below:
Both the API and storage layers are pluggable, allowing one to use different queue and storage engines. Netflix uses Dynomite for storage but one can switch to another solution as long as an interface is implemented.
Workflow tasks are of two types: Worker, which are user tasks executed on a remote machine, and System, executed on the engine’s JVM. The latter are meant to branch, fork, and join the execution of worker tasks. Worker tasks communicate with the Conductor via HTTP or gRPC on HTTP/2.
Netflix has decided to create an orchestration rather than a choreography engine because the latter did not scale well in their tests. The issues mentioned were:
- Process flows are “embedded” within the code of multiple application
- Often, there is tight coupling and assumptions around input/output, SLAs, etc., making it harder to adapt to changing needs
- Almost no way to systematically answer “how much are we done with process X”?
Conductor has been open sourced under an Apache License 2.0 on GitHub.