At QCon San Francisco Idit Levine presented "Debugging Containerized Microservices", and outlined the issues of debugging a distributed microservice-based system, and provided three potential approaches to overcoming the inherent challenges. The talk also introduced a new open source microservices debugger that Levine is working on, Squash, which integrates with the VS Code IDE and allows developers to debug microservices applications deployed onto Kubernetes.
Levine, Founder and CEO of solo.io, began the talk by stating that debugging microservices is currently very difficult. A monolithic application typically consists of a single process, and attaching a debugger to this process allows the viewing of the complete applications state and the flow of execution. A microservice-based application is inherently a distributed system consisting of multiple processes communicating over a network, and this adds significant complexity to the challenges of debugging effectively.
The remainder of the talk presented three potential approaches to debugging microservices: distributed tracing, using the open source Squash microservices debugger that Levine has created, and using a service mesh.
Distributed tracing tooling, such as Open Zipkin -- which implements the Open Tracing Cloud Native Computing Foundation (CNCF) hosted API specification -- can be used to monitor and understand the flow of execution through a microservices-based application. The advantages of this approach are that it is easy to output to any logging tool, even from OSS components; it enables critical path analysis, and developers can drill down into request latency and other associated trace context metadata in very high fidelity; and operators can conduct system topology analysis and identify bottlenecks due to shared or contended resources. Disadvantages to this approach include that tracing does not allow run-time debugging or modification of application state; the approach often requires wrapping/decorating and changing the code, which can also incur a performance penalty at the runtime; and there is no holistic view of the application state - developers can only see what was printed.
The second approach presented was demonstrated via the open source Squash microservices debugger that Levine and here team at solo.io are working on. Squash currently only supports debugging of microservice applications within CS Code that are written in a language that can be debugged by Delve (Golang) or GDB (C++, Objective C, Java etc) and that are deployed to the Kubernetes container orchestration platform -- although more IDE, language and runtime platform support is planned in the future (and community contributions are encouraged).
The Squash architecture consists of a Squash server that is deployed and runs on Kubernetes, which holds the information about the breakpoints for each application, orchestrates and controls the squash clients. The Squash clients deploy as daemon set on Kubernetes node, and the client wraps as docker container, and also contains the binary of the debuggers. Squash uses an IDE as its user interface -- currently only VS Code --- and after installing the Squash extension, Squash commands are available in the IDE command palette.
The third approach presented for debugging microservices was to utilise the capabilities of a service mesh, such as Istio / Envoy. A service mesh data plane, such as Envoy, touches every packet/request in the system, and is responsible for service discovery, health checking, routing, load balancing, authentication/authorization, and observability. The service mesh control plane, such as Istio, provides policy and configuration for all the running data planes in the mesh. These properties provide ideal points of introspection and execution flow control, although outside of Istio's integration with the Open Zipkin and Jaeger distributed tracing systems and centralised logging, currently not many debugging tools exists within this space.
Levine concluded the talk by suggesting that the ultimate solution would be to integrate all three of the debugging approaches mentioned, and encouraged the audience to get involved via the solo.io Slack channel, and contribute to the work she is doing with Squash.
The slides for Idit Levine's talk "Debugging Containerized Microservices" (58Mb PPTX) can be downloaded from the QCon SF website, and the Squash microservice debugger code can be found within the project's GitHub repository. The video for the talk will be made available via InfoQ over the coming months.
