At KubeCon NA held in Seattle in December 2018, several tools for debugging containerised microservices were presented throughout the conference sessions and the sponsored booths demonstrations. A notable separation appears to be occurring within the market, between "active" and "passive" debugging tools. Two examples within these categories are Squash and Rookout, respectively.
Active debugging is what developers would typically expect from a debugging tool and usage session i.e. the ability to set breakpoints, stop and step through application execution, and watch and dynamically mutate variables. Active debugging is challenging to implement within a microservices-based application because each application can be written in a different language, and application execution flow can encompass multiple operating systems processes that may span multiple network boundaries. On the positive side, as this general approach has existed for quite some time, the debugging tooling and associated functionality are typically provided as part of the language SDK or included within IDEs, and no modification to the code or associated dependencies is typically required.
Passive debugging could be looked on as advanced logging and tracing, because although developers can set breakpoints, add watches, and dynamically add logging statements, they cannot typically halt the program execution -- they are merely passive observers. The advantages of this approach is that debugging can be unobtrusive (i.e. a user's request to an application is not paused or blocked during the debugging process), and the cycle of hypothesis identification and testing via setting multiple breakpoints can be rapid. However, typically this approach does require the modification of a codebase or the inclusion of a library or SDK, and the execution of this dependency could be placed within application's critical path.
InfoQ first watched the tooling demonstration at the Rookout booth, and chatted afterward with several engineers on the team. Rookout is a commercial SaaS-based "passive" microservice and serverless debugger, which allows the addition of dynamic breakpoints and logging statements (among other actions) against code running within an arbitrary environment. All debugging configuration can be specified via an IDE-like web interface, which also shows the code and resulting debug information.
Rookout web UI with "breakpoint" set in top window, and debug info showing in bottom (image source: Rookout Demo Video)
The Rookout team describe their breakpoint functionality as "non-breaking breakpoints", as the corresponding application execution does not actually pause or halt as it would with a traditional active debugger. They also state that "no restarts, redeployment or extra coding is required" in order to set these breakpoints, and this can lead to very quick hypothesis testing and bug detection.
As a result of a Rookout breakpoint being hit within an application's execution flow, an engineer can view stack traces and global variable values, as well as specify individual variable "watches". InfoQ learned that in the case of Java debugging, the underlying mechanism that provides the breakpoint functionality is based on java.lang.instrument, which allows Java programming language agents to instrument programs running on the JVM. The instrumentation of an application is accomplished by adding a Rookout dependency to the codebase e.g. via Maven or Gradle.
InfoQ also sat down with Idit Levine, founder and CEO at solo.io, and discussed the open source microservice debugger Squash, and a related debugger for Kubernetes applications named KubeSquash. Levine notes that current debugging of microservices is often assisted by two approaches: first, the use of distributed tracing, particularly OpenTracing, which helps with the tracing -- with associated debug metadata or "baggage" -- of a transaction or workflow for post-mortem analysis; and second, by using a service mesh like Istio which monitor the network to identify latency problems. These tools, however, support only passive debugging, and do not allow developers to "actively" halt execution and mutate variables within the application during runtime.
Squash aims to "bring the power of modern popular debuggers to developers of microservices apps that run on container orchestration platforms". Using a proxy configured to support two-way traffic, such as Envoy, Squash bridges the gap between a engineer's local IDE and applications running on the orchestration platform. Debuggers such as gdb, delve and JDWP Java tooling can be run locally and attached to remotely running applications. Engineers can choose which containers, pods, services or images they require during debugging, and can set breakpoints in their code, follow the value of variables in-flight, step through the code while jumping between microservices, and change these values during run time.
A series of slides from Levine's KubeCon NA 2017 presentation demonstrates the process of debugging with Squash:
KubeSquash is a Kubernetes-specific project that appears to share a similar history as Squash. KubeSquash provides a command line tool "ksquash" that supports initiating a debugging session using either gdb or dlv.
KubeSquash is extremely easy to launch, and requires no server. Its user interface is dead simple: invoke with a single command ksquash, target the desired pod, and the debugging session is initiated automatically with zero configuration or effort.
Other tooling that exists within the active debugging space includes: the CNCF-hosted Telepresence, and Philip Lombardi, engineer at Datawire, presented a related session "Effective Development with Kubernetes: Techniques, Tools & Telepresence"; and txn2's kubefwd, for which Kelsey Hightower recently tweeted his support. Several other developer workflow tools either currently support, or are in a position to support, this type of functionality, such as Garden, Tilt and Skaffold.
It is also worth noting that the distance between what the industry is now referring to as "observability" tooling and passive debugging tooling within the microservice and container based domain is clearly shrinking. InfoQ has covered examples of this from the Google, Facebook, Lyft, Honeycomb, and LightStep teams.