Grafana has released Grafana Beyla, an open-source eBPF auto-instrumentation tool for application observability. Beyla is able to report span information and RED metrics (Rate-Errors-Duration) for both Linux HTTP/S and gRPC services. This is accomplished without having to make code modifications for inserting probes.
Grafana Beyla supports HTTP and HTTPS services written in Go, NodeJS, Python, Rust, Ruby, .NET, and others. For Java applications, only HTTP is currently supported. gRPC services written in Go are also supported.
After installing Beyla, a running service on that host can be instrumented by providing details about the executable along with a metrics exporter. The executable can be selected by either its name (using the environment variable EXECUTABLE_NAME) or by any open port in the application (using the environment variable OPEN_PORT). The following example configures Beyla to instrument an application that owns port 8080 while sending traces to standard output and RED metrics to localhost:8999/metrics:
$ BEYLA_PROMETHEUS_PORT=8999 PRINT_TRACES=true OPEN_PORT=8080 sudo -E beyla
Querying the service will return logs in the format
Request_time (response_duration) status_code http_method path source->destination request_size
The EBPF tracer instruments the process, creates the traces, and passes them along to the routes decorator. By defining routes in the routes decorator, HTTP paths (e.g., /user/1234/info) can be mapped into HTTP routes (e.g., /user/{id}/info). From there the data is forwarded to the various exporters and endpoints.
Metrics can be exported in both OpenTelemetry and Prometheus formats. The OTEL metrics exporter exports data to an external OpenTelemetry metrics collector. The OTEL traces exporter can be used to send span data to an external OpenTelemetry traces collector. The Prometheus HTTP endpoint creates an HTTP endpoint to allow for scraping of metrics in Prometheus format. Finally, internal metrics for Beyla can be pulled in Prometheus format.
Beyla can be run as a standalone Linux process, using Docker, or as a Kubernetes sidecar container. The Docker approach can be used to instrument processes running inside containers. Linux with a Kernel of 4.18 or higher is required and eBPF must be enabled in the host.
The concept of RED metrics was coined by Tom Wilkie, CTO at Grafana, in 2015. Wilkie created it after realizing that the USE method wasn't applicable to services. The USE method looks to monitor utilization, saturation, and errors for a given resource. In the RED method, the focus is on monitoring the rate (number of requests per second), errors (number of those requests that are failing), and duration (amount of time requests are taking). As Wilkie notes:
The RED Method is a good proxy to how happy your customers will be. If you’ve got a high error rate, that’s basically going through to your users and they’re getting page load errors. If you’ve got a high duration, your website is slow. So these are really good metrics for building meaningful alerts and measuring your SLA.
Grafana Beyla is available in public preview and can be downloaded from GitHub under the Apache 2.0 license. More details about the release can be found on the Grafana blog and in the Beyla documentation.