Instrumentation at Scale: Having Your Performance Cake and Eating It Too

Transcript

Brian Martin: My talk is, "Instrumentation at Scale, Having Your Performance Cake and Eating it Too." My name is Brian Martin. My pronouns are he/they. I am a co-founder at IOP Systems, which roughly stands for Infrastructure Optimization and Performance. My company, we were the infrastructure performance team at Twitter. Then we spun off and are our own company. I work on a few open-source projects. One is called Rezolus, which is a systems performance telemetry agent that does like eBPF stuff. RPC-Perf, which is a benchmarking tool. I've worked on Pelikan, which is a cache backend. All three of these are written in Rust. Let me just talk a little bit about Rezolus first. It uses eBPF heavily so we can get instrumentation from the Linux kernel and dynamically probe things to get really fine-grained metrics. Besides the basic things like CPU, network, and disk, we're able to instrument the scheduler.

We're able to instrument syscall paths. We're able to get some really cool granular metrics about like the TCP stack and stuff like that as well. It offers Prometheus exposition, because that's the common way to consume telemetry. It's open source. It's available on rezolus.com, which just redirects to the GitHub. Check it out if you need eBPF systems telemetry.

I'm also going to be talking a little bit about a metrics library that I work on called metriken. It is low overhead metrics. It's used in all of the projects that I work on. It's really optimized for performance-critical paths. It was built out of necessity for getting metrics in RPC-Perf and Pelikan, and later we leveraged it in Rezolus as well. It uses a link-time hack to basically discover the metrics, so I can instrument libraries and get those metrics pulled together in the application. It's just a cool thing about how it works. Since I mentioned eBPF, what is that? It's the Extended Berkeley Packet Filter, but now it's used for a lot of other things where you can run sandboxed programs in the Linux kernel. This allows us to observe kernel behavior that we wouldn't be able to see otherwise without making any code changes to the kernel. We're able to attach to kernel functions, tracepoints, even user space functions, and we're able to share data back via BPF maps to user space, so very cool technique for getting detailed stuff from kernel space.

Why Instrumentation Matters

Let's begin our talk about instrumentation. Why does instrumentation matter? You need visibility into your systems to understand what's going on. Without enough instrumentation, you're going to miss performance issues, like you're just not going to know about them. You're not going to be able to diagnose production incidents. I think the worst page to get is not the system is down, but the system is slow. Instrumentation helps answer the question of why. Without enough instrumentation, you can't improve performance because you can't improve what you can't measure. Instrumentation is not optional. This is absolutely required for running production services. I think it's absolutely required when designing a library. It's very important. It's not without problems. The problem is always cost. There's always some overhead when you add instrumentation. The more that you're instrumenting, the more those costs matter. If it's instrumentation in a hot path, it's going to be very critical.

Basically, the number of times you're calling it is a multiplier on the cost. If it's very expensive and called infrequently, maybe that's ok. As you start calling it more, that cost matters much more. Let's talk about some real costs. This is the same operation with different implementations. We're going to get into the details about this later. This is a little teaser so you understand how important this is. Counter increments can range between 5 nanoseconds for an uncontended atomic to over 1 microsecond. That's a 200 times cost difference right there. Histograms in some of the implementations that I've tested, run between 7 nanoseconds uncontended up to double digit microseconds fully contended, so 1400x difference. I think it really matters both in terms of the implementation as well as the runtime and whether it's contended or not. We'll dive into that a little bit later.

Learning Objectives

What we're going to hopefully learn today is that there's this cost spectrum of instrumentation. We're going to look at what some real libraries do. Techniques for what I'm starting to call fearless instrumentation. In Rust we have fearless concurrency. I want to be able to instrument things without the fear of the cost of adding instrumentation.

Metrics Types

A quick primer on metrics types, so we're all on the same page. First thing that we have is a counter, which is a monotonically non-decreasing value. It either goes up or it stays the same. It never goes down. This is like the number of requests that you've served. Counters can reset, and that's the one case where you'll see discontinuity in the value. Otherwise, you're never taking away. You've done the work. You're incrementing. There are also gauges. These are point-in-time values or spot values. There's really no rules about these. It's just the current level of something, like a queue depth or water level if you're looking at a river, or temperature. Like temperature, it's possible that these need to represent negative values. That's gauges. Then the third type is histograms. There are some other ways to achieve the same thing other than using histograms. Like you might use DD sketches.

Really, we're caring about the distribution of values when we're thinking about histograms. What that means is that we're counting the number of times we've seen something with a value similar to some other value. We're bucketizing our values and counting the number of times that we see things that match that. Latency, we're going to track the number of requests that have had latency similar to that. This is all framed in the context of Rust. I've been working on Rust for over 10 years now. There are design decisions in Rust metrics libraries just like every other library. None of this is really specifically about Rust. All these libraries that do metrics have different design decisions that they've had to make. Even the fastest one might not be the right choice for you. I think being aware of what the tradeoffs that might not be that obvious when you're just going on crates.io and trying to pick a metrics library, you really need to understand what some of those design decisions were and what the tradeoffs are. None of this is to pick on any of the libraries that I'm going to be talking about. I think all of them have reasons for why they did the things they did. Just in terms of looking at practically how these libraries perform. Let's dive in and cover what those are.

The first one is just simply called metrics. In Rust, a lot of crates have very simple names. This one is focused on flexibility. It uses this thing called the facade pattern, which is what's used for logging and tracing in Rust as well. What that means is you can decorate your code with things. Then you can choose what metrics backend actually do the collection. This means that you can also switch those at runtime. It's a very powerful pattern. It makes it very suitable for both people writing libraries as well as people writing applications or services. As a library author, you can just decorate your code and some application developer later will be able to just say, I want to use this metrics backend, and magically they see your metrics. If they don't register like a recorder for those metrics, they shouldn't really be paying any overhead.

This is zero cost abstraction stuff going on. Very cool pattern. Very powerful. The next crate is called Prometheus. This is from the PingCAP folks working on TiKV. It's focused on Prometheus exposition, funnily enough. It's very much about having the correct naming labels or attributes, whatever you want to call them, and basically exposing these metrics to Prometheus. It uses a registry. Metrics need to be registered into that to be exposed. I feel like it's more designed towards applications rather than libraries. I think it would be hard unless you control the full stack to use this to instrument both. There's another crate called goodmetrics. The Momento folks work on this. It's really focused on high cardinality metrics. It's trying to solve yet a different set of problems where you might need metrics on a per account basis because you're running a large service with lots of customers.

It has OpenTelemetry integration. I think that this is also designed for applications, not for library authors. Next is metriken, which is the one that I work on. It's focused on efficient metrics primitives. As I mentioned earlier, it has this link-time registry for static metrics. There is the ability to have dynamic metrics and register those at runtime. This one is also really best when it's vertically integrated. It doesn't have that lovely facade pattern that the metrics crate does. I built this for reasons.

Performance Implications

Let's understand some of the performance implications of these design tradeoffs. We're going to first look at counter implementations, because these are the most important thing that you're going to be doing. This is based on the concurrency hierarchy. First thing would be ThreadLocal variables, where incrementing a ThreadLocal counter is going to be less than a nanosecond. An atomic increment, when uncontended, is going to be on the order of 5 nanoseconds. A compare-and-swap, or check-and-set, compare exchange, however you want to call it, is going to be on the order of 10 nanoseconds when it's uncontended. What is contention? Contention is when you have multiple threads that are updating the same memory location. These are shared variables. What drives the cost? Primarily cache line synchronization. If one CPU has updated the value, it needs to mark that it is dirty and tell the other cores, you're going to need to get this later if you want to use it.

This means that there's a bunch of memory bus traffic, because you're bouncing cache lines around. For CAS operations, you're going to generally need to do a retry loop, because your compare is going to fail. You're not going to actually set it. You need to get what the new value is, decide how you want to change it now, and then try again. The key thing here is that contention amplifies the differences in performance of different implementations.

Counters

Looking at the two ways that you could do counters, here the blue line, which is lower on the chart, is for an atomic increment, and the red line is for CAS operations. We're looking at the latency in nanoseconds, and we're looking at the concurrency. This is an AWS Graviton c8g.8xlarge. This is the 32 vCPU instance size Graviton. Here we see how that CAS retry loop has a lot more cost than just an atomic fetch_add, and now we're going to look at how those Rust metrics libraries fall on the scale. The first one is the metrics crate. As I mentioned earlier, this has the facade pattern, so there's a little bit of overhead on top of what the bare atomic primitives are. The metrics crate, since there are different recorders that you can plug in, there's the debug recorder, there's also a Prometheus exposition recorder. There's a bunch of others, but these are just the two that are co-located together in the repo that I was able to test.

You pay a little bit of overhead. This is going to be things like vtable dispatch, so some dynamic trait dispatch stuff that the compiler can't really see through to optimize further. You're paying a little bit of overhead there, but for the most part, it tracks these two things. I think the key thing here is that as you swap to a different metrics recorder, like the debug one isn't really useful. It doesn't actually get you metrics out of this system. You need to use something like the Prometheus one, and that has a much higher cost than the atomic increment does. You're looking at getting close to 1.5 microseconds fully contended on a 32-core machine versus sub, I think that's going to be on the order of like 300 nanoseconds or so for the contended atomic increment. Looking at the Prometheus library, this one tracks both the atomic primitives much closer.

Here the thing to note is that there are two counter types that are exposed by this metrics library. There's this thing called counter, and there's a thing called IntCounter. Counter is implemented as an atomic double, which doesn't really exist. There aren't atomic operations to do that. It's all implemented with CAS operations. If you want high-performance counters using this crate, you need to use integer counters. Hopefully you're counting things that are naturally integers anyway. If you're trying to add up fractional seconds spent doing something, you might consider scaling that to nanoseconds and just using an integer counter, because it's going to save you a lot of time. goodmetrics, next one, totally just follows the atomic increment. Nice. Then the one that I work on as well just follows atomic increment. Looking at all of those in one spot, we can look at how they perform. I picked 1, 8, and 32 cores as the scaling numbers.

We can just look at how different these actually are, depending on what metrics backend you've picked, what counter type that you've picked. All the ones that are just atomic fetch_add are actually fairly cheap. A couple hundred nanoseconds fully contended isn't really that bad. In the uncontended case, they perform all fairly similar. Yes, definitely as you scale up, the costs start to add up pretty significantly.

What's the real-world meaning, because we're talking microseconds and nanoseconds? It's hard to think about that. This is assuming that we're just tracking one metric on a 32-core system. It's all very efficient. Your business logic costs you absolutely nothing. We're just thinking about how many times we could handle requests if the only thing we were doing is updating metrics. In something like metriken, this is the ideal case. Contended cost, a few hundred nanoseconds. Max throughput, 119 million requests per second. That's more than I think most services are going to be able to hit. If you look at something like metrics with the Prometheus backend, the contended cost was 1.4 microseconds. Now your max throughput is 23 million requests per second. That's like a 5x difference there, which is pretty crazy in what your system's ceiling is. Now, 23 million is still a lot, but you're paying an extra microsecond for each request.

Can we do better than this? In the ideal case, we were doing atomic increments. Can we do better than that? When we have a single shared counter, we have that cache line bouncing around our system. All the cores are competing. We have this range of performance, depending on how far we've scaled vertically. We can do per-CPU counters. This might be a way to shard the counters, give each CPU its own, just increment it. Now there's no contention. You read by just summing across all these. This is a scatter-gather pattern almost. It works. The read time cost is a little higher, but the write cost is what we're trying to optimize for, because that's going to be in our hot path. This type of technique is used extensively in the Linux kernel, because it needs to be hyper-exposed metrics and is always in the hot path.

It's not as simple as that. There's this thing called false sharing, where adjacent counters might be in the same cache line. A cache line is 64 bytes. That's eight 64-bit counters. If CPU 0 is updating counter 0 and CPU 1 is updating counter 1, they're actually going to be sharing the same cache line. You still have contention, but now you've scoped your contention down. You have eight cores that are adjacent to each other, all fighting for the same cache line. It's going to be better, but we can do even better than that. If we have multiple counters that we're going to be updating together, we can group those. We can have a counter group. We can pack things together and pad it out to an even number of cache lines. Now we've eliminated that false sharing. We have actually truly uncontended atomic access to them, this should be atomic u64.

We're going to get down to that 5 nanoseconds per increment, regardless of how many threads that our application has. There's no linear scaling factor to the cost anymore. We can do this in eBPF. It's actually very similar. We just index into an array using the CPU ID, and yes, basically an offset into that counter group. This allows us to do eBPF instrumentation as well. That's very low overhead. We can put it in the hot path without that fear of what we're going to be doing to performance on machines of different sizes and stuff like that. There are three main approaches for counters. CAS loops, which are going to be fairly expensive, ceiling of around 27 million requests per second on a 32-core VM. Atomic adds gets you much faster at 119 million requests per second. Then if we're able to actually do that sharded approach, all of a sudden, it's 6.4 billion requests per second. I think now we can fairly comfortably instrument everything without any fear about what we're doing to performance. Five nanoseconds is very cheap. The approach to even just how we're counting things, it makes a huge difference in how your service is going to scale, how your library is going to perform, all that stuff.

Gauges

Next up we have gauges, but these are really going to be very similar to counters, so I'm going to just skip this. There's not really that much more to say. We can use those same atomic primitives. They're going to scale the same way. We can do the same tricks for the most part.

Histograms

We'll move on to histograms, which I think is a lot more exciting, probably because I'm a nerd and have worked on histograms for a while. What is a histogram? What do we have? We have an array of buckets for counting values that fall into certain ranges. We might have a struct like this, where we have a bunch of counters all packed together in an array, and we have some way for mapping values into bucket indexes. The core operation is, given a value, what bucket does it fall into? Then after that it's just a counter. Let's go on a little side quest here about indexing into histograms, because I think this is very important. As I looked at some of these metrics crates, this is definitely overlooked. Strategy one, possibly the most naive thing we can do is a linear search. We might precompute what our bucket boundaries are.

We might let the user specify just a real hodgepodge of different bucket boundaries, and we'll just walk the array comparing our value to those boundary values, and once we get there we'll just increment our counter. This is the worst thing that you can do. It's going to be very slow with a lot of buckets. This is a search problem, so we can use binary search and speed it up. That's still better, but we're still doing like 6 comparisons for only 50 buckets. This is definitely not the way to go still. Direct indexing, so we can change this from a search problem to more of a hashing-shaped problem where we can compute the bucket index directly from the value. We've traded off some flexibility, but now we've gotten it down to it not mattering how many histogram buckets we have. It's fast and consistently fast.

There are a few ways that we can do this. We can do linear ranges. We can just divide it by a thousand and just take the integer component after that and use that as the index. We can divide it by a million if we want less buckets. This has some problems though. You're fighting to balance bucket count and resolution, and you have a large relative error at the low end. For your small values, your bucket's fairly wide compared to those values, so you have a higher percentage of error. As you go up the histogram to where higher values are being tracked, that relative error has shrunk pretty dramatically. We can do a little bit better than that if we want to preserve fixed relative error across. We can do log-based bucketing. This has better relative error, but the problem is logarithms are slow. There's still the potential for having a large relative error.

With this example, it's a base-10 log, so you're within an order of magnitude, but not really any better than that. To address the slowness, we might use Log2, which we can do with just a single instruction, counting the number of leading zeros, subtract that. Now we're not doing an actual log operation. Much faster, still have that large relative error.

The next improvement on this is to use something like a HDR Histogram, or my CEO, Yao, wrote this thing called H2Histogram, which is also another way of achieving direct indexing. It uses Log2 outer buckets, so we're just doing that count leading zeros operation, so very fast. Then we have sub-buckets within each of these ranges, so now we can control what the error is. We can put bounds on a relative error. H2 expresses it as relative error, and HDR Histogram tends to think more about precision, like the number of digits, but they're both very similar approaches. Just looking at these standalone, let's just look at our indexing performance into each of these. This is going to be with a non-atomic histogram, so we're just vanilla increments, mainly looking at what the indexing speed is. The HDR Histogram crate comes in at 2.65 nanoseconds to index in and do that increment, so the indexing is probably on the order of 1 to 2 nanoseconds of that time.

The H2Histogram, which is just the histogram crate, is a little bit less than that at 2.15 nanoseconds, so there's like a 20% difference there, just because of how they're implemented. Really, the key takeaway here is these are both very fast operations to index into these histograms and increment something, so you really can't go wrong between these two. I have a bias towards the H2Histogram, just from some of the design decisions that were made and how it actually works, and it being a little cheaper I think is nice.

Since basically the indexing is free, almost, couple nanoseconds, and the next operation that we're doing is just counting, these should look like atomic counters in terms of how they scale. Spoiler is that it's not going to look like that. In fact, the metrics crate, so here this is the Prometheus backend for it, because the debug backend actually doesn't handle histograms. It just queues up all the observations that it's seen. In my test this would just run out of memory and crash, which isn't cool for a debug thing. I don't think that's great. Looking at an actual backend for it that does actual work, here's what we see, and this is not scaling super well. We're hitting over 2 microseconds to increment our histogram, and we know that should be a couple hundred nanoseconds. It does get worse. The Prometheus crate does something slightly different, and I don't know why it doesn't scale fully linearly at the end.

Here's what it did, we're over 3 microseconds now. This isn't looking too good for histograms. goodmetrics does something different yet again, and now we're up into double digit microseconds. This is a really concerning pattern that we've developed, but I put my library last so I know what's going to happen. We can, it should just be an atomic counter. It should be that easy. What is going on here? Why do these other libraries not perform as well as we can reason that they should? They've all taken different approaches, and here we're just going to have a table of how bad this really gets at 32 cores. Again, this was on a c8g.4xlarge, or whatever the size is, that's 32 vCPU. This was basically just incrementing a log normal distribution. I didn't really set any histogram parameters to anything very specific. I just did this the most naive way possible with each of these libraries.

Huge range of different performance. The metrics crate, it's doing that linear search across the array, so we know that's not going to be super fun. Even if we make the histogram larger to remove contention, now we have a bigger thing to do a linear scan on. I don't know why they didn't do a binary search at least. It was baffling to me when I looked at it. There's also some overhead for it passing values over to the recorder. Then there's multiple atomic operations that are happening for each increment because it's tracking other values. The thing to understand, that histograms in Prometheus world, you have the actual buckets. You have a total count of all observations as a separate variable. You have a sum of all the values that you've seen. Unfortunately, once you're incrementing something like this a lot, you tend to use an atomic double that doesn't exist.

You actually have the range to record things. That means that you're doing CAS operations to track the total sum of all observations. Prometheus crate, it's doing six atomics per increments because it's trying to do some clever things in terms of having a hot and a cold histogram so you can take consistent snapshots of your histogram state. It has a CAS loop for tracking the sum. goodmetrics, mutex acquisition. It winds up serializing all the updates. All our threads are fighting to lock things and incrementing in a serial fashion. It really does simplify the implementation. Under the hood, goodmetrics is able to do some dynamic re-scaling of the histogram. There are reasons why it's done that, but the mutex cost is very high when you're contended. My library just has lock-free increments. Very efficient direct indexing with the H2 strategy. A single atomic operation per increment. We have nice scaling properties there.

There are tradeoffs, and I've mentioned some of them. One is consistent snapshots. To do this, you need to either have multiple atomics and do some hot-cold strategy, or have some locking strategy or something so you can snapshot your histogram state. We don't really need to be doing it that way, and I'll get into that now. The histogram crate and metriken use an eventually consistent approach. This means that we have much faster updates because we don't have all these multiple atomics or a lock or anything. The downside is, as we're reading from the histogram, buckets are being updated towards the end, so we have this skew that's happening. It doesn't really matter. The thing that we're most often looking at is the difference between free-running histograms at two points in time. As long as scanning through this takes roughly the same amount of time, that skew cancels out roughly.

These are all approximate data structures anyway, so I think a little bit of error is totally acceptable. I think being able to get the cost down from double-digit microseconds to a few hundred nanoseconds, that's worth a little bit of error. We can use the same approach in eBPF. It's just an algorithm. Again, we just have this array of memory-mappable buckets, so we don't need to make any syscalls to read this into user space. We just index in, and we do our atomic increment.

Lessons Learned

What have we learned? There are tough choices to make when you're picking a metrics library in Rust. You have to choose between performance, so you can get really good performance. Maybe it isn't the best thing to use as a library author where you want people to bring their own metrics recorder with them, be able to opt out of metrics. The counters versus histograms. A lot of these libraries, you have to choose what you want to record, and you might have to limit the number of histograms because the implementation's very expensive once you scale to a certain number of threads. Counters, fairly cheap in all of these, so you're probably going to lean more heavily into counters, but you're going to miss all the really cool stuff that you can see with histograms when you're looking at the distribution of sizes of requests, the distribution of latency of responses.

All these sorts of things, I think, are very important. If your histogram cost is high, you're not going to be able to use those fearlessly. The strong versus eventual consistency thing with the histograms, I think that this is just fundamentally a problem with all things where you are trying to read from a bunch of places that are all being incremented at different times. The synchronization really helps if you need a consistent snapshot, but I'm going to continue to argue that you don't really need that.

Design Principles

The design principles behind all of these. Counters, atomic fetch_add is the most efficient thing that you can do. Avoid CAS loops, prefer normal atomic primitives. Reduce contention with sharding if you need to be able to scale out to very many cores or very many threads. I would probably use different approaches if I was thinking about threads or whether I'm in a context where I'm able to cheaply get the CPU ID that I'm running on currently. In eBPF, that's super easy, basically free to know what CPU you're on. In user space, you have to maybe make some different decisions about how you're doing the sharding. There's some really good stuff online about how you can do this in a self-adjusting way. For counters, the cost, assuming that you're just doing atomic fetch_adds, you're not doing any sharding from one thread up to 32 threads on a 32-core system, you have this range of cost, c8g.8xlarge.

Histograms, eventually consistent is going to be much more efficient because you don't have that synchronization happening. It can be as cheap as a counter, which is what it should be. The implementation matters a lot more than it did with our counters. Our counters, you could probably get away with any of those and not really notice that much of an impact, but with the histograms, it's going to matter a lot if you're adding 10 microseconds onto every request because you're trying to track latencies. You're probably going to steer away from those histograms, you wonder why you can't get that visibility, so the implementation matters a lot.

Key Takeaways

The key takeaways, implementation matters. It's unfortunate that there are four Rust metrics libraries that you could probably pick from, some with more adoption than others, and the tradeoffs, just none of them are particularly great for all use cases. The cost really matters across these. Here's just a recap of those numbers. I don't want to pay 10 microseconds to increment a histogram, especially if I'm doing that in the hot path in eBPF in kernel space and trying to track the latencies for every syscall that's happening on the system. I think that this is the difference between having fearless instrumentation and trying to really ration what you're instrumenting. As we talked about at the beginning, if you're lacking in instrumentation, you're going to lack in visibility into how you're performing, what went wrong when there is a performance incident in production, which there will be, and you're not really going to be able to improve things because you don't know where you're at currently.

I'm going to argue that we need to be able to instrument very heavily. We want to be able to get metrics about how all our code is running, how our service is running, be able to look at queue latencies, all sorts of things where histograms are very important. If we're using the wrong implementation, we're not going to be able to do that. With the right primitives, we can have fearless instrumentation, though. We can have atomic increments. We can make sure that we don't have any CAS loops. We can directly index into things, and not do a search problem every time we want to update a histogram. We can shard to just eliminate contention entirely. The same techniques work in user space regardless of what language you're working in. It also works in eBPF, so it's cool. We can use these things across both. These are just core lessons about how to do metrics, really. It isn't really a Rust talk, necessarily, other than we did a survey of Rust metrics libraries, but I think once you apply all these techniques, you can instrument comprehensively without any fear.

Resources

You can learn more about Rezolus, which has the eBPF telemetry, on rezolus.com. The metrics library's up on the IOP Systems GitHub, along with a lot of the other open-source work that we do.

Questions and Answers

Participant 1: How much does CPU architecture matter: ARM, RISC-V, Intel?

Brian Martin: It's going to matter, but I think these numbers are probably close in terms of order of magnitude. I tested on ARM just to look at it there. I know it can be a little sensitive. If you're looking at something where you have a multi-socket system, so you're doing true NUMA, you're going to have even higher costs for bouncing those cache lines around. The architecture is going to matter, but I think in general, as long as you're making those fast choices out of all of these, that's going to perform well.

Participant 2: I wanted to ask about the relative error rate, and if you could explain a little bit more why it's skewed towards the latter parts of the histogram.

Brian Martin: The relative error, so you have those logarithmic outer buckets. By dividing those into linear ranges within each of those, you can get higher resolution by increasing the number of linear buckets. Like for H2Histogram, that number of inner buckets is controlled by what we call the grouping power. If you dial that up, you have a wider histogram, but you control the relative error. If you look at what the minimum value that's going to be incremented into that bucket and the maximum value, assume that you're observing the minimum. What's that difference divided by the actual observation? As you increase the number of buckets, that naturally shrinks. In H2Histogram, you can say that, A, I'm ok with a 5% error because this is a latency metric, and I'm going to look for bigger changes than that. Or I'm memory constrained, and I'm ok with 12.5% error. Or I'm writing a benchmarking tool, and I really want to be able to compare latencies across runs, so I might want below 1% in terms of the relative error, so I can compare my tail latencies and know that I'm in roughly the right spot.

Participant 2: How do these libraries coexist in user space with Java-based applications?

Brian Martin: Yes, they don't really. These are all Rust-specific solutions. You could FFI into some of them, but the way the facade pattern works for the metrics crate, I don't think you'd be able to really leverage it. Yes, unfortunately.

Participant 2: How about in kernel space?

Brian Martin: In kernel space, again, so since I work on eBPF observability, I actually just have C implementations of the H2 indexing. I can just keep my eBPF all in C. Then it's basically a memory mapped range, so I can just read from that in user space. Since I'm using the same exact indexing strategy, and I know the grouping power and all the other parameters of the histogram, I can just copy it into user space and then expose it via Prometheus exposition.

Participant 3: Can you apply the per-CPU sharding technique to histograms to get down to the 10-nanosecond ballpark?

Brian Martin: You could. You definitely could shard the histograms to get things down further. You could do. You might not even need to necessarily have a counter per CPU. You could. One of the difficult things with eBPF is since you're not able to really dynamically set what the size is, so I tend to hardcode that the maximum number of cores I'm going to have on a system is 1,024. If you have that multiplier on a histogram that's 8 KB, you might be able to afford that. That might be totally fine. Maybe getting rid of some of that contention is important enough to spend the memory there, but you have to balance just how much memory you have available to do that. Yes, same techniques work there.

Participant 4: How expensive is operation to get CPU ID actually comparing to just incrementing the counter itself?

Brian Martin: In eBPF, it's basically free. You're just reading from a register, so it's already there.

Participant 4: If you do like pure Rust, for example?

Brian Martin: It's going to be much more expensive. I think what people tend to do is they try and detect contention. If you're using something like CAS, I think this is easier to see. If I'm detecting contention here in this index, I'm going to try the next one after that. Things naturally fall into a place where they're going to have less contention. I don't think it's a perfect way of doing things. I think in something like Rust and user space, I'd probably try and use like a thread ID or what the idea of my Tokio runtime thread is or something like that to key things a little bit differently.

Participant 5: Are you suggesting that for a different metric, you use different libraries? It is counter, use this, if it's histogram, use the other library?

Brian Martin: I'm going to say that you have to use what's going to make sense for what your goals are. If you're a library author, it seems like at least right now, there's not really a better choice than metrics.rs or the metrics crate, just since that facade pattern is so powerful. I think you could write a recorder for that that has a lot less overhead for histograms. I think the counter implementation could also be much faster. I think you could drag those down, closer to what ideal should be. I think just that facade pattern overhead with the dynamic dispatch, you're always going to be paying for that just with the way that was actually implemented. The fact that at runtime, you can pick the recorder instead of predetermining that at compile time, I think that's a huge tradeoff. If performance is the ultimate goal, just use the one that I work on, because that's fast.

See more presentations with transcripts