Transcript
Khawaja Shams: How many of you find yourselves to be experts in inference? My name is Khawaja. I will try to keep this light-hearted, and there will be a little bit of repetition because one of the things I learned as a teacher was sometimes repetition really helps to internalize the concepts. These concepts are interesting, but once they click they make a lot of sense. I started my career building cameras for the Mars rovers. I was at NASA for eight years doing distributed systems. Then I ran DynamoDB at AWS, which is a large-scale NoSQL database. Today I run a startup. I'm a co-founder at a startup called Momento, where we work really hard to make access to data really fast. This works for AI inference stacks as well as the boring cache as well.
Perspective of a Distributed Systems Engineer
I was really happy to see that not everybody here or most people here don't consider themselves an inference expert. I can share with you that as somebody who came from a distributed systems background, I personally found the inference stack to be extremely intimidating. There are these crazy new words like attention heads, KV projections, CUDA kernels, tensors, quantization that made me feel like I need to go back to my linear algebra class. Luckily, in the world of inference, there's a lot of space for distributed systems people to make an impact. The same problems that start to appear when web scale came back in our histories where you had to take care of problems like routing, you had to take care of problems like your tail latency for your data access, how do you think about your caches and so forth. All those problems still exist inside the inference stack.
Currently, in today's world, one of the most scarce resources is GPUs. You can't produce them fast enough. Even if you could, there aren't enough data centers to go around. While the world is busy producing more GPUs, building more data centers, this is going to be a renaissance for us distributed systems people because we can help improve a very critical metric, which is the utilization. Some of the concepts you'll learn today, they all feed into how do you make the inference stack have higher utilization. If you have better utilization, if you can use the capacity better, then that leads to better availability, it leads to better performance, and it leads to better economics.
Two Stages of Inference
When we talk about inference, this is when I go to ChatGPT and I say, please say hello to me, and it takes like 2 seconds for that thing to happen. There are actually two stages of that prompt being inferred. The first stage is the prefill stage. This is where the prompt actually gets processed. I'll give you a quick example, and I'll try to make these as relevant to the way humans talk to each other as I possibly can to help you internalize all these concepts. Think back to when you were in high school or earlier, you may recall this riddle where people will show up and say, you are driving a bus, and that bus has no passengers. Then at the first stop, you get one person. At the next stop, you get two people. Then at the third stop, it doubles. Then at the fourth stop, a third of the people leave and so forth.
Now imagine if I sat here for the duration of this talk, and I just kept going on and on about that particular riddle, and in the end, all I'm going to ask you is who's driving the bus? The answer is you, but you cannot answer that question until you are done processing all of the input. That is the prefill stage. This is where the input prompt that you have is getting converted into machine code that maps to how the model thinks. The decode side is the actual answer. This is when you're saying the answer being like, I'm driving the bus. That's the decode side. In the inference stack, these two are very separate stages. It'll make more sense a little bit how these stages map to each other. The prefill side is very compute bound. You are just sitting there parsing the words that are coming out, and you're converting them.
It runs all in parallel, takes up a lot of CPU. The decode side has to reference all of those things, so it has to be very memory bound. You might have heard this metric that the whole world obsesses over, it's time to first token. This is the metric that users care about because it lets you improve the perceived interactivity, like how fast this thing is responding. The first token will not land on your screen until the prefill is completely done. Just like you can't even begin to answer the question about the school bus until I finish the question.
Journey of a Prompt
Let's go through this from the perspective of a prompt. Today, in the magic school bus, you are the prompt. You are going to get entered. The user is going to enter you, and you're going to fly through the packets on the internet. You're going to land on a routing box. That particular prompt has to get tokenized. The programming analog here is when you take your C program and you compile it into assembly, that's what's happening. That's the tokenization phase. It's not one-to-one. A single word may not become a single token. It's roughly one-to-one, but it's not quite that much. Then after the prompt is generated, what the tokens generated, the routing layer is going to actually go and check, have I seen this before? Like, have I heard this riddle before? Or, have I heard parts of this riddle before? You go and you check.
Sometimes as I was asking the question, some of you were already thinking the answer is me. Because you've seen this before. You've already compiled this analogy or this riddle. Then the prefill starts to happen. In the prefill stage, if you haven't seen it before, you're generating the machine code. You're generating the 1s and 0s that map back to the way the model pays attention. Then you store that data inside what's called the KV cache. Then once the prefill stage is completely done, you go in the decode loop, which produces one token at a time. Once the response is complete, as each token comes in, it gets streamed back to your screen. The costs here on the prefill side is just compute. On the KV cache side, it's very storage heavy. It's in the gigabytes. On the decode side, it's very memory bandwidth heavy. All the user care about is how quickly they get their tokens.
Now, KV caching, it seems somewhat orthogonal as a user of inference services. If you understand how KV caching works, it'll help you drive your inference stack better, even if you're not running your own inference stack. It'll help you drive your Claude Code sessions better too. KV caching is hard because it takes a lot of storage. GPU memory is extremely limited. KV caches, it's not easy to share them between sessions. We'll get into each of these three. For the purposes of this talk, I learned by doing, and I created a few experiments. These experiments were done on a very modest model. This model is 4 billion parameters. Some of the larger models, like when you're using GPT-5.5, those are trillion parameters model. They go into trillions. Claude Opus 4.8, trillions of parameters. This is a 4 billion parameter model. The experiments were run on a G6 instance type, which is actually readily available inside of AWS, costs a couple of bucks an hour.
You can go reproduce these. For this very modest 4 billion parameter model, let's compute how big a single token is for your KV cache. That single token, your LLM model has 36 layers. Each layer has 8 KV heads. Each token gets converted, multiplied to embeddings of 128 dimensions. Then you have 1 byte for the key and 1 byte for the vectors that follow. You multiply those numbers together and your hello just got converted to 72 kilobytes. That's a lot of data. One of the experiments that we'll get into in a bit was 20,000 tokens. Just to put things in perspective, your Claude Code sessions are constantly churning 100,000-plus tokens back and forth. 72 kilobytes per token, a 20,000 token prompt is 1.6 gigabyte. That might be ok if you owned that GPU for yourself. In a fleet that your ChatGPT provider or your Claude Code provider behind the scenes is running, that's running on GPUs. The GPUs have RAM from the 2000s in a certain extent. The GPU that I was playing around with, the cheaper one, only has 24 gigs of RAM. The really expensive Blackwell models have 192 gigabytes of RAM. RAM inside of GPU is not very big, so storing gigabytes of data for each user becomes very expensive very quickly.
Then there's a second part which is, it's really hard to share KV caches across sessions. This is because each individual word that gets converted into tokens, KV is a little misleading because people think KV means for this token, here's the value. It's not a HashMap lookup. What you're really asking is for this token, after it's been preceded by these other tokens, what is the value? This applies to humans as well, the way we perceive language. If I say time flies like an arrow, or if I say fruit flies like a banana, if you were to go and encode flies independently, the word flies has two different meanings. It depends on the context that preceded it. You can't just individually go cache individual tokens and expect them to be reusable across sessions and so forth. Your KV cache that's running behind your ChatGPT queries and so forth is generally pretty personalized to you, because you and the other users might be asking very different things.
In fact, one of my team members just committed a patch to Llama.cpp, which is an inference engine that you can run. One of the things that Llama.cpp was doing was, it was injecting a timestamp before the prompt got sent. That would invalidate the entire KV cache, and the whole prefill had to be done again. You can imagine a simple mistake like, I'm going to insert the timestamp, seems so benign. It means your GPU has to do all of that work again. It doesn't have the reasoning skills of a human where it can say, yes, that's just a timestamp, let me ignore it. No, the prefix does not match, therefore I must compute it.
Prompt Anatomy
Let's take a look at the anatomy of the prompt. There are some parts that are shareable. The first part is the system prompt. If you are running Codex, the system prompt is actually not all that complicated, but it's very long. The system prompts by saying, you are Codex, a coding agent. Then it has all kinds of instructions about what you should and shouldn't do. Like, don't over-explain, don't write comments unless somebody asks for it. I highly recommend, if you can go get access to it, you can see some of the Claude Code prompts on the open internet now after a leak as well. Just read through what the system prompt looks like. It is like you're talking to a child. It's like, important in urgent. Anybody watch The Office from back in the days where Michael Scott had Urgent A, Urgent B, Urgent C? It's like that.
It's like, there's important, there's super important, and there's caps used. In fact, the Codex prompt, I kid you not, says, no matter what, do not talk about goblins. I can help explain my theory offline about why that token is in there. The system prompt, especially for coding agents and things like that where somebody is using the same thing over and over again, that is very common across your user base. Every Claude Code session is going to start with the same system prompt. That part you can share across users. If a user is online, their prior turns are also, you've seen it recently, you probably still have it. If I were to continue my riddle now and say, what is the maximum speed do you think that vehicle will go? You know the answer without having to think through and reprocess the whole bus story. If I ask, how fast will it fly?
You will also know the answer because you know it's a bus. It's still in your cache. If the person comes back reasonably quickly, within minutes or hours or whatever, the prefill for their session is likely still there from their prior turns. The new stuff is what they just added. Like, how fast can this vehicle go? Or, will it fly? That's the part that needs to be regenerated if you got a KV hit. If you pay attention to your Claude Code session, you'll notice that your cache hit rate is actually quite high. The reason for that is when Claude Code takes your turns, it's not sending a brand-new prompt every time. It's just appending to the last one. That's why your token consumption goes up the longer your session goes. Then eventually you run into that compaction side because your context just grows and grows and grows, but they keep appending so that your prefixes keep matching.
Learn by Example
Let's go to the experiment. Just to help you internalize some of this, again, modest model, 4 billion parameters. Cheap GPU. We use floating point, FP8 just means floating point 8. It's a float. If you remember floats from your coding schools or your classes. Then we use vLLM. APC just means automatic prefix caching. We did a sweep where we said, run one concurrent session, run two, run three, run four, up to 48 concurrent sessions. The workflow was a mini-agentic coding workflow that we were trying to simulate. It starts with a 10,000-token system prompt that says you're a coding agent. Don't talk about goblins. Then you take 12 turns and each turn you add like 1,067 tokens. Then you just go back and back and back. At the peak, you get 22,800 tokens at turn 12. We wanted to see where does the GPU hit a cliff in performance between 1 to 48 concurrent sessions.
I'll skip to the good part where when we got to 12 sessions, it looked pretty good. The time to first token at p50 was only one second. It was a complex prompt. It's a long riddle. It has to be prefilled and everything. Just to give you, 22,000 tokens is roughly like 10 pages of written text. For a computer to just go parse through that and at p50 give you a second, it's pretty good. At p99, the time to first token was 2.6 seconds. That's with 12 concurrent sessions running and it's doing a little bit of prefill every time, getting a cache hit for the prior turns before. The reason why it does so well is because the KV cache actually still fits, and we did the math and it turns out that it can sit around 13. There's overhead and so forth. Then we added two more.
Same experiment, but we did 14 concurrent sessions. Now you are adding a little bit more load where the data just doesn't fit in memory anymore. Between turns, it's getting lost, so it has to be regenerated. If anybody was around in the 2000s and may remember your Windows PC, remember the word swap space? You can still see it in your MacBook too. Remember when your computer would go into using swap and everything will come to a halt? It's like that. Where the GPU just can't keep it in memory anymore so things start to evict and it has to recompute. The p99 jumps from 2.6 seconds before at 12 concurrent to 38.9 seconds. That's a major noticeable jump. It just comes down to being not able to fit everything in memory. The cliff is actually quite bad because the sessions that get run out of memory, like that end up being evicted from the GPU, they end up doing really badly.
The lucky sessions, that's why the p50, the 50th percentile ends up still being ok because at 50th percentile, you're still getting some cache hit rates. You're getting lucky and that's why it's raising so gradually. We've all heard of tiered storage and tiered caching. Memory is very expensive because GPUs are very expensive. What about DRAM? As much as the world is saying that DRAM is getting expensive, it's still way cheaper than GPU RAM. VRAM is the GPU RAM. It's like tens of gigs. We saw up to 192 gigs. On these massive boxes with GPUs, you can get terabytes of DRAM. This is just like the RAM on your laptop and so forth. The DRAM, we're living in a world where I can actually say this, RAM is too slow. Like you have terabytes of it, but now you got to copy the data from the RAM to the GPU on every turn.
You have terabytes of it and it's still going to be cheaper than recomputing it. Then there's NVMe, this is your SSDs, which can have tens of terabytes on a regular EC2-style server that you might have. Not only is it even slower than DRAM, it actually has write endurance problems. One of the things that goes wrong is if you keep writing over and over and over again, you'll just trash the SSD and it'll just stop working. The KV cache, because the data is so large, you are constantly going to be writing and you can exhaust the lifetime of your SSD pretty quickly. The tier below that is you can always just recompute. This is a cache that never has a miss because if you don't have the data, you can regenerate it as well. That's simpler, but it causes you to use more GPUs, and GPUs are scarce. Each tier basically trades latency for capacity.
Storage
Now, let's take a quick detour into, there is local caching that we talked about and then there is remote caching. We started going down this path of how do I make a good remote cache for this environment because GPUs are still expensive. A lot of times people who build cache, including me and my own team, we really like to talk about sub-millisecond latencies or 100 microsecond latencies, but here the latency to first byte is not that important. You're trying to download gigabytes of context about a user. What really matters is your throughput. Within the throughput, what matters is can you saturate the network card that you have because you don't want to leave any bandwidth unused. You want to get the data as quickly as possible because then you're not sitting there having your GPU wait. The G6 instance type that I use for my experiments has a modest 25 gigabit NIC.
When I say 25 gigabit NIC is modest, I feel a little guilty saying that. Twenty-five gigabits is a pretty decent network card. Just to put it in context, the instances that have the really expensive GPUs that I hope to be able to afford one day, they cost like 50 bucks an hour. They have terabits per second of bandwidth. A lot of the challenges here become, how do I consume all of that bandwidth? Some of this comes down to, how do I divide my objects into lots of smaller objects of megabytes, which in the world of caching, they're considered large objects, but it's better than downloading a single 1.6 gigabyte object. You can download a bunch of them in parallel. Your goal when building something like this for a remote KV cache is to saturate the NIC. Techniques like smaller chunks does help. The open-source software I work on is called Valkey.
This is, again, being a distributed systems person, being able to help the inference world a little bit. Anybody heard of Redis? Valkey is a Redis fork backed by Amazon, Google, and so forth. There's been a lot of performance optimizations in that space. One of the optimizations that got made is copy avoidance. Redis, from the very beginning, branded itself as having single-threaded. Because if you're single-threaded, you can move really fast. There's no locks. That works really well when you have tiny 5 kilobyte, 10 kilobyte objects. What happens when you have a 4-megabyte object is that your main thread is busy writing to the kernel buffers to write to the NIC so that the data can go on the network. While it's doing that, it's not actually answering any of their calls. For the KV caching workflow, you're not only downloading, you're also checking, do you have this prefix?
Do you have that prefix? All those calls will get locked behind it. In the latest version of Valkey, one of the interesting things that happened is that there are I/O threads. The I/O threads deal with the network, and the main thread just says, yes, you want that object? Please copy offset by this to this from this address onto the NIC. I/O thread goes and does its thing while the main thread moves on. The reason why I bring this up is that simple optimization makes a big difference on whether you can consume the entire NIC or not. Because until the 4 megabyte is copied from the main thread to the I/O thread or whatever, you can't even start to write the first byte. Then the software layer that's going and downloading each individual object can't actually pass it on and start to stitch it together until it downloads each individual chunk.
This gets really expensive. Copy avoidance, funny enough, has actually been the most profound optimization that we have done not just on the server side but also on the client side. The inference engines, like if you might have heard of vLLM or SGLang and so forth, often written in Python. Everybody thinks Rust is going to be much faster. I can say that because I'm a big Rust fan too, if I can make fun of people who think Rust will make them go faster. This is a classic example of when Rust will slow you down. Somebody had the bright idea that, yes, this isn't Python, I'm going to write my client in Rust and I'm going to do a native binding. What happens is that that Rust layer now has to download the 4-megabyte object and it can't pass it to the Python layer until the 4-megabyte object is done downloading.
Then it has to copy it over to the Python space which then has to go copy it over to the CUDA space which then has to go copy it over to the GPU. All those copies, they add up. I harp on this example because that is a classic example of your CS skills coming in handy for this very complicated looking inference stack. Minimize copies wherever possible.
With that out of the way, let's see if we can use DRAM. We'll go back to local caching and see what happens when we add a little bit of DRAM. We notice that the first row is just 8-bit floating point. The p99 cliff happened at 14 connection, and our latency was 38.9 seconds for p99. Then we said, let's just add 16 gigs of local RAM, the DRAM. That actually helped meaningfully. Our p99 at 14 connections actually went down from 38 seconds to a more bearable three and a half seconds. We ran a sweep to see, how much does it help. Actually, it turns out, for this particular workflow, if you add 32 gigabytes of local DRAM, you can push your concurrency cliff from 14 to 28. 8-gig didn't really help much, it went to 14. Really having the 16 was a key tipping point. The impact becomes more and more profound the higher your concurrency gets. It's still pretty bad. You really want the data to be in the GPU whenever possible if you want the fastest possible inference.
Synthesis
After harping on why you should really think about how much concurrency you need, I have some bad news. If you are running a multi-tenanted cluster of inference and you have users that will ask short questions, and you have users that will ask long questions, and sometimes you'll have a very spiky mix of them. A new release of Claude Code might completely change the way your traffic looks if you're running an inference service and that gets really tricky. You really can't make any assumptions on being able to admit by concurrency count. In the early days that's what everybody did. It's like they just said, ok, I'm going to run this much concurrency. What happens is somebody shows up with a 100,000 or 200,000 token prompt and it just thrashes everybody away from the GPU VRAM, and that causes problems. What you really have to do is think hard about, for this model that I'm hosting, what is my per token cost?
How much budget do I have for my KV cache on the GPU? Then, what is my cliff estimate for a particular token size? When you make your admission control decision, when you want to decide, do I add another request to this GPU or not? You really have to be looking at how your KV cache consumption is going, and are you going to cause an eviction the moment that you add this particular request. It's very workflow specific, and this is where the multi-tenancy makes it harder. If you have single-turn questions like, ChatGPT, what's the meaning of life? That's like 1,000 tokens, perhaps. Very hard question but the prefill for that is really fast, I can tell you that. Those are very short single-turn questions. The tiny agentic coding sessions are tens of thousands. Code reviews might be a few thousand and so forth. The key message I want you to take away is that the token consumption really depends on the workflow, and if your fleet is supporting a variety of them, you got to be very mindful of that too.
Routing and Processing
Thus far we have covered storage aspect of it, if you're running the inference fleet. Now let's talk about routing choices. There are still web servers involved, all this still happens on HTTP, so more good news for systems people. The request goes to a web server and the web server has to decide which GPU am I going to send this particular request to. If you do a naive thing, back 20, 30 years ago, there was the situation where people came up with sticky sessions. Sticky sessions are beautiful because the likelihood of the user landing on that node that still has their stuff on the cache is higher, so sticky sessions work really well in the web space. Here people still try to do sticky sessions to the GPU, so the web server will try to remember, this particular request for your coding agent session went to this GPU, so let me just go back to that GPU.
Because the odds of it having the KV cache locally on its SSDs or whatever is meaningfully higher. The problem is that these are extremely spiky workloads. I might take my next turn immediately. I might go for lunch and come back. I might never come back. This creates these like very spiky bursts. If you're trying to just do naive sticky sessions, you can get into situations where a particular GPU becomes really hot. It's like a bank run. Everybody shows up to that particular node and you get essentially a noisy neighbor hotspot issue. Your fleet-wide p99 is going to follow your hottest node. Load imbalance becomes a real problem in this space. The industry is doing a lot of interesting things there. The Mooncake folks are doing proactive KV migration so that if they can say, this node is more likely to get hotspotted, let me copy some stuff over.
NVIDIA's Dynamo is like, how much preference do I want to give it so that I can avoid this hotspot and so forth. It does make a difference. If things do fit in the cache, the KV-aware routing and session affinity does help. I ran a quick experiment with the same thing where I just had one GPU. My output metric was just, I was outputting 110 tokens per second. As soon as I went to two GPUs, my output, it didn't go to 220, it went to 128. Which means I was only 58% efficient. The same thing on two GPUs but saying, I'm going to put each session to the place that it went to before, it did 201 tokens per second. It's not quite double but it became pretty efficient. I tried that with three GPUs, I got to 278 instead of 330, and then with four I was at 324.
The session awareness does help but with some caveats. One of the core distributed systems principle is, do constant work. If you're trying to do constant work and you also want to try to scale your components independently of each other, with prefill and decode that becomes really hard, running on the same node. A new technique is emerging where people are saying we're going to turn inference into a distributed systems problem where we're going to do the prefill on one node and we're going to do the decode aspect on another node. This is called disaggregation. If you get this right, you can scale these very dynamically. You can improve yourself or you can protect yourself from a situation where a single long prefill thing causes all the other decodes to slow down because the GPUs are busy doing the prefill side.
Let's talk through from a component level how this works. NVIDIA has this framework called Dynamo, which unfortunately is very confusing because I worked on DynamoDB which has nothing to do with NVIDIA's Dynamo. NVIDIA's Dynamo, one of its centerpieces, it's got a routing layer. The routing layer is actually the pinnacle of all of these lifecycles. This is the web server that I referred to earlier. The request lands on the routing layer. The routing layer will go ask a node to say, can you take this prompt and tokenize it? Then it's going to go and see, does anybody have it? Who is the node that is most likely to have the longest part of this prefix available? Then it's going to send that node a prefill request. This is to generate the machine code that a model can understand. When that node has finished generating the prefill tokens, it will store it in a KV cache, and then it transfers metadata back to the router, and says, ok, the prefixes, prefills, these are the chunks that you need and here are the hosts where they're at.
That gets transferred over to the decode node which then goes and says, everybody, can I have this object? Can I have this object, and so forth? Then it loads it and then it can start the decode process. The nice thing here is that you can focus the nodes to be either completely prefill and be completely decode. When you bring up let's say a vLLM, your node can switch positions. You can say, ok node, I'm going to schedule a bunch of prefill jobs on you, and the next second I can say I'm going to schedule a bunch of decode jobs on you, and it's totally fine with that. This allows the router to actually become quite intelligent and dynamic in the way it's scheduling.
To make this a little bit more memorable as to why disaggregation is so useful, we're going to do a quick exercise that will hopefully help you remember it. I like to call it the ABCD exercise. I'm going to time you all. I want you to say the numbers 1 through 12 as fast as you can. I'm going to say 1, 2, 3, and then you are going to just say 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. That's your job. 1, 2, 3, go. Pretty fast. Now I want you to say A through L as fast as you can. L, I think, is the 12th letter. 1, 2, 3, go. Still pretty fast. A little slower than numbers but you remember your ABCs. I want to make it a little bit harder. Now I want you to say A1, B2, C3 and so forth.
1, 2, 3, go. Many of us just gave up halfway through. This is what happens when you're asking the same node to do multiple things at the same time. If you time yourself saying 1 through 12 and A through L and you added that together, I bet you that was less than a quarter of the time it took you to say A1, L12. You can cache it. If you do this enough times, you'll do it really fast. For the first time, you see this becomes a really hard problem. This ability to disaggregate is actually quite useful. You can have special purpose-built hardware for the prefill versus decode. They use different types of things. In fact, there are specific hardware that are now being built that are optimized primarily for the decode stage so that you can prefill elsewhere and then decode really fast. The distributed systems concepts that we talked about try to do bounded work, pinning the roles, balanced by the pressure, and then you can do constant work if you make your KV cache mobile.
Like if you don't say it runs locally on the same host. Some of these things may look like they will slow you down because now you have to do a network call. In isolation, they will make you slower because you have to go fetch it. If you're thinking about multi-tenancy, if you're thinking from a fleet-wide perspective, it improves your throughput and actually improves the latency that your users get because they're not sitting there waiting once in a while for that hotspot to conclude. These things about six months ago were still super controversial, but now it seems like everybody is converging on cache-aware placement remotely, disaggregation of prefill and decode, and then the mobile KV state.
Conclusion
I hope that what you took away from this is relevant to you even if you're not running your own inference stack today. Some reminders for you. Every token that you are using becomes state that somebody has to carry. If you feel the temptation because you're really nice and you like to say please, just know that's like an extra 72k that Anthropic has to carry, and you're hurting their margins. The fact that you're saying please probably is why they couldn't become a trillion-dollar company, were only 965. Just if only you didn't say please enough. That's state that everybody has to carry. When you're doing this, if you're familiar with The Office, be like Kevin. Why use more words when few words do trick. Use as few words as you possibly can. When you are inside your Claude Code session, like if you have side quests that you want to do, launch sub-agents or launch a different window and run it so that you're not diluting the context because of the prefix caching.
If you do this often enough, you will actually start to appreciate that the quality of your output will get better and you'll hit the compaction window a lot less frequently if you took your side quest and ran them as sub-agents so that you're not diluting your prefix cache. As users, long context is powerful, but use it deliberately and carefully. Don't fill it up with all these pleases. As builders, if you are running an inference stack and if you're not today, I suspect you will in the next couple of years because databases in the '90s, were really hard, and if you wanted a hardcore database, you had to go pay this company a lot of money to get a whole rack full of databases, and now people just turn on Postgres. This will get commoditized and you will run your own inference stacks soon enough. It'll become more common.
When you're doing that, think hard about what are the underlying scaling dimensions. Then as designers, just remember, when you're designing your system prompt, when you're doing prompt engineering, when you're deciding where to put the user input, just think about locality and think about what can you trim from the prompt without losing the meaning, so that your users can have a better experience even if you're providing the application and not just the inference stack. This is all called mechanical empathy. If you are aware of the system and how it operates underneath, you can drive it a lot better and a lot faster. Design for the state you create.
Questions and Answers
Participant 1: The question I have is, first off, I don't actually know, but does the KV caching hit rate affect the price you pay as a customer? Second is, I believe that in the MCP protocol, when you change this list of tools for your server, for example, that causes the prefix of the thing to change. Was that just a giant mistake, would you say?
Khawaja Shams: It's hard to say if that's a giant mistake or not. To answer your first question, Anthropic and even OpenAI, I think they give you like a 90% discount if you have a cache hit rate. That doesn't mean that it's ten times cheaper for them. It's just a convenient number. It's not exactly 90% off, but it's like 0.89 or something. They give you a pretty nice discount if you get a cache hit. In fact, if you use a proxy, you can actually see what your cache hit rate was. Designing for that is good.
To answer your second question, I'm not sure if messing up the prefix is a mistake or if it's deliberate to make the model perform better because you don't want the model to be distracted by the previous list of tools. Because you might also remove some tools as well. There are little techniques though that come into play, like, for instance, if you ask it for a file that's changed and the original file was already in the prompt, it'll send a git change rather than the entire new file. There's techniques like that that the Claude Code agent is taking into play. It's hard for me to speculate on whether resetting the prefix is a mistake or not. It might have been a deliberate choice.
Participant 2: I'm just trying to understand, like Redis key value, obviously, one then maps to a value over there. From what I understood from your talk, you are loading all of the context that was previously hit with the previously churns in every time a new query comes in. I'm just trying to understand the separation of the system prompt and then the original context and the new churn, how the actual similarity matching is happening there, and where it decides to include or not include things. Because your data goes to 35 seconds from 3 seconds. You figure if it was semantically matching, it would have a little bit more of a subtle gradient to it because some of it matches and some of it doesn't. It's just binary. It goes from 3 seconds to 38 seconds. Why is it going from 3 seconds to 38 seconds if it's semantically trying to evaluate the key-value store to do the matching?
Khawaja Shams: The name KV cache is quite unfortunate, especially for somebody who works in the caching world, KV means key value. Except it's not a HashMap lookup. It's not a value. The V is like a vector of everything that preceded behind it and embedded into a single floating point. You really need the entire prefix. You need the KV for the first token, the second token, the third token. If you have lost and if you have evicted even like one link in the chain, you have to go recompute the whole thing. The 38-second jump is because it's having to do precomputation. It's like the GPU is actually having to do a lot more computation because it lost some of the data. Today in a cache you can get very clever about simple techniques like LRU, which tend to be pretty good for eviction. For instance, if you evict the first block of a particular prompt or whatever, you may as well just get rid of the rest, because they're likely not going to match. It's not doing a semantic search or anything like that. It's looking for very precise values which end up being like these attention mechanisms that are referencing every token that preceded the particular token for which you're looking up the value.
Participant 2: That makes sense, but why is it always so binary? Why wouldn't it just be like, it's halfway through, the token got rejected and so then it's 20 seconds instead of 38 seconds. Why is it always 3 seconds to 38 seconds in every example?
Khawaja Shams: Ever have too many browser tabs and like your browser comes to a halt? That still happens today.
Participant 2: Every moment.
Khawaja Shams: Yes, so it's like that. It's just like it's thrashing and it's losing. It's evicting the data that it now has to go recompute and that's causing pipeline stalls and that's causing delays, because now it's causing you to use up more compute. There are other things that can happen too, like precompute could be very compute heavy. It could cause your GPU to start running hot and run into thermal throttling as well. Having specific hardware for prefill versus decode helps, like not having to run it simultaneously is also super useful.
See more presentations with transcripts
