What is KV cache, and why is it becoming an AI memory bottleneck?

KV cache is the short-term memory large language models use during inference. It stores the model’s recent working state so the system does not have to recalculate the full prompt for every new token.

That sounds like a technical detail. But it's one of the biggest AI bottlenecks today.

As context windows get longer and more users hit a model at once, KV-cache memory demand rises fast.

For investors, the key question in AI inference is no longer just which chip can do the most math at the lowest cost. It is also which systems can store, move, and manage the most live memory efficiently.

KV cache is basically the model’s short-term memory

KV cache sounds technical. The idea is simple. When a model generates text, it has to keep track of what came before. Instead of doing that work over and over for each new token, it stores the useful pieces and reuses them.

That stored information is the KV cache.

Without it, the model would have to reread the whole prompt again and again for every token it writes. Generation would get much slower and much more expensive.

So this is not some minor feature. KV cache is the model’s short-term memory while it answers a request.

The problem is that this memory need keeps growing

KV-cache memory grows in two straightforward ways.

First, it grows as the context window gets longer. The more text the model has to keep in play, the more KV cache it needs.

Second, it grows as more people use the model at the same time. Each active user usually needs a separate live memory state.

So serving costs do not stay flat once a model is loaded. They rise with use. A model handling long prompts for many simultaneous users needs far more memory than one serving short prompts to a small number of people.

NVIDIA offers a useful example: for one Llama 3 70B user with a 128k-token context, the KV cache alone can take about 40 GB of memory. That is an enormous amount for one active session. Multiply that across many users, and memory becomes a hard limit fast.

That is why long context is not a free feature. It is a memory promise.

The hardware story is already moving in this direction

You can already see the shift in GPU products.

NVIDIA’s H100 has 80 GB of memory and 3.35 TB/s of memory bandwidth. The newer H200 pushes that to 141 GB and 4.8 TB/s. That is more than a routine upgrade. It signals that larger, faster memory is becoming a central selling point for AI inference.

Because inference speed is not only about how much math a chip can do. It is also about how much live model memory it can hold on chip, and how quickly it can read and move that memory.

If a server cannot keep enough KV cache ready, performance slips. Throughput drops. Latency rises. Or the system starts pushing data into slower memory.

That is why memory-heavy chips are getting more valuable for long-context AI workloads. In this part of the market, memory is becoming as important as compute.

Software also matters because it can make memory go further

This is not just a hardware story.

Software can create a real advantage by using memory more efficiently. A good example is vLLM’s PagedAttention system. Its value is not that it changes the model. It cuts wasted KV-cache memory and makes memory easier to share and manage across requests. The paper reports 2-4x throughput gains over some earlier systems at similar latency.

That is a big deal because better software lets the same hardware do more work.

NVIDIA’s TensorRT-LLM points in the same direction. It includes tools for KV-cache reuse, eviction decisions, moving some cache to host memory, limiting attention windows, and changing the cache data type. One telling detail: by default, it can give up to 90% of free GPU memory to KV cache.

That says a lot. Inference software is increasingly turning into memory-management software.

So the winners will not be only the companies with the best chips. They will also be the ones with the best serving software.

The next big step is shrinking KV cache itself

Managing memory better is one step. Cutting the memory requirement is even better.

That is where compression comes in, as explored in KV-cache compression only matters when serving platforms make it usable.

Google Research’s TurboQuant solved one AI memory bottleneck, not the whole problem. Google says it can shrink KV-cache memory by at least 6x on long-context benchmarks, push the cache down to very low bit levels without retraining the model, preserve accuracy, and still improve speed. It also reports up to 8x faster attention-logit computation in one H100 comparison.

If results like that hold up in real deployments, they will matter a lot. Compression does not just organize memory better. It reduces the amount of memory the system needs in the first place.

That could make compression one of the next major battlegrounds in AI inference.

Where the value is likely to go

This leads to a more useful market view than the broad claim that “AI infrastructure wins.”

The likely winners fall into three groups.

First, memory-rich chip vendors, especially NVIDIA, because long context and high concurrency directly raise the value of more GPU memory and higher bandwidth.

Second, inference and serving software providers that can cut wasted KV-cache memory, share memory across requests, and move data between fast and slow memory more efficiently.

Third, companies building compression tools that can shrink KV cache without hurting output quality. That area is earlier, but it could become very important because it attacks the memory problem at the source.

The losers are the companies still treating inference as only a compute race. Compute still matters. But in real-world serving, memory is becoming one of the main constraints.