AIXX Pedia › Tech Concept › KV Cache Inference Cache
KV Cache (Key-Value Cache) is the region of GPU memory used to store intermediate attention results during LLM inference — every token generated has its Key and Value cached for later tokens' attention computations to reference. KV Cache size grows linearly with context length — for long-context inference, the memory bottleneck is the KV Cache.
KV Cache is the 'hidden killer' in inference memory planning: a 7B model's weights may take only 14GB, but the KV Cache for a long context can eat another few dozen GB — counting only model weights when sizing memory is a classic beginner mistake. Large-memory cards (H200 NVL 141GB / MI325X 256GB) owe their advantage in long-context scenarios to their large KV Cache headroom.
| Item | Value / Basis |
|---|---|
| Growth Pattern | Linear with context length |
| Bottleneck Scenarios | Long context / large-batch concurrency |
| Optimization Techniques | PagedAttention / MQA / GQA / Quantization |
| Memory Share | Can exceed the weights themselves |
The memory advantage of the B300 288GB is most evident in long-context inference — its KV Cache headroom is over 3x that of the 80GB tier. But actual KV Cache size depends on batch size, context length, and model architecture (MHA vs GQA differ greatly) — size the memory spec using inference stress-test data for your target model before purchasing.