SECTIONS
EDITIONS
NETWORK
AIXX Pedia The encyclopedia of compute hardware

AIXX Pedia › Tech Concept › KV Cache Inference Cache

KV Cache Inference Cache

KV CACHE · Technical Concept Entry

Overview OVERVIEW

KV Cache (Key-Value Cache) is the region of GPU memory used to store intermediate attention results during LLM inference — every token generated has its Key and Value cached for later tokens' attention computations to reference. KV Cache size grows linearly with context length — for long-context inference, the memory bottleneck is the KV Cache.

KV Cache is the 'hidden killer' in inference memory planning: a 7B model's weights may take only 14GB, but the KV Cache for a long context can eat another few dozen GB — counting only model weights when sizing memory is a classic beginner mistake. Large-memory cards (H200 NVL 141GB / MI325X 256GB) owe their advantage in long-context scenarios to their large KV Cache headroom.

Key Figures KEY NUMBERS

ItemValue / Basis
Growth PatternLinear with context length
Bottleneck ScenariosLong context / large-batch concurrency
Optimization TechniquesPagedAttention / MQA / GQA / Quantization
Memory ShareCan exceed the weights themselves

What It Means on the B300 ON B300

The memory advantage of the B300 288GB is most evident in long-context inference — its KV Cache headroom is over 3x that of the 80GB tier. But actual KV Cache size depends on batch size, context length, and model architecture (MHA vs GQA differ greatly) — size the memory spec using inference stress-test data for your target model before purchasing.

Boundaries and Caveats LIMITS

  • KV Cache cannot be compressed to zero: Quantizing the KV Cache (FP8/INT4) saves memory but can cost accuracy — use with caution in accuracy-sensitive workloads.
  • Isolation risk in multi-tenant sharing: multiple users share the same card's KV Cache space — one user's long context can crowd out space for others.

Video Overview VIDEO

References REFERENCES

  1. NVIDIA official site — Blackwell Ultra / HGX B300 product page https://www.nvidia.com/en-us/data-center/hgx/
  2. Market and Sales Data: AIXX Exchange (Market data section of this library)
Versions and errata | This entry is maintained by the AIXX Pedia editorial team, AI-assisted drafting with human fact-checking. Updated 2026-10-20 · Version v1.0 · All revisions are logged and traceable.
This Library
Editorial PolicyAbout This Library (Disclosure of Interests)Corrections Channel
Editorial Constitution Summary
Every claim sourcedDrawbacks always stated · corrections publicly loggedNo paid entries acceptedZero traffic-funneling or sales elements within entriesMarket data attached only as data modules
AIXX Family
AIXX PediaAIXX ExchangeAIXX News
Market data source: AIXX Exchange · © 2026 AIXX Pedia · An AIXX Product