LLM VRAM calculator: weights, KV cache, and which GPUs fit

The formula

Serving memory has three parts: the weights, the KV cache, and a runtime allowance. Only the KV cache depends on your traffic.

weights            = parameters x bytes per parameter
                     (quantized formats keep embeddings and the LM head at 16-bit)
KV cache per token = 2 x layers x KV heads x head dimension x bytes per value
KV cache in use    = KV per token x tokens per sequence x concurrent sequences
fits when          weights / TP + allowance + KV cache per GPU <= gpu_memory_utilization x GPU memory

The 2 counts keys and values. TP is the tensor-parallel size, the number of GPUs one copy of the model is split across.

Worked example. Llama 3.1 8B has 32 layers, 8 KV heads and a head dimension of 128, so at BF16 the cache is 2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes (128 KiB) per token. A sequence at 8,192 tokens holds exactly 1 GiB. The weights are 8,030,261,248 parameters x 2 bytes = 14.96 GiB. An RTX 4090 has 24 GB; vLLM's default gpu_memory_utilization of 0.92 makes 22.08 GiB usable. Take away the weights and a 2 GiB allowance and 5.1 GiB is left, which is room for five concurrent 8k sequences. That is arithmetic from the formula, not a measurement.

KV heads, not attention heads

The cache stores keys and values for KV heads, and most current models have far fewer of those than query heads. Llama 3.1 8B has 32 query heads sharing 8 KV heads (grouped-query attention), which makes its cache four times smaller than if every head kept its own keys and values. Llama 3.3 70B also has only 8 KV heads, across 80 layers: 320 KiB per token. Qwen3 32B has 8 KV heads across 64 layers: 256 KiB per token. Calculators that use num_attention_heads overstate the cache by the grouping factor.

What changes the answer

• Paged attention. vLLM allocates the cache in blocks (16 tokens each by default) as sequences grow, so memory follows the tokens actually in use, not the maximum length you allow. That is why the calculator multiplies by your real context length. vLLM still refuses to start if a single sequence at max_model_len cannot fit.
• Prefix caching. Sequences that share a prompt prefix, such as a long system prompt, share its cache blocks. The calculator assumes no sharing, so for chat traffic with a common prefix it is an upper bound.
• KV-cache quantization. An FP8 cache (--kv-cache-dtype fp8 in vLLM) halves cache memory. The effect on output quality depends on the model and the task. Long-context retrieval is a reasonable place to look first, but measure it on your own evaluation before you rely on it.
• Weight quantization is not "parameters x bits". Quantized checkpoints usually keep the embedding table and LM head at 16-bit, and group-wise formats store scales. For Llama 3.1 8B, the embeddings and LM head are 1.05 billion of the 8.03 billion parameters (128,256-token vocabulary x 4,096 hidden x 2). INT4 at group size 128 costs 4.16 bits per weight once the scale and zero point are counted. So the 4-bit model is 5.33 GiB, not 4 GB.
• Mixture of experts. Every expert has to be in memory, so a MoE model's memory follows its total parameters. Its decode speed follows its active parameters. Qwen3 30B-A3B needs 61 GB (56.9 GiB) at BF16 but reads only about 3.3 billion parameters per token.
• Sliding-window layers. gpt-oss alternates full-attention layers with 128-token sliding-window layers, and Gemma 4 runs 1,024-token windows on five of every six layers. A sliding-window layer only ever caches its window. Engines with a hybrid KV-cache manager (vLLM V1) allocate it that way, and so does the calculator. At 128k tokens, Gemma 4 31B needs 10.8 GiB per sequence. An engine that cached the full context on every layer would need 110 GiB.
• Linear attention. Qwen3.5, 3.6 and 3.8 use Gated DeltaNet on three of every four layers. Those layers keep a fixed recurrent state per sequence instead of a growing cache. For Qwen3.8 27B, that state is 147 MiB per sequence, stored in FP32 as its config specifies. The full-attention layers add 64 KiB per token. At short context the state dominates; at long context the cache does.
• Multi-head latent attention (MLA). DeepSeek-V3.2, Kimi K2 and GLM-5.3 cache a single 576-value latent per token per layer, shared by all heads. That makes the cache small: 70,272 bytes per token for Kimi K2 at BF16, against 327,680 for Llama 3.3 70B. DeepSeek-V3.2 also caches a 132-byte FP8 key per layer for its sparse-attention indexer.

Tensor parallelism does not always split the cache

With tensor parallelism, vLLM splits KV heads across GPUs, but a GPU never holds less than one head. When the tensor-parallel size reaches the number of KV heads, heads are replicated. Qwen3.5 122B-A10B has only 2 KV heads on its full-attention layers, so going from 2 to 8 GPUs barely shrinks the per-GPU cache (457 MiB to 402 MiB per 32k-token sequence; the part that still shrinks is the linear-attention state).

MLA is the extreme case. The latent is shared by all heads, so every tensor-parallel GPU keeps the full cache. At 32k tokens, a DeepSeek-V3.2 sequence takes 2.4 GiB on each of 8 GPUs. A Llama 3.3 70B sequence of the same length takes 10 GiB in total, which splits to 1.25 GiB per GPU. This is why large MLA deployments use data-parallel attention. The calculator models tensor parallelism only, and says which GPU counts a model's heads allow.

The runtime allowance

When vLLM starts, it loads the weights. Then it runs a profiling forward pass to measure peak activation memory and accounts for memory outside PyTorch's allocator (the CUDA context, NCCL buffers) and for CUDA graphs. Whatever is left of gpu_memory_utilization x GPU memory becomes KV cache. The calculator stands in for the middle part with one number per GPU. The default of 2 GiB is a stated round number, not a measurement. vLLM logs the real figures at startup, so replace it with yours.

GPU memory is the marketed figure treated as GiB. The capacity the driver reports can be a few percent lower, for example with ECC enabled, so leave headroom if you are within a few percent of the limit.

The decode ceiling column

Generating one token means reading every active weight and the sequence's whole cache from GPU memory. Memory bandwidth divided by those bytes is therefore an upper bound on single-stream decode speed. For Llama 3.1 8B at BF16 on an H100 SXM (3,350 GB/s) at 8k context, that is 3,350 GB/s / 17.1 GB = 196 tokens per second. Real engines land below it. Batching reads the weights once for many sequences, so total throughput goes far above one stream. Use the ceiling to compare GPUs, and the break-even calculator with your own measured throughput to compare costs. Measured numbers on this site are planned, not published.

What the calculator does not model

• Pipeline and expert parallelism, and data-parallel attention.
• Speculative decoding. A draft model, or a model's multi-token-prediction layer, adds weights and its own cache.
• LoRA adapters, CPU offload, and GGUF k-quant formats (which mix bit widths per tensor).
• Activations from image or audio inputs. The weights of the multimodal presets do include the vision encoder.

Where to run it

If the model fits a 24 to 32 GB card, hourly rentals of RTX 4090 and RTX 5090 cards are the cheapest way to test it: RunPod lists both, and Vast.ai is a marketplace where individual hosts set the price. For 70B-class models and long contexts you need 80 to 180 GB data-center GPUs from providers such as Lambda or DigitalOcean. The GPU price table compares on-demand prices from eleven GPU clouds and from AWS, Google Cloud and Azure, most of which pay this site nothing.

Before you rent anything, run the break-even calculator. For many open models, an API serving the same model costs less than one idle GPU.

Also available as Markdown.