Guide
How much VRAM do you need to run an LLM?
GPU memory, not compute, decides whether a model will run locally at all. If the model doesn't fit in VRAM, llama.cpp and Ollama spill layers into system RAM and generation slows from tens of tokens per second to a crawl. Total VRAM has three parts:
- Weights = parameters × bits per weight ÷ 8. Q4_K_M averages ~4.85 bits, Q8_0 ~8.5 bits, FP16 16 bits.
- KV cache = 2 × layers × KV heads × head dim × context × users × bytes per value. DeepSeek's multi-head latent attention stores one compressed 576-dim latent per layer instead, so its cache is far smaller.
- Overhead ≈ 1 GB CUDA/Metal context + ~6% of weights for activations and scratch buffers, plus a little per extra user.
Model weights and quantization: Q4 vs Q8 vs FP16
Weights are the fixed cost. Every parameter must sit in memory, so a model's size in billions of parameters sets the floor. Quantization shrinks each parameter from 16 bits to 8 or about 4, trading a little accuracy for a much smaller footprint. Here is what that means for two popular Llama models at an 8K context:
| Quantization | Llama 3.1 8B Instruct | Llama 3.3 70B Instruct |
|---|---|---|
| Q4_K_M (4-bit) | 7.21 GB | 49.1 GB |
| Q8_0 / FP8 (8-bit) | 11.1 GB | 83.2 GB |
| FP16 / BF16 (16-bit) | 19.0 GB | 153.4 GB |
Q4_K_M is the default for local inference with Ollama and LM Studio: around 70% smaller than FP16 with a quality loss most people never notice in chat. Q8_0 is nearly lossless and a good choice when you have the memory. FP16/BF16 is what production servers such as vLLM run by default, and it is the baseline for fine-tuning. Below about 3 bits per weight, quality drops off quickly, so a smaller model at Q4 usually beats a bigger one squeezed to Q2.
KV cache growth with context length
The KV cache is the variable cost, and it is what catches people out. It stores attention keys and values for every token in the context window, so it grows linearly with context length and with the number of concurrent users. For Llama 3.3 70B Instruct at Q4, the KV cache takes 2.68 GB at an 8K context, 42.9 GB at 128K, and 171.8 GB if four users each hold a 128K context. A model that fits comfortably at 8K can run out of memory at long context. Quantizing the KV cache to 8 bits (the "KV cache" setting above) halves it, with little quality impact for most workloads.
Consumer vs cloud GPU sizing
Consumer GPUs (RTX 4060 Ti 16 GB, RTX 3090/4090 24 GB, RTX 5090 32 GB) run 7B–14B models at Q8, and the 24–32 GB cards handle 30B-class models at Q4. Llama 3.3 70B Instruct at Q4 needs about 49.1 GB, or 3× RTX 4090s. Apple Silicon trades speed for capacity: a 128 GB or larger Mac can load 70B-class and mid-size mixture-of-experts models that no single consumer GPU can hold, at lower tokens per second because unified memory has less bandwidth than GDDR or HBM. Cloud GPUs such as the L40S (48 GB), A100 and H100 (80 GB) and H200 (141 GB) are the right fit for FP16 serving, long contexts and many concurrent users, where KV cache, not weights, dominates memory.
Leave 10–15% headroom rather than filling VRAM to the last gigabyte; drivers, the display and fragmentation all take a share. If you are sizing a self-hosted LLM for production rather than a hobby setup, the next question is cost: the LLM API vs self-hosting calculator compares a rented GPU against per-token pricing, and the LLM pricing calculator shows what the same workload costs on hosted APIs. Hosted prices for open-weight models such as Llama, Qwen and DeepSeek are in the LLM price table.