Skip to content
LLM Pricing Calc
GPU rental rates

LLM VRAM Calculator: How Much GPU Memory Do You Need?

Check the hardware requirements for running a local LLM: memory split into weights, KV cache and overhead, plus which consumer GPUs, Macs and cloud GPUs can run it.

Quantization
Context length
KV cache

Total VRAM required

49.1 GB

Llama 3.3 70B Instruct · Q4_K_M · 8K context · 1 user

KV cache: 320 KB / token

Model weights
42.8 GB
KV cache
2.68 GB
Runtime overhead
3.57 GB
Hardware compatibility
HardwareUsable memoryFitEst. speedCloud $/hr
RTX 4060 Ti 16GBConsumer GPU16 GBNeeds 4×~17 tok/s—
RTX 3090 24GBConsumer GPU24 GBNeeds 3×~43 tok/s$0.66
RTX 4090 24GBConsumer GPU24 GBNeeds 3×~46 tok/s$1.02
RTX 5090 32GBConsumer GPU32 GBNeeds 2×~54 tok/s$1.38
MacBook Pro M4 Max 64GBApple Silicon48 GBDoesn't fit——
MacBook Pro M4 Max 128GBApple Silicon96 GBFits~8 tok/s—
Mac Studio M3 Ultra 512GBApple Silicon384 GBFits~12 tok/s—
NVIDIA L4 24GBCloud GPU24 GBNeeds 3×~14 tok/s$1.32
NVIDIA L40S 48GBCloud GPU48 GBNeeds 2×~26 tok/s$1.72
NVIDIA A100 80GBCloud GPU80 GBFits~31 tok/s$1.39
NVIDIA H100 80GBCloud GPU80 GBFits~51 tok/s$2.39
NVIDIA H200 141GBCloud GPU141 GBFits~73 tok/s$3.29
NVIDIA B200 180GBCloud GPU180 GBFits~121 tok/s$5.49
AMD MI300X 192GBCloud GPU192 GBFits~80 tok/s$2.49

VRAM and memory bandwidth are manufacturer specs. Hourly rates are RunPod Community Cloud on-demand rates; other providers and regions differ. Apple Silicon usable memory assumes macOS's default GPU wired limit (~75% of unified memory). "Needs N×" means tensor-parallel across N identical GPUs. GPU rates: Official pricing ↗

Guide

How much VRAM do you need to run an LLM?

GPU memory, not compute, decides whether a model will run locally at all. If the model doesn't fit in VRAM, llama.cpp and Ollama spill layers into system RAM and generation slows from tens of tokens per second to a crawl. Total VRAM has three parts:

  • Weights = parameters × bits per weight ÷ 8. Q4_K_M averages ~4.85 bits, Q8_0 ~8.5 bits, FP16 16 bits.
  • KV cache = 2 × layers × KV heads × head dim × context × users × bytes per value. DeepSeek's multi-head latent attention stores one compressed 576-dim latent per layer instead, so its cache is far smaller.
  • Overhead ≈ 1 GB CUDA/Metal context + ~6% of weights for activations and scratch buffers, plus a little per extra user.

Model weights and quantization: Q4 vs Q8 vs FP16

Weights are the fixed cost. Every parameter must sit in memory, so a model's size in billions of parameters sets the floor. Quantization shrinks each parameter from 16 bits to 8 or about 4, trading a little accuracy for a much smaller footprint. Here is what that means for two popular Llama models at an 8K context:

VRAM by quantization for Llama 3.1 8B Instruct and Llama 3.3 70B Instruct at 8K context
QuantizationLlama 3.1 8B InstructLlama 3.3 70B Instruct
Q4_K_M (4-bit)7.21 GB49.1 GB
Q8_0 / FP8 (8-bit)11.1 GB83.2 GB
FP16 / BF16 (16-bit)19.0 GB153.4 GB

Q4_K_M is the default for local inference with Ollama and LM Studio: around 70% smaller than FP16 with a quality loss most people never notice in chat. Q8_0 is nearly lossless and a good choice when you have the memory. FP16/BF16 is what production servers such as vLLM run by default, and it is the baseline for fine-tuning. Below about 3 bits per weight, quality drops off quickly, so a smaller model at Q4 usually beats a bigger one squeezed to Q2.

KV cache growth with context length

The KV cache is the variable cost, and it is what catches people out. It stores attention keys and values for every token in the context window, so it grows linearly with context length and with the number of concurrent users. For Llama 3.3 70B Instruct at Q4, the KV cache takes 2.68 GB at an 8K context, 42.9 GB at 128K, and 171.8 GB if four users each hold a 128K context. A model that fits comfortably at 8K can run out of memory at long context. Quantizing the KV cache to 8 bits (the "KV cache" setting above) halves it, with little quality impact for most workloads.

Consumer vs cloud GPU sizing

Consumer GPUs (RTX 4060 Ti 16 GB, RTX 3090/4090 24 GB, RTX 5090 32 GB) run 7B–14B models at Q8, and the 24–32 GB cards handle 30B-class models at Q4. Llama 3.3 70B Instruct at Q4 needs about 49.1 GB, or 3× RTX 4090s. Apple Silicon trades speed for capacity: a 128 GB or larger Mac can load 70B-class and mid-size mixture-of-experts models that no single consumer GPU can hold, at lower tokens per second because unified memory has less bandwidth than GDDR or HBM. Cloud GPUs such as the L40S (48 GB), A100 and H100 (80 GB) and H200 (141 GB) are the right fit for FP16 serving, long contexts and many concurrent users, where KV cache, not weights, dominates memory.

Leave 10–15% headroom rather than filling VRAM to the last gigabyte; drivers, the display and fragmentation all take a share. If you are sizing a self-hosted LLM for production rather than a hobby setup, the next question is cost: the LLM API vs self-hosting calculator compares a rented GPU against per-token pricing, and the LLM pricing calculator shows what the same workload costs on hosted APIs. Hosted prices for open-weight models such as Llama, Qwen and DeepSeek are in the LLM price table.

More AI cost tools

LLM VRAM FAQ

How much VRAM does Llama 3.3 70B Instruct need?

At Q4_K_M with an 8K context, Llama 3.3 70B Instruct needs about 49.1 GB in total: 42.8 GB of weights, 2.68 GB of KV cache and 3.57 GB of overhead. That is more than any single consumer GPU: it fits on one NVIDIA A100 80GB, NVIDIA H100 80GB, NVIDIA H200 141GB, NVIDIA B200 180GB, or AMD MI300X 192GB, or a MacBook Pro M4 Max 128GB or Mac Studio M3 Ultra 512GB, or 3× RTX 4090s split with tensor parallelism. At FP16 it needs about 153.4 GB, which takes 3× H100 80GB.

How much VRAM do I need to run an LLM?

Roughly: parameters (billions) × bits per weight ÷ 8 = GB for the weights. Add the KV cache, which grows with context length and concurrent users, plus 1–2 GB of runtime overhead. A 70B model at Q4_K_M needs about 43 GB for weights alone, so it does not fit on a single 24 GB RTX 4090.

What is the KV cache and why does it matter?

The KV cache stores the attention keys and values for every token in the context so they are not recomputed. Its size = 2 × layers × KV heads × head dimension × context length × bytes per value × users. At 128K context it can be larger than the model weights themselves, which is why long-context inference often runs out of memory.

Which quantization should I use?

Q4_K_M is the most popular choice for local inference: about 70% smaller than FP16 with a small quality loss. Q8_0 is nearly lossless at half the size of FP16. FP16/BF16 is the original precision used by most production serving stacks such as vLLM.

Do MoE models need less VRAM?

No. Mixture-of-experts models such as Qwen3-235B-A22B or gpt-oss-120b must keep all experts in memory, so VRAM depends on total parameters. Only speed benefits from the smaller number of active parameters, because fewer weights are read per token.

How much of a Mac's unified memory can the GPU use?

By default macOS lets the GPU wire roughly 65–75% of unified memory. This calculator assumes 75% (48 GB on a 64 GB MacBook Pro, 96 GB on 128 GB). You can raise the limit with sudo sysctl iogpu.wired_limit_mb, at your own risk.

Is this the same VRAM Ollama and llama.cpp use?

Close. Ollama and llama.cpp load GGUF files, whose size matches the weights figure here for the same quantization. Ollama also allocates the KV cache for the full context you set with num_ctx, and extra copies when it serves parallel requests, so set the context and users fields to match your Ollama settings. If the total exceeds your VRAM, Ollama offloads layers to system RAM and generation slows sharply.

How accurate is the tokens-per-second estimate?

It is a memory-bandwidth estimate for a single user: bandwidth × ~65% efficiency ÷ bytes of active weights read per token. Real speed depends on the inference engine, batch size, and whether the model is split across GPUs. Treat it as a ballpark.