LLM VRAM calculator: how much GPU memory does your local AI need?
The LLM VRAM calculator shows how much GPU memory your local AI really needs: not just for the language model, but for the whole RAG stack with KV cache, embedding model, reranker and overhead, for as many concurrent users as you enter. It also shows which GPU memory tier fits.
Nothing leaves your browser · as of 3 November 2026
Memory decides, not the processor.
A language model only runs smoothly if it sits entirely in the graphics card’s memory, the VRAM. If it does not fit, it becomes slow or does not start at all. That is why the first question for LLM hardware is not “how fast?” but “how much VRAM?”.
Many VRAM calculators only count the model for a single user. In production, every concurrent request adds a working memory, the KV cache.
A RAG platform also runs an embedding model for search and often a reranker that sorts the results. Both are small but need memory of their own. The calculator adds it all up.
How the calculator works.
All values in GB, with 1 GB = 1024³ bytes. The formulas follow published sources; where there is none, the table says “assumption”.
| Component | Formula | Basis |
|---|---|---|
| Model weights | Parameters × bits per weight ÷ 8. FP16 = 16, FP8 = 8, Q8_0 = 8.50, Q6_K = 6.56, Q5_K_M = 5.70, Q4_K_M = 4.89 bits | llama.cpp quantize docs |
| KV cache per token | 2 (keys and values) × layers × KV heads × head dimension × 2 bytes (FP16) or 1 byte (FP8) | Hugging Face Transformers, LLM optimisation guide; Ainslie et al., GQA (arXiv 2305.13245) |
| Total KV cache | KV per token × context per request × concurrent users | Kwon et al., PagedAttention (arXiv 2309.06180) |
| Runtime overhead | (weights + KV cache) × 15% + 1 GB per model process | Assumption, deliberately cautious |
| Embedding, reranker | Parameters × 2 bytes (FP16) × 1.15 + 1 GB per model | Assumption |
| Recommendation | Smallest memory tier where the need uses at most 90%; 1 GB more per extra GPU. Above 85% counts as tight. | vLLM docs, engine arguments |
Sources, retrieved 3 November 2026: Kwon et al., PagedAttention (arXiv 2309.06180) (800 KB of KV cache per token for a 13B model without GQA, which the calculator uses to check its formula); llama.cpp quantize docs (bits per weight, 14.96 GB at FP16 and 4.58 GB at Q4_K_M for an 8B model); vLLM docs, engine arguments (gpu_memory_utilization, default 0.9 or 0.92 depending on version; the calculator conservatively uses 0.9; plus the KV cache format); Jiang et al., mixture-of-experts model (arXiv 2401.04088) (MoE models keep all experts in memory).
Three typical scenarios, worked through.
Example 1 step by step: a 7–8B model at 4 bit takes 4.58 GB. The KV cache needs 128 KiB per token; at 8,192 tokens and 10 users that is 10.00 GB. With overhead, the language model comes to 17.76 GB, embedding and reranker to 2.22 GB each. Together 22.2 GB: a 24 GB GPU offers only 21.6 GB at 90%, so the pick is 32 GB.
| Scenario | Model | KV cache | Total | Recommendation |
|---|---|---|---|---|
| 1 · Team assistant: 7–8B, 4-bit, 10 users, 8,192 tokens | 4.6 GB | 10.0 GB | 22.2 GB | 1 GPU, 32 GB (69%) |
| 2 · Large model, small team: 70B, 4-bit, 5 users, 4,096 tokens | 40.2 GB | 6.3 GB | 58.9 GB | 1 GPU, 80 GB (74%) |
| 3 · Full precision in production: 70B, FP16, 25 users, 8,192 tokens | 131.4 GB | 62.5 GB | 228.4 GB | 4 GPUs × 80 GB (72%) |
All examples with a medium embedding model and reranker (~0.6B each) and KV cache at FP16. Percentages: share of GPU memory the stack uses.
Which GPU memory tier fits which stack?
The calculator recommends by memory size, not by model or manufacturer. “Usable” means 90% of the memory; the rest stays free as a buffer.
| Memory tier | Usable (90%) | Example that fits (with embedding and reranker) |
|---|---|---|
| 1 × 16 GB | 14.4 GB | 7–8B at 4-bit, 4 users, 4,096 tokens: 13.0 GB |
| 1 × 24 GB | 21.6 GB | 7–8B at 4-bit, 10 users, 8,192 tokens, no reranker: 20.0 GB |
| 1 × 32 GB | 28.8 GB | 7–8B at 4-bit, 10 users, 8,192 tokens: 22.2 GB |
| 1 × 48 GB | 43.2 GB | 7–8B at FP16, 10 users, 8,192 tokens: 34.1 GB |
| 1 × 80 GB | 72.0 GB | 70B at 4-bit, 5 users, 4,096 tokens: 58.9 GB |
| 2 × 48 GB | 86.4 GB | 30–34B at 4-bit, 10 users, 16,384 tokens: 72.9 GB |
| 2 × 80 GB | 144 GB | 7–8B at FP16 with FP8 KV cache, 50 users, 32,768 tokens: 137.6 GB (tight) |
| 4 × 80 GB | 288 GB | 70B at FP16, 25 users, 8,192 tokens: 228.4 GB |
| 8 × 80 GB | 576 GB | 70B at FP16, 25 users, 32,768 tokens: 444.1 GB |
With several GPUs, each GPU needs about 1 GB for itself, and the GPU count should divide the number of KV heads (8 for all presets here, 4 for the small MoE model).
What the calculator deliberately simplifies.
The calculator answers “what fits?”, not “what will the server show later?”. Many serving programs reserve free memory for the KV cache up front, so the display sits at around 90% almost regardless of load.
It calculates the worst case: all users fill their context at the same time. In practice the need is often lower.
To be honest: 15% runtime overhead and 1 GB per model process are cautious assumptions, not measurements. Real values depend on the serving software, its version and settings. The 4-bit figures were measured on an 8B model and may differ slightly for other sizes. Large indexing runs with the embedding model can briefly need more memory. Check the numbers with a test run before you buy.
Questions about the LLM VRAM calculator.
How much VRAM does a 70B model need?
The weights of a 70B model alone take about 131 GB at FP16, about 66 GB at FP8 and about 40 GB at 4 bit (Q4_K_M). The KV cache comes on top: about 2.5 GB per user at 8,192 tokens of context. At 4 bit with 5 concurrent users, 4,096 tokens, embedding and reranker, the calculator gives 58.9 GB, which fits on one GPU with 80 GB.
Is a 16 GB graphics card enough for a local LLM?
For a 7–8B model at 4 bit and a few users, yes: with 4 concurrent users, 4,096 tokens, embedding and reranker the stack needs 13.0 GB. For 10 users at 8,192 tokens it is already 22.2 GB, so you need 32 GB. Larger models fit into 16 GB only with a very small context or not at all.
Why does the KV cache need so much memory?
For every token in the context, the model stores intermediate values (keys and values) in every layer. For a 7–8B model that is 128 KiB per token, so 1 GB per request at 8,192 tokens. The cache grows with context length and with every concurrent request; the model weights stay the same.
What does LLM quantisation achieve?
Quantisation stores each weight with fewer bits. According to llama.cpp, 4-bit Q4_K_M uses about 4.89 bits per weight, just under a third of the memory of FP16. The KV cache stays at 16 bit unless you explicitly switch it to FP8. Answer quality can drop slightly; test it with your own questions.
What counts as a concurrent user?
A request that is being answered right now, not a registered account. 200 employees with access rarely create more than a few requests at the same time. In the serving software the value corresponds to the number of parallel sequences, such as max_num_seqs.
How accurate is the LLM VRAM calculator?
Weights and KV cache follow fixed formulas and match published figures, such as 14.96 GB at FP16 and 4.58 GB at 4 bit for an 8B model. The runtime overhead of 15% and 1 GB per model process are deliberately cautious assumptions. Depending on the serving software, the real need can be higher or lower.
Can I run the embedding model and the language model on separate GPUs?
Yes. According to the calculator, a medium embedding model or a medium reranker needs about 2.2 GB each, including overhead. On a small GPU of their own they free memory for the language model and the KV cache. In that case, size both parts separately.
Are my inputs stored or sent anywhere?
No. The calculator runs entirely in your browser. No inputs are stored or sent to a server.
Your local AI, sized to your hardware.
Mihaela checks with you which model, which quantisation and how many concurrent users fit your hardware, before you run RAGSuite on your own servers.