Local LLM hardware: plan your own AI server for the company
Your own AI server brings the language model and document search into your company network. Three questions decide the local LLM hardware you need: how many people ask at the same time, how large is the model, and which documents should the AI know? This guide turns the answers into memory, form factor, power, software and operation.
Vendor-neutral · no hardware prices · as of 4 November 2026
Key points
- The key figure is
GPU memory , not compute. It has to hold the model, a working memory for every running request (KV cache), the embedding model and headroom. Concurrent requests drive the need: with 10 requests of 8,192 tokens each, the KV cache of a 7–8B model needs 10 GB, more than twice the model at 4 bit (4.6 GB).- For up to 5 concurrent requests with a 7–8B model, one GPU with 24 GB is enough. With a 13–14B model and 20 requests you need 80 GB or two GPUs.
- Power turns into heat in the room, and rack servers are loud. Plan room, cooling and UPS
before you order, and name owners for updates, monitoring and backups.
Three questions before you choose hardware.
A server for local AI is more than a computer with a big graphics card. It serves many requests at once, runs around the clock and needs looking after. Answer the three questions and you can derive memory, form factor and software instead of buying by datasheet.
On the third question: with RAG, every question sends the matching passages into the model. With our planning values (system prompt 500, five passages of 512, history 1,000 and answer 1,000 tokens) that is 5,060 tokens. 8,192 tokens of context per request is therefore a sensible planning value, and this guide uses it.
How many ask at the same time?
Count requests being answered at the same moment, not accounts. Each one needs its own memory for its context.
How large is the model?
Parameter count and quantisation set the baseline: a 7–8B model at 4 bit takes 4.6 GB, a 70B model 40.2 GB.
Which documents?
Volume and type decide disk space, indexing time and how much text goes into the model per question.
What has to fit in GPU memory.
When choosing a GPU for AI, memory (VRAM) comes first. It holds four blocks: the
The example shows: with 10 concurrent requests, the KV cache is the largest block, not the model. So never plan with “model size = memory need”. A 10% buffer stays free. That is deliberately cautious: without a setting of its own, vLLM uses up to 92% of GPU memory (gpu_memory_utilization).
If a model does not fit in GPU memory, llama.cpp can offload parts to the CPU and system memory, according to the llama.cpp project page. That works but is much slower and no answer for many concurrent users.
AI hardware by team size: four size classes.
For each class, the table uses the upper limit of concurrent requests and the worst case: all requests use 8,192 tokens of context at once. A medium embedding model and a reranker come on top. The figures come from the same formulas as the LLM VRAM calculator; check your own numbers there.
| Concurrent requests | Model class (example) | Memory need | GPU memory | CPU · RAM · NVMe | Network |
|---|---|---|---|---|---|
| 1 to 5 | 7–8B, 4-bit | 16.5 GB | 1 GPU, 24 GB (69%) | 8–16 cores · 64 GB · 2 × 1 TB mirrored | 1 GbE |
| 5 to 20 | 13–14B, 4-bit | 49.6 GB | 1 GPU, 80 GB (62%), or 2 × 48 GB | 16–32 cores · 128 GB · 2 × 2 TB mirrored | 10 GbE |
| 20 to 100 | 30–34B, 4-bit | 256.9 GB | 4 GPUs × 80 GB (81%) | 32–64 cores · 512 GB · 2 × 4 TB mirrored | 10 to 25 GbE |
| over 100 | 70B, FP8 | 368.5 GB at 100 requests | 8 GPUs × 80 GB (59%), beyond that several servers | 64+ cores · 1 TB+ · 4 TB+ mirrored | 25 GbE+, fast links between servers |
Memory need and GPU memory: own calculation with the LLM VRAM calculator’s formulas (KV cache at FP16, 15% runtime overhead, 1 GB per model process, 90% usable; percentage = utilisation). Model choice has a big effect: an MoE model with ~30B parameters in total and ~3B active needs 124.4 GB at FP8 for 100 requests and fits on 2 × 80 GB. Use several GPUs only if the model and its KV cache together do not fit on one; with many concurrent requests, one model instance per GPU can be an alternative to a single distributed instance. Across several servers, the vLLM docs on parallelism and scaling call for a fast link between nodes, such as InfiniBand. All memory figures in GB = 1024³ bytes, as in the LLM VRAM calculator.
To be honest: the CPU, RAM, NVMe and network columns are our own experience values, not vendor figures. As a rule of thumb we plan system memory at least equal to total GPU memory, so models load quickly and the database, search index and platform have room. The NVMe drives hold model files, original documents, index and database, and are mirrored so a drive failure does not stop operation.
Workstation, rack server or rented server: build, buy or rent?
Building an AI server yourself or buying a workstation is the fastest route for tests, for home use and for small teams. From about five concurrent requests and in continuous operation, a rack server in a server room becomes the norm. Without a server room, you rent a dedicated GPU server in a German data centre and run your own software on it.
And if you decide not to run hardware at all: the RAGSuite Engine offers the same platform with a data centre in Germany.
| Criterion | Workstation | Rack server | Rented GPU server |
|---|---|---|---|
| Good for | Tests, pilots, 1 to 5 requests | Team operation, from about 5 requests | Operation without a server room |
| Location | Office or utility room | Server room or server cabinet | Provider’s data centre in Germany |
| GPUs | usually 1 to 2 | up to 8, depending on chassis | depending on the offer |
| Noise | can be office-friendly | usually too loud for offices | not your concern |
| Power and cooling | Heat stays in the room | dedicated circuit, air conditioning, UPS | with the provider |
| Maintenance | usually yourself, no on-site service | vendor support, remote management, spare parts | hardware by the provider, software by you |
| Data | physically with you | physically with you | with the provider, usually under a data processing agreement |
Power, heat, noise and UPS: the part that is often forgotten.
Practically all the power a server draws is released into the room as heat. A simple example: a server with a continuous load of 1 kW uses 24 kWh a day and 8,760 kWh a year, and the air conditioning must remove that 1 kW without a break. The real figures are in your server’s datasheet.
The BSI, Germany’s federal cyber security agency, provides a framework for the room. For a small company with one server in its own room, the BSI IT-Grundschutz module INF.5 (2023 edition, German) says it is often enough to meet the requirements of INF.5 instead of module INF.2 (data centre and server room); a specially equipped cabinet is also possible. Requirements include controlled access (INF.5.A3), temperature and humidity within the limits in the devices’ datasheets (INF.5.A10) and a UPS whose runtime is enough for a safe shutdown (INF.5.A16).
Power
GPU servers draw a lot of power under load. Have the ratings from the datasheet checked and the circuit planned by a qualified electrician.
Heat
Cooling must remove the power drawn continuously, also at night and at weekends, when building air conditioning is often turned down.
Noise
Rack servers cool with fast fans and are usually too loud for work rooms. Plan a separate room or an insulated cabinet.
UPS
The UPS bridges short outages and shuts the server down cleanly during longer ones. Batteries age: test regularly and resize after changes.
The software stack: five layers.
Five layers sit on the hardware. Keep the driver and inference server versions matched; most faults happen there, not in the model.
The inference server makes the model available on the network. vLLM is built for many concurrent requests, runs on Linux according to the vLLM installation guide (not natively on Windows, only via WSL) and offers an official Docker image. Ollama is the easiest start; according to the Ollama FAQ it processes one request per model at a time by default (OLLAMA_NUM_PARALLEL), and the memory need grows with parallelism times context length. llama.cpp runs on very different hardware.
| Layer | Job | What to watch |
|---|---|---|
| Operating system | Linux distribution with long-term support (LTS) | vLLM does not support Windows natively |
| GPU driver | Driver and compute library from the GPU maker | Version matched to the inference server; test updates first |
| Container runtime | Docker with Compose v2 | GPU access inside containers usually needs an add-on package from the GPU maker |
| Inference server | vLLM, Ollama or llama.cpp; plus embedding model and reranker | Set concurrent requests; interface internal only |
| RAG platform | RAGSuite Community Edition: ingest documents, search, answers with sources; roles and project access in the Enterprise Edition | PostgreSQL, Redis, ChromaDB; web interface :9191, API :9090 |
Running the AI server: updates, monitoring, backups, access, offline.
After installation the real work begins. An AI server needs the same care as any other server, plus a few AI-specific metrics: how full is the KV cache, and how many requests are waiting?
On data protection, just this: your own server keeps data in-house, but it replaces neither a permission concept nor documentation. The GDPR-compliant AI guide explains what is involved.
Updates
Update driver, inference server, models and platform separately, on a test system first. With RAGSuite, ragsuite update --restart keeps database and configuration.
Monitoring
Watch GPU memory, temperature, running and waiting requests. According to the vLLM metrics docs, vLLM exposes metrics at /metrics, such as KV cache usage.
Backups
Back up database, search index, configuration and original documents. Keep model files in an internal mirror so a rebuild works without internet.
Access
Make the model interface reachable only by the platform. By default Ollama listens only on 127.0.0.1:11434. Sign-in, with two-factor authentication if you want it, is handled by the platform; roles and per-project access come with the Enterprise Edition.
Offline mode
Turn online features off (Ollama: OLLAMA_NO_CLOUD=1), load models from the internal mirror and test operation with the network disconnected.
Before you order.
Tick what is done. Nothing is stored.
Questions about local LLM hardware.
What does your own AI server cost?
It depends mainly on five drivers: the GPU memory tier and number of GPUs, the server chassis with power supplies and cooling, system memory and NVMe storage, power and air conditioning over the whole service life, and staff time for operation and updates. The biggest lever is GPU memory, and that is driven mostly by concurrent requests. Many models and tools are open source, including the RAGSuite Community Edition.
Is a graphics card with 24 GB enough?
For a 7–8B model at 4 bit and up to 5 concurrent requests of 8,192 tokens each, yes: with an embedding model and reranker the stack needs 16.5 GB, which uses 69% of the card. With 10 concurrent requests it is 22.2 GB, which only fits without the reranker (20.0 GB). A 30–34B model at 4 bit does not fit with an embedding model; the weights alone take 18.7 GB.
Do I need a server, or is a laptop enough?
For a first test with a small model, a laptop is enough. For a team you need a server: it runs continuously, is reachable on the company network, serves several requests at once and can be monitored and backed up. A laptop quickly becomes a bottleneck with several users.
How many users can an AI server handle?
What matters is not the number of accounts but how many requests are being answered at the same moment. Every concurrent request needs its own memory for its context. Whatever does not fit waits in a queue. The LLM VRAM calculator shows how many concurrent requests fit your hardware.
Can I build an AI server myself?
Yes, for tests and small teams a self-built workstation is a common route. Make sure of a strong enough power supply, airflow and room for the GPU, enough system memory and mirrored NVMe drives. For continuous operation in a company, a server with vendor support, remote management and spare parts is usually the more robust choice.
What software does an AI server need?
Five layers: a Linux operating system with long-term support, the GPU driver, a container runtime such as Docker, an inference server such as vLLM, Ollama or llama.cpp, and a RAG platform that reads in your documents, searches them and shows answers with sources, such as the RAGSuite Community Edition.
Does your own AI server work without internet?
Yes, if the models, embedding model and platform run locally. You only need the internet for downloads and updates, and even those can come from an internal mirror. According to its documentation, Ollama’s cloud features can be turned off with OLLAMA_NO_CLOUD=1. Test offline operation with the network disconnected.
Buy, or rent a GPU server?
A rented dedicated GPU server in a German data centre saves you the server room, power and cooling planning and hardware maintenance. You still run the software yourself. Buying tends to pay off if the data must stay physically on your premises or the hardware will be evenly used for years. When renting, you usually need a data processing agreement.
Your AI server, sized to your team and documents.
Mihaela reviews user numbers, model and documents with you and derives hardware and operation from them, before you order.