What is an LLM gateway? Tasks, limits and how it differs from a RAG engine

What is an LLM gateway?

Many companies no longer use just one language model. One team works with a large hosted model, another with a small local one, and for some tasks a cheaper model is good enough. Each application would then have to manage its own keys, providers and error handling.

An LLM gateway takes that work away. It sits between the applications and the models and offers them a shared endpoint. The application says what it needs; the gateway forwards the request to the right model and returns the answer. Think of a gateway as a switchboard: everyone calls the same number, the switchboard connects them and keeps a record of the calls.

Gateways exist as open-source software you run yourself and as a service. Exactly which features are included differs from product to product.

In shortA layer in between: one endpoint for many language models.

What an LLM gateway takes care of

Most gateways cover five tasks. Not every product does everything; the table shows what matters.

For IT, this is mainly a matter of order: instead of many separate integrations, there is one place where models, keys and costs are managed.

In shortRouting, keys, costs, logs and rules in one place.

LLM gateway or RAG engine: what is the difference?

Gateways and RAG engines are often mentioned together, but they solve different problems. The gateway decides which model answers. The RAG engine decides what it answers from.

A RAG engine is therefore more your AI’s memory than its traffic control. It reads in your stores, builds a search index and passes your documents to the model so it answers from them instead of guessing. How that works is explained in the post on the RAG system.

In shortThe gateway controls the route to the model, the RAG engine supplies the knowledge.

When you need a gateway, a RAG engine or both

Whether you need a gateway depends on how many models and applications you run. Whether you need a RAG engine depends on whether the AI should know your company knowledge.

To start with, a RAG engine with one or two models is often enough. A gateway pays off when many applications and teams use models and you want to control costs and rules centrally. If your gateway offers a compatible endpoint, it can usually be added as a custom model in a RAG engine; check the details in the documentation of both products.

In shortMany models and apps: gateway. Company knowledge: RAG engine. Large setups: both.

Your own API keys and where your data goes

Many teams use models with their own API key, also called bring your own key (BYOK). The model provider then bills you directly, and you keep control of the contracts. A gateway helps manage these keys centrally.

Important for data protection: a gateway carries not only questions but, with RAG, also passages from your documents. Gateway logs can therefore contain confidential content.

In shortWith your own key you are billed directly; check where questions and passages end up.

Language models in RAGSuite

RAGSuite is not an LLM gateway but a knowledge engine: it connects your sources, finds the right passages and delivers answers with sources. You choose the language model per project, from several hosted providers to a local model via Ollama. You can switch at any time; if the embedding model changes, you rebuild the search index with one click.

In RAGSuite Engine, model usage is included through credits (how credits work). Switch on your own API key and the plan price stays the same, with no credits used. The pricing page and the AI cost calculator show what that costs.

For AI tools that should query your knowledge, RAGSuite also provides an MCP server. That way, tools outside RAGSuite also get answers from your documents, with sources.

In shortRAGSuite is the knowledge engine; you choose the model per project, local included.