ToolsComparison
Five LLM gateways compared: routing, failover, governance and where each fits
LLM gateways sit between your applications and model providers to handle routing, keys, failover, budgets and logging. We compare Bifrost, LiteLLM, Portkey, Kong AI Gateway and Cloudflare AI Gateway on the decisions that actually differ.
Once a company is calling more than one model provider from more than one application, the same code appears everywhere: credential handling, retries, fallbacks, spend tracking, logging. An LLM gateway collapses that into one service. The category has matured quickly, and the products now differ less on the basic feature list than on architecture, deployment model and operating philosophy.
This comparison looks at five gateways that represent the main approaches: two self-hosted open-source gateways, one hosted control plane, one extension of an existing API gateway and one edge-network offering. Details reflect public documentation at the time of writing; verify current specifics before deciding.
Figure 1: The value of a gateway is that this path is configured once instead of reimplemented in every application.
The contenders
| Gateway | Model | Written in | Best fit |
|---|---|---|---|
| Bifrost | Self-hosted, open source (also hosted) | Go | High-throughput and latency-sensitive workloads; teams wanting a single binary |
| LiteLLM | Self-hosted, open source (also hosted) | Python | Broadest provider coverage; Python-native teams |
| Portkey | Hosted control plane (self-host option) | Managed | Teams that want observability and guardrails without running infrastructure |
| Kong AI Gateway | Plugin on Kong Gateway | Lua/Go on Kong | Organisations already standardised on Kong for APIs |
| Cloudflare AI Gateway | Hosted, edge | Managed | Teams on Cloudflare wanting caching, analytics and rate limits with minimal setup |
What actually differs
Overhead at load
Every gateway adds latency. The question is how much, and how it behaves as requests per second climb. Go-based gateways such as Bifrost are designed around microsecond-scale added latency at thousands of requests per second on a single instance. Python-based gateways such as LiteLLM are easier to extend and have historically carried more overhead per request, which matters at high volume and less at low volume. Hosted gateways add a network hop whose cost depends on where your application and the provider are; edge-based gateways minimise it if your traffic is already on their network.
If you serve a chat product at low volume, none of this matters. If you serve an agent platform making millions of tool-calling requests a day, it dominates.
Failover semantics
All five retry and fall back. The differences are in what they distinguish:
- Dead key versus transient error. A revoked credential should be rotated out immediately; a 5xx from the provider should be retried with backoff. Gateways that treat both the same either hammer a dead key or give up too early on a live one.
- Provider-level versus model-level fallback. Falling back from one provider’s model to an equivalent model at another provider requires a mapping the gateway has to maintain.
- Load balancing across keys and regions. Useful for staying under per-key rate limits and for latency.
Bifrost and LiteLLM expose the most configuration here; Portkey’s fallback and load-balancing “configs” are expressive and versioned; Kong and Cloudflare cover the common cases with less granularity.
Governance and policy
Governance means: who can call what, with what budget, and what gets logged. The models differ:
- Virtual keys and budgets per team are table stakes and all five offer some form.
- Guardrails (PII redaction, content filters, prompt-injection checks) are strongest in the hosted control planes and in gateways with plugin systems.
- Policy as code. Kong’s declarative configuration integrates with existing API governance; Bifrost and LiteLLM configure through files or a UI; Portkey and Cloudflare configure through their dashboards and APIs.
Deployment and data residency
This is the constraint that most often decides the choice on its own. A hosted-only gateway cannot serve an air-gapped or VPC-isolated deployment, and a hosted gateway means prompts and completions transit a third party. Self-hosted gateways keep everything inside your boundary at the cost of running them. Kong inherits whatever deployment model your Kong installation has.
Figure 2: One constraint usually settles the choice before any feature comparison matters.
Recommendations
- You have data-residency or air-gap requirements: self-host. Choose Bifrost if throughput and latency matter and you want minimal operational surface; choose LiteLLM if your team is Python-first and you value its very wide provider list.
- You want the least operations and strong observability: Portkey, or Cloudflare AI Gateway if you are already on Cloudflare and your needs are caching, analytics and limits rather than deep policy.
- You already run Kong: Kong AI Gateway extends the platform you have, and consistency across your API estate usually outweighs feature-by-feature comparisons.
Whatever you pick, put an evaluation harness behind it. A gateway makes switching models trivial, and re-benchmarking quarterly is where the savings come from.
GenAI Brief has no commercial relationship with any of the vendors above. Feature claims are drawn from public documentation.
Frequently asked questions
What is an LLM gateway?
A service that sits between applications and model providers, exposing one API for many models. It stores provider credentials, applies budgets and rate limits per caller, retries and reroutes failed calls, and logs what each request cost.
Do I need a gateway if I only use one provider?
Usually not at first. The moment a second provider, a second team or a compliance requirement arrives, the per-application logic for keys, retries and spend becomes duplicated, and a gateway is the way to collapse it.
