Bifrost's MCP gateway upgrades target tool bloat, per-user auth and REST APIs

Maxim AI’s open-source gateway Bifrost shipped three changes to its Model Context Protocol (MCP) layer in v2.3.1 on 19 October 2026. An openapi connection type turns an OpenAPI or Swagger document into an MCP server, one tool per operation. Gateway tool search lets models without a native tool search defer most of their tools. And the Starlark sandbox behind Bifrost’s code mode gained json and math modules, structured error values and Python-style tracebacks.

The release is the newest in a run that began with v2.0.0 on 26 August. Per-user OAuth and token exchange for upstream MCP servers arrived then, along with an MCP catalogue that can load from disk on air-gapped networks. Virtual MCPs followed in v2.1.0 on 8 September. v2.3.0 on 13 October added provider-injected tools, forwarding of the instructions MCP servers publish, per-execution budgets for code mode and a hardened OAuth server.

Taken one at a time, these are configuration fields. Taken together, they show where the MCP problem has moved. Connecting one server to one agent was solved in 2025. What breaks a platform team in 2026 is the forty-first server: hundreds of tool definitions eating the context window on every turn, credentials that must belong to the person asking rather than to a shared service account, and an internal estate of REST APIs that nobody has wrapped in MCP. Bifrost’s releases since August are aimed at each of those. This piece walks through what shipped, how it works according to the documentation and changelogs, and where the remaining limits sit.

What shipped, release by release

The table lists the MCP features in order of release. All dates and field names come from Bifrost’s published changelogs.

The order matters less than the grouping. Three of these features (tool search, code mode limits and scripting, Virtual MCPs) are about how many tool definitions a model has to read. Two (per-user OAuth with token exchange, and OAuth hardening) are about whose credentials reach the upstream server. One (OpenAPI servers) is about where tools come from in the first place. The rest are operational plumbing that a regulated or offline deployment needs before it can run any of the others.

The problem moved from connecting servers to carrying them

MCP’s original pitch was simple: write a server once and every compliant client can call it. That part works. The cost shows up later, because the classic integration pattern puts every tool definition from every connected server into the model’s context on every turn.

Anthropic quantified this in November 2025. In its post introducing advanced tool use, the company described a five-server setup (GitHub with 35 tools, Slack with 11, Sentry, Grafana and Splunk) as “58 tools consuming approximately 55K tokens before the conversation even starts.” It added that Jira alone uses about 17,000 tokens, and that it had seen tool definitions consume 134,000 tokens inside Anthropic before optimisation. The post also named the second-order failure: with large libraries, “the most common failures are wrong tool selection and incorrect parameters, especially when tools have similar names.”

A companion post, Code execution with MCP, described the other half of the cost. Intermediate tool results pass through the model, so a two-hour meeting transcript fetched from one system and written to another flows through the context twice. Presenting tools as code that the model reads on demand, Anthropic wrote, cut one workflow “from 150,000 tokens to 2,000 tokens”, a saving of 98.7%. Cloudflare reached the same conclusion independently and gave the pattern its common name in its Code Mode post, which converted MCP tools into a TypeScript API and asked the model to write code against it.

The protocol itself has started to acknowledge the cost. The 2026-07-28 revision of the MCP specification says servers should return tools/list in a deterministic order “to enable client-side caching and improve LLM prompt cache hit rates”, and adds required ttlMs and cacheScope hints on list results. Bifrost made its own MCP tool ordering deterministic for prompt-cache stability in v2.0.0, and in v2.2.0 kept tool schema property order stable across syncs so that reordering alone no longer invalidates a provider’s prompt cache.

The specification does not, and probably should not, decide how many tools a model sees. That decision sits with whoever assembles the request, which for a growing number of teams is a gateway. For background on the pattern, our explainer on MCP gateways covers why teams put one between agents and servers in the first place.

A worked example: 40 servers, 600 tools

Consider a mid-sized engineering organisation that has standardised on MCP. It runs 40 servers: the obvious SaaS ones (code hosting, chat, ticketing, incident management, observability, documents), a dozen internal services wrapped by platform engineers, and a long tail of team-specific utilities. Between them they expose 600 tools.

Using Anthropic’s published figure as a yardstick, 58 tools at roughly 55,000 tokens works out to a little under 1,000 tokens per tool definition. At that rate 600 tools would need well over half a million tokens of definitions. That is our extrapolation, not a measurement, and real tool schemas vary widely, but the order of magnitude is the point: the full catalogue does not fit in most context windows, and where it fits it is paid for on every turn.

Bifrost’s own code mode benchmark gives a closer reference for a deployment of this size. In its third round, with 508 tools across 16 servers, classic MCP consumed 75.1 million input tokens over 65 queries and code mode consumed 5.4 million, a 92.8% reduction, with estimated cost falling from $377.00 to $29.00 and pass rates of 100% in both modes. The documentation puts the per-query figure at roughly 1.15 million input tokens for classic MCP against 83,000 for code mode. The savings grow with tool count: the 96-tool round saved 58.2% and the 251-tool round 84.5%.

What the organisation actually needs, though, is not one technique but a policy per server:

• Heavy servers with many tools and large results (document stores, databases, observability) go into code mode, where the model reads stubs only for the servers it opens and intermediate data stays in the sandbox.
• Broad but shallow catalogues, such as a ticketing system with 80 small tools, are deferred behind tool search so the model loads three or four definitions when it needs them.
• Team-specific bundles are published as Virtual MCPs, so the support team’s agent sees ticketing plus docs search and the data team’s agent sees the warehouse and the BI tool, each through its own endpoint and key.
• Internal REST services that never got an MCP wrapper are generated from their OpenAPI documents rather than hand-built.
• Per-user systems, where the upstream must see the employee and not a robot account, use per-user OAuth or token exchange.

Every item on that list maps to a Bifrost release from the last three months. The rest of this piece takes them in turn.

Gateway tool search brings deferred loading to every model

Tool search is the simplest of the context fixes to explain. The client sends every tool definition but marks most of them defer_loading: true. The model initially sees only a search tool and any non-deferred tools; when it needs a capability it searches, and the matching definitions are expanded into context. Anthropic’s tool search documentation describes two server-side variants, regex and BM25, a default of five results per search and a ceiling of 10,000 deferred tools per request. It recommends the feature once definitions pass 10,000 tokens or a deployment aggregates more than 200 tools across MCP servers. In Anthropic’s internal MCP evaluations, Opus 4 went from 49% to 74% accuracy with tool search on, and Opus 4.5 from 79.5% to 88.1%.

The catch is that tool search is a model-provider feature. A request that defers tools only works if the model on the other end knows what defer_loading means. For a gateway routing the same agent traffic to several providers, that produces an awkward choice: send all 600 tools to a model without tool search, or reject the request.

Bifrost’s answer arrived in two steps. In v2.2.0 it routed Anthropic tool search requests on Bedrock Claude models to AWS’s InvokeModel API, which the changelog describes as “the only Bedrock API AWS allows it on”, and carried the resulting server_tool_use and tool_search_tool_result blocks back intact on streaming and non-streaming paths. In v2.3.1 it removed the dependency on the provider entirely. The v2.3.1 release notes say requests that defer tools on “a model with no tool search of its own (Gemini, older GPT, Bedrock non-Claude) no longer get every tool loaded up front or a rejection.” Instead the deferred tools “sit behind a bifrost_tool_search function that Bifrost resolves in the gateway (select: or BM25, at most 3 model calls per request)”, on both streaming and non-streaming Responses requests.

Mechanically, the model sees an ordinary function. It can ask for tools by exact name with a select: query, or describe what it needs and let BM25 rank the deferred catalogue by relevance. Bifrost returns the matching definitions, and the model calls them as if they had been loaded all along. The cap of three search rounds per request stops a confused model from paging through the whole catalogue one query at a time.

The same release fixed the native paths so the gateway does not get in their way. Deferred tools on OpenAI gpt-5.4 and later, including through Azure and Bedrock, now get the tool_search that OpenAI requires instead of a 400 error, and Claude’s server-side tool search blocks replay unchanged. The practical effect is that an agent can declare its tools once with defer_loading and keep that declaration as it fails over between providers, which is the kind of portability teams put a gateway in place to get.

Figure 1: Deferred loading now behaves the same whichever model the request lands on; only where the search runs changes.

Code mode gets budgets and a better scripting surface

Code mode is the heavier tool. Instead of exposing a server’s tools directly, Bifrost replaces them with four meta-tools: listToolFiles to discover servers, readToolFile to load compact Python-style .pyi stubs, getToolDocs for full documentation of one tool, and executeToolCode to run a script. The script is written in Starlark, a Python dialect designed for embedding, and runs in the gateway with the original request’s authorisation and tool filters. According to the code mode documentation, “the model receives only what the script prints and the result value, never the raw tool responses.”

That design has an obvious operational question attached: what stops a generated script from looping forever, allocating gigabytes or calling a destructive tool two hundred times? v2.3.0 answered it. Code mode now “runs in-process on a pinned Starlark runtime with per-execution budgets instead of child processes,” the v2.3.0 changelog says. The defaults are 64 KiB of source, one million computation steps, 64 MiB of estimated allocation, 64 KiB of print output, 64 nested tool calls, 1 MiB per value crossing the tool boundary and a nesting depth of 64. Each is configurable under mcp.tool_manager_config.code_mode_limits, reloads without a restart, and returns a tool error when exceeded. Because the budget applies per execution, the documentation advises sizing max_memory_bytes against the gateway’s memory and the number of code mode requests expected to run at once.

The same release scoped code mode discovery to the caller. listToolFiles, readToolFile and getToolDocs now show only the tools the request’s virtual key may execute, so a script cannot discover, let alone call, a tool its key does not grant. And nested tool results are logged only after output guardrails run, so text that a guardrail blocks or redacts no longer reaches the model through the execution log. Teams that already run AI guardrails on prompts and responses at the gateway get that coverage inside code mode scripts too.

v2.3.1’s scripting changes are smaller but matter for reliability. The sandbox now predeclares json and math. A failed nested call reaches the script as an error value the script can test (hasattr(r, "error")) rather than a plain string it has to parse, so a script can fall back to a search when a lookup fails or call fail() to abort the whole run. Runtime errors carry a Python-style traceback, and the error hints match what the interpreter actually emits. Models write better recovery code when the error they see looks like one they have seen in training.

Agent-mode auto-execution has a two-stage check that is worth understanding before turning it on. Before running, Bifrost scans the script’s source for server.tool() calls and runs it without asking only if every call it can see is on the server’s tools_to_auto_execute list. The documentation calls that scan “best-effort”. The authoritative check happens at each tool call while the script runs, so a tool reached through indirection (getattr(server, name)(...)) is still refused if it is not on the list.

Figure 2: Code mode moves the tool catalogue and the intermediate data out of the context window and into a budgeted sandbox in the gateway.

Virtual MCPs replace the whole-catalogue key

Tool search and code mode reduce what a model reads. Virtual MCPs reduce what a caller is allowed to see at all, which is the more durable fix for the 40-server organisation.

Introduced in v2.1.0 as a replacement for MCP tool groups, a Virtual MCP is, in the documentation’s words, “a named bundle of tools drawn from one or more of your MCP servers, exposed as its own MCP endpoint at /mcp/<slug>.” An administrator picks servers, chooses all tools (["*"]) or a named subset from each, gives the bundle a name, and attaches it to virtual keys. The slug is derived from the name, unique across all endpoints and immutable after creation, so client configurations do not break when someone renames the bundle. A disabled Virtual MCP stops being served without losing its assignments. The feature is part of open-source Bifrost and requires governance to be enabled.

The documentation draws one distinction that teams need to get right. Through the MCP gateway, the slug narrows the connection: tools/list on /mcp/support-tools returns only that bundle’s tools, even if the same key holds other grants. Through the LLM gateway (/v1/chat/completions), there is no slug, and every Virtual MCP attached to the key is folded into one tool union. Request headers such as x-bf-mcp-include-clients and x-bf-mcp-include-tools can narrow that union per request but, per the filtering documentation, never widen it, and a key with no MCP configuration gets no tools by default.

v2.1.0 also made whitelisting explicit everywhere: ["*"] grants all tools including future ones, [] grants none, and a named list grants exactly those. v2.2.0 let access profiles and governance projects reference Virtual MCPs by name, so the same config.json works across staging and production. Underneath, every request now settles its identity onto one resolved access grant that governance checks, charges and filters with, so the tools a caller sees and the limits it is billed against come from the same decision.

For the worked example, this is where 600 tools become something manageable. The support agent’s key carries a 20-tool bundle. The data team’s carries 35. Nobody’s request starts with the full catalogue, and tool search or code mode handles whatever breadth remains inside each bundle.

Per-user identity: OAuth, token exchange and the audience rule

The second hard problem is identity. A shared admin token is fine for a weather API. It is not fine for a code host, a document store or an HR system, where the upstream must enforce what this employee can see. The MCP specification is strict on the point. Its security best practices call “token passthrough”, where a server accepts a client’s token without checking it was issued to that server and forwards it downstream, an anti-pattern. The authorisation section requires clients to send an RFC 8707 resource parameter naming the MCP server and requires servers to accept only tokens issued for them as the audience.

A gateway sitting between thousands of users and dozens of servers has to satisfy that rule on every hop. Bifrost’s v2.0.0 offered two ways to do it.

Per-user OAuth (auth_type: "per_user_oauth") has each end user authorise the upstream service once under their own account. Bifrost stores one token per identity and MCP client and resolves it lazily, on the first tool call that needs it. The per-user OAuth documentation positions it for third-party SaaS (Notion, GitHub, Sentry) that runs its own OAuth. Setup needs one admin test login so Bifrost can verify the configuration and discover tools, and supports RFC 8414 discovery and RFC 7591 dynamic client registration so most fields can be left blank. v2.0.0 added oauth_config.resource for RFC 8707.

Token exchange (auth_type: "token_exchange") stores nothing per user. On every tool call, Bifrost takes the caller’s own identity-provider token and swaps it, via RFC 8693 or the equivalent on-behalf-of grant, for a short-lived token scoped to that MCP server’s audience. The token exchange documentation calls it “strictly delegated: there is no shared service-account fallback.” It suits first-party and internal servers that trust the organisation’s identity provider. For Microsoft Entra ID, which uses the jwt-bearer grant with requested_token_use=on_behalf_of described in Microsoft’s on-behalf-of flow documentation, v2.0.0’s use_idp_credentials lets the exchange reuse the SSO login application’s credentials, which the documentation says Entra requires. Okta, Keycloak, Auth0 and generic OIDC providers use RFC 8693 directly.

v2.3.0 tightened the OAuth server Bifrost runs for inbound /mcp clients. Remote callback URIs are accepted only when listed in allowed_redirect_uris (loopback addresses and Cursor’s callback are always allowed), existing registrations are rechecked at authorisation and token exchange, and the consent page shows the exact callback. Rotating a virtual key now revokes the MCP OAuth refresh tokens minted with its old value. v2.3.1 added a bare "*" wildcard for redirect URIs, flagged by a startup warning and a dashboard alert. These are the controls that keep a gateway from becoming the confused deputy the specification warns about. Our guide to MCP security and authentication covers the threat model in more depth.

OpenAPI documents become MCP servers

The third problem is supply. Most of the APIs an agent should use already exist and already have an OpenAPI document. Far fewer have an MCP server, and writing one for each internal service is the kind of work that sits on a backlog for quarters.

v2.3.1’s openapi connection type removes that step. The OpenAPI Servers documentation describes the mapping. Each operation’s operationId, the field the OpenAPI specification defines as a unique string identifying the operation, becomes the tool name, sanitised to letters, digits and underscores; operations without one are named from method and path. The summary and description become the tool description, followed by METHOD /path. Path, query, header and cookie parameters become top-level properties of the tool’s input schema, each encoded in its declared style, and JSON request bodies with named properties are flattened alongside them. HTTP methods map to MCP annotations: GET and HEAD are read-only and idempotent, PUT idempotent, DELETE destructive, POST and PATCH destructive and non-idempotent. Schemas are normalised to the dialect model providers accept, so a 3.1 type: [..., "null"] becomes nullable and objects in an allOf are merged.

The document can be supplied inline (spec), as a file next to config.json (spec_file) or by URL (spec_url), up to 5 MiB, in Swagger 2.0 or OpenAPI 3.0, 3.1 or 3.2, as JSON or YAML. A dry-run endpoint (POST /api/mcp/openapi/preview) lists the tools a document yields, the operations it cannot express and why, the security schemes and the resolved base URL before anything is created. In the dashboard, the New MCP Server sheet offers the same parsed view, with checkboxes to untick operations, and a Replace spec action that matches operations by method and path so auto-execute settings and tool prices follow an operation even when its name changes.

Credentials come from the gateway, never the model. Header parameters named Authorization, Accept or Content-Type are ignored, as OpenAPI specifies. For apiKey and HTTP bearer or basic schemes, an administrator stores a value (which accepts env. and vault. references) or switches the scheme to per-user, in which case each caller supplies the header and Bifrost forwards it unchanged. A redirect to another host, or from HTTPS to plain HTTP, is not followed when the call carries credentials.

The result is a regular MCP client, which is the useful part. Everything else in this article applies to it: tool filtering, auto-execute lists, per-tool pricing, virtual-key access, code mode, the hosted /mcp/<slug> endpoint and Virtual MCP bundling. An internal billing API with 140 operations can be imported, cut to the twelve read-only operations the finance agent needs, put behind code mode, bundled with the document store into a finance Virtual MCP and attached to one key. That is a configuration change, not a project.

Figure 3: The OpenAPI document already carries most of what an MCP tool needs; the gateway supplies the credentials and the policy.

Server instructions, injected tools and the offline catalogue

Three smaller features fill in gaps that show up once the big ones are in place.

Server instructions. MCP servers can publish natural-language guidance on how their tools should be used. Under earlier protocol revisions it arrived in the initialize response; the 2026-07-28 revision moves it to the result of a new server/discover call, where the schema describes it as guidance clients can include “in a system prompt”. The Bifrost overview notes that servers such as GitHub’s, Grafana’s and Outline’s use it to carry workflow rules. Since v2.3.0, Bifrost captures those instructions, forwards them on its own /mcp endpoint and injects them into MCP-bearing chat and Responses requests as <mcp_server name="..."> blocks, scoped to the servers the request can see. Text containing the tag is neutralised, so one server cannot forge a boundary and attribute its guidance to another. Each server is capped at 4,096 bytes and the total at 16,384 by default. Code mode servers are deliberately excluded from that aggregate; their guidance appears in the stub file the model reads before calling them, so it is paid for once per server opened rather than on every request.

Provider-injected tools. v2.3.0 lets a provider carry injected_tools, entries of the form {name, mcp_client_name, tool_name}. Bifrost adds each MCP tool to every chat and Responses request routed to that provider, runs it server side when the model calls it, and returns only the final answer with usage summed across turns. An entry named web_search replaces any native web search the client sends. In practice, an organisation can make one approved search or retrieval tool the only one any model reaches through a given provider, with recursion capped by mcp_agent_depth.

The air-gapped catalogue. v2.0.0 added two settings for disconnected networks: mcp_library_sync_interval: 0 disables catalogue sync, and file:// URLs load the MCP server library from disk. Combined with the v2.3.0 private-network allowlist for OAuth-protected MCP servers on internal addresses, it lets the whole stack run without reaching the public internet.

Where it sits against the platform vendors

Most of the ideas here are not unique to Bifrost, and the release notes do not claim they are. Anthropic ships tool search and programmatic tool calling as features of its own API. Cloudflare popularised code mode in its Agents SDK. OpenAI’s newer models accept deferred tools through their own tool_search. Each of those implementations works for the vendor’s own models or runtime.

What distinguishes a gateway implementation is that it is model-neutral and sits where policy already lives. Anthropic’s tool search helps Claude; Bifrost’s gateway tool search applies the same defer_loading declaration to Gemini or a non-Claude Bedrock model on the same request path, and passes Claude’s native search through untouched. Cloudflare’s code mode runs inside a Cloudflare Worker; Bifrost’s runs inside the gateway under the virtual key’s tool grant, with the output guardrails and audit logging the rest of the traffic already gets. OpenAPI import exists in several API-management products, but the output there is usually an MCP server that still needs separate auth, filtering and cost tracking. Here it is a client like any other.

Bifrost’s MCP gateway also sits on the same binary as its LLM gateway, so the tool union a model sees, the provider it lands on and the budget it is charged against are decided in one place. The vendor reports 11 µs of added overhead at 5,000 requests per second for the gateway path. Our ranking of MCP gateways for 2026 compares the wider field on these criteria.

The limits sit in the protocol and the category

None of this makes the underlying problems disappear, and teams adopting these features should plan around several limits that belong to MCP and to the techniques themselves.

The protocol is mid-transition. The 2026-07-28 specification removes the initialize handshake and protocol-level sessions, moves server instructions to server/discover and replaces the HTTP GET stream with subscriptions/listen. Every gateway, client and server will spend the coming months running dual-era code, and servers will migrate at different speeds, so the same field can arrive by two routes for a while. Bifrost’s v2.3.0 session-recovery work (reconnects on “session not found” errors, immediate tool refresh) is useful precisely because the ecosystem is in this state.

Search adds a step, and code adds a sandbox. Anthropic’s own guidance says tool search “adds a search step before tool invocation” and pays off when context savings and accuracy gains outweigh the latency. Code mode trades fewer model turns for a script the model has to write correctly. Bifrost’s benchmark shows pass rates holding at 100%, but teams with their own tools should measure on their own tasks before switching every server over.

OpenAPI documents are only as good as their authors. A spec with missing operationIds, thin descriptions or every operation as POST produces tools with poor names and wrong safety annotations. Anthropic’s guidance that tool search “matches against names and descriptions” applies doubly to generated tools. Some API shapes do not map onto tool arguments at all: file uploads, binary and streaming request bodies, and APIs that accept only OAuth flows a gateway cannot run on a model’s behalf. And a large API surface is a large tool surface; a 400-operation document imported wholesale recreates the context problem it was meant to solve, so selection matters as much as generation.

Delegated identity depends on the identity provider. Token exchange works only when the upstream MCP server trusts the organisation’s issuer, which rules out most third-party SaaS. Okta’s documented best practice is a dedicated custom authorisation server per protected resource, and Entra’s on-behalf-of grant has its own audience rules. In Bifrost specifically, token exchange requires an enabled SCIM identity provider, which comes with the enterprise tier, as do the access-profile, project and cluster scoping layers on top of open-source Virtual MCPs.

What to watch next

Three questions will decide how much of this becomes standard practice rather than gateway-specific configuration.

First, whether the MCP specification grows a native notion of deferred or searchable tools. The 2026-07-28 revision added caching hints and deterministic ordering, which help prompt caches, but leaves the question of which tools a model should see to the client. If a future revision adds a discovery-time hint for tool relevance, gateway tool search becomes an implementation of a standard rather than a translation layer.

Second, whether OpenAPI-to-MCP generation produces tools models use well. The mapping is mechanical; the quality of the descriptions is not. Expect gateways to add description overrides, tool-use examples and per-operation instruction text, and expect platform teams to start treating their OpenAPI documents as prompts.

Third, how fast the server side catches up on identity. Per-user OAuth and token exchange solve the gateway’s half of the audience rule. The other half is MCP servers that validate audiences properly, publish protected-resource metadata and support token exchange from enterprise issuers. Until they do, the gateway is doing work the specification assigns to every server.

For the organisation with 40 servers and 600 tools, the direction is already clear. The catalogue will keep growing, so the number of tools any single request carries has to stop growing with it. Bifrost’s releases since August are a bet that the gateway is the right place to enforce that, and for teams already routing model traffic through one, it is hard to argue with the location.