Model gateway

Introduction

Use the model gateway when you run a shared server that holds the provider keys and want users’ local backends, such as a desktop app, to call models without ever holding a key. You do not need it if every machine has its own keys.

The desktop calls the gateway, the gateway calls the provider, and each call is recorded in the gateway’s usage store. The user’s machine never sees a provider key.

The gateway is opt-in. Set MODEL_GATEWAY_ROLES (comma-separated; callers need one of them) or every route answers 404. A caller without a role gets 403.

Lending a profile on the server

Mark a profile shared: true in its file.

.pilotoo/profiles/team-model.yaml

name: Team model
model: anthropic:my-model-id
shared: true

.env

MODEL_GATEWAY_ROLES=member

A profile is served only when its provider is of kind openai, openai-compatible, deepseek or anthropic and has its key set. claude-subscription and anthropic-bedrock are never lent.

Using it from the local backend

The local backend declares a provider of kind pilotoo-gateway with the gateway’s root URL, and a profile whose model is <provider>:<shared id>.

providers/shared.yaml

kind: pilotoo-gateway
baseUrl: https://gateway.example.com

profiles/shared-team.yaml

name: Team model
model: shared:team-model

The host passes modelGateway: { token(), headers?() } (PilotooModule.forRoot, or host of the local backend):

const server = await startLocalServer({
  // ...
  host: {
    modelGateway: {
      token: () => session.accessToken,
      headers: () => ({ 'X-Tenant-Id': '7' }),
    },
  },
});

The token and the headers are read on every request, so a renewed token is used. headers() adds headers to every gateway request, the catalog and the model calls, for example to say which tenant a desktop acts for. authorization, x-api-key, content-type, host and the other credential and framing headers are dropped.

Caution

The gateway URL must be https or loopback. The token and tenant header are sent only to the origin decided at start, so a profile that points elsewhere never receives them.

The registry loads the catalog at boot and again when a run names a shared id it lacks: that run fails with a ProviderNotConfiguredError and the next works. An unreachable gateway or an unknown shared id is a boot warning, never a boot problem. The shared profile’s params apply under the local profile’s, and the shared kind judges the result. See Models and the local backend.

Setting monthly caps

Variable	Meaning
MODEL_GATEWAY_ROLES	Comma-separated roles that may use the gateway. Unset or empty means off, every route 404.
MODEL_GATEWAY_MONTHLY_TOKENS_PER_USER	Positive integer: tokens one user may use through the gateway per calendar month (UTC). Unset or empty means no cap.
MODEL_GATEWAY_MONTHLY_TOKENS_PER_TENANT	The same for a whole tenant.
MODEL_GATEWAY_MAX_CONCURRENT_PER_USER	Positive integer, default 4: the calls one user may have in flight while a cap is set.

• The count is the total tokens of the calls recorded under agent model-gateway in the current UTC month, read from the usage store. The server’s own agents do not count.
• A call is checked after the caller is authorized and the profile is found, before the body is read.
• Past a cap the gateway answers 429 with its own message, naming the cap (per user or per tenant) and the day it resets (the first of next month, UTC), plus retry-after in seconds until then, and never calls the provider.
• The cap 429 carries x-should-retry: false, which both client SDKs honour, so neither sleeps out a retry-after of weeks.
• A value that is not a positive integer stops the boot, naming the variable. So does a cap with no usage store (usageStore: null): a cap that cannot be enforced must not look enforced.
• @pilotoo/local-server clears all three cap variables, as it does MODEL_GATEWAY_ROLES.
• The agent id model-gateway is reserved. The agent editor, the config tools, and the boot of system agents and templates refuse it, so no agent of a tenant can spend a cap.

The cap is soft

The overshoot is bounded, not zero.

• A call is counted once it ends, so a user can end up past their cap by the tokens of the calls that were in flight when the last one was checked: at most MODEL_GATEWAY_MAX_CONCURRENT_PER_USER of them.
• A call past that many in flight is refused with 429 (rate_limit_error, retry-after: 5, no x-should-retry, so a client retries it) before it reaches the provider.
• A tenant cap can be passed by that many calls for each of its users in flight at once. The limit is kept per process, as the totals are.
• Each total is cached for at most 60 s (at most 10,000 kept), concurrent reads of one total share one query, and every call the gateway records is added to the cached totals at once.
• A store that fails to answer fails the call (500) rather than lifting the cap.

Showing the quota

When a cap is set, GET /model-gateway/profiles also answers quota: { user?: { used, limit }, tenant?: { used, limit }, resetsAt } (resetsAt ISO 8601, UTC; only the caps that are set; totals can be up to 60 s old). Without a cap the key is absent. A usage store that fails or takes more than 2 s costs the listing its quota, not its profiles.

A pilotoo-gateway provider keeps the quota with its catalog and stops showing it once its resetsAt has passed. ProviderInfo.quota and the editor’s profile summaries carry it, and ProfilesList from @pilotoo/ui/settings shows it in one line, with its numbers and reset date written for the locale prop when given.

Routes

Route	Answers
GET /model-gateway/profiles	{ profiles }: the shared: true profiles this gateway can serve, as { id, name, kind, modelId, params, acceptsImages, salvageLeakedToolCalls, ownEndpoint? }. Never a key, a variable name or an upstream address.
POST /model-gateway/profiles/:id/chat/completions	For openai, openai-compatible and deepseek profiles: the provider’s answer, status, content type and body unchanged (a stream stays a stream), with its retry-after and retry-after-ms. A provider 401 or 403 (the host’s key was refused) becomes a 502 with the gateway’s own message. 404 for any other kind, an unknown id or a profile that is not shared.
POST /model-gateway/profiles/:id/v1/messages	The same for anthropic profiles.
GET /model-gateway/skills	{ skills, omitted }: the global layer and the caller’s tenant layer (tenant wins a name), as { name, description, content, files }, for a sidecar to keep read-only. A skill that would take the answer past 4 Mi characters is named in omitted.

What the gateway does to a call

• Overwrites model with the profile’s, calls the provider’s own address (baseUrl, else the kind’s default) with the real key, and forwards no header of the caller’s except content-type and accept (and anthropic-version and anthropic-beta for Anthropic).
• Reads the token usage off the answer and records one call under agent model-gateway, after the stream has ended and without awaiting the store. A call that reported no usage is recorded with zero tokens. For OpenAI-family streams the gateway sets stream_options.include_usage on every stream.
• Bounds the body by MODEL_GATEWAY_BODY_LIMIT_BYTES (32 MiB), not by JSON_BODY_LIMIT_BYTES, which every other route keeps. Past the limit is 413; a body that is not a JSON object is 400.
• Answers 415, before reading the body, to a Content-Encoding other than absent or identity, to a Content-Type that is not application/json (or +json), and to a charset other than UTF-8.

Caution

Call configureBodyLimit(app) in your bootstrap, as the demo API does. A host that does not still serves the gateway, but under Nest’s default JSON parser (about 100 KB), so a larger body is refused with 413.

Errors your client SDK can read

The gateway’s own refusals on the two passthrough routes are answered in the error body of the API the route imitates, with status and headers (retry-after) kept, so a client SDK shows the message. That covers the cap 429 and the in-flight 429, 400, 401, 403, 404, 413, 415 and 502, and the 503 of an identity service that timed out.

Route	Error body
chat/completions	{ error: { message, type, code } }. The cap is type: "insufficient_quota", code: "monthly_token_limit". The rest are authentication_error, permission_error, rate_limit_error, server_error or invalid_request_error, with code: null.
v1/messages	{ type: "error", error: { type, message } }. rate_limit_error for both 429s, and authentication_error, permission_error, not_found_error, request_too_large, api_error or invalid_request_error.

A provider’s own refusal is still relayed unchanged. Whatever else throws (a usage store that fails, or does not answer within 5 s, while a call is checked) is a 500 that names nothing of its cause.

Next steps

• Usage for the store the caps read.
• Hosting it from another app to run the local backend that consumes the gateway.
• Models for providers and profiles.