Skip to content
Pilotoo
v0.1

0.1.0 Latest

ChangelogHow versions work

Borrow server models from a desktop app

Introduction

Section titled “Introduction”

You will lend one model profile from a shared server to local backends. The desktop app calls the server’s model gateway with its user’s token, the gateway calls the provider with the real key, and the key never leaves the server.

What you need

Section titled “What you need”

Steps

Section titled “Steps”

1. Share a profile on the server

Section titled “1. Share a profile on the server”

Mark the profile shared: true. Only profiles on a provider of kind openai, openai-compatible, deepseek or anthropic, with a key set, are served.

.pilotoo/profiles/team-model.yaml
name: Team model
model: anthropic:my-model-id
shared: true

2. Turn the gateway on for some roles

Section titled “2. Turn the gateway on for some roles”

The gateway is off until you name the roles that may use it. Unset means every route answers 404; a caller without a role gets 403.

.env
MODEL_GATEWAY_ROLES=permission:assistant.desktop

Optionally cap usage per month (UTC):

.env
MODEL_GATEWAY_MONTHLY_TOKENS_PER_USER=2000000
MODEL_GATEWAY_MONTHLY_TOKENS_PER_TENANT=50000000

Call configureBodyLimit(app) in your server’s main.ts. Without it the gateway still works but under Nest’s default JSON limit of about 100 KB, so larger requests get 413.

3. Point the local backend at the gateway

Section titled “3. Point the local backend at the gateway”

Add a provider of kind pilotoo-gateway in the local config folder, and a profile that names a shared profile as <provider>:<shared id>.

<configDir>/providers/shared.yaml
kind: pilotoo-gateway
baseUrl: https://assistant.example.com
<configDir>/profiles/default.yaml
name: Team model
model: shared:team-model

baseUrl must be https, or http for localhost, 127.0.0.1 or [::1].

4. Give the backend the user’s token

Section titled “4. Give the backend the user’s token”

The host passes modelGateway to the local backend. Both functions run on every request, so a renewed token is used.

sidecar/launcher.mjs
const server = await startLocalServer({
  // ...the options from the desktop recipe
  host: {
    tools: hostTools(env),
    modelGateway: {
      token: async () => readSignedInToken(),
      headers: async () => ({ 'X-Tenant-Id': currentTenantId() }),
    },
  },
});

headers is optional and adds headers to every gateway request. Credential and framing headers such as authorization and content-type are dropped.

How it works

Section titled “How it works”
  • At boot, and again when a run names a shared id it does not know, the local backend asks GET /model-gateway/profiles for the catalog. An unreachable gateway is a boot warning, not a failure; a run on that profile fails until it answers.
  • The catalog lists { id, name, kind, modelId, params, ... } and never a key, a variable name or an upstream address.
  • For a call, the gateway replaces model with the profile’s, adds the real key and forwards the provider’s answer unchanged, a stream included. It records the token usage under the agent model-gateway.
  • The local backend drops MODEL_GATEWAY_ROLES from its own environment: its single token must never lend the user’s keys to anything else.

Next steps

Section titled “Next steps”