Running Colony behind an LLM gateway
Running Colony behind an LLM gateway
Section titled “Running Colony behind an LLM gateway”Many platform and InfoSec teams require every LLM call to go through an approved gateway (TrueFoundry, LiteLLM, Portkey, Kong AI Gateway, Azure API Management). The gateway owns credentials, budgets, rate limits and the audit log. Colony is a workload inside that platform: it surfaces the gateway’s decisions, does not fight them, and labels its traffic so the platform team can see what it spends.
This page covers claude.provider: gateway for the Claude Code engine. For the config keys themselves, see ClaudeGatewayConfig.
What Colony sets
Section titled “What Colony sets”For every Claude Code invocation under provider: gateway, Colony sets:
| Environment variable | Source |
|---|---|
ANTHROPIC_BASE_URL | claude.gateway.base_url |
ANTHROPIC_AUTH_TOKEN | process.env[claude.gateway.auth_token_env] (default name ANTHROPIC_AUTH_TOKEN). Claude Code sends it as Authorization: Bearer; there is no x-api-key option on the gateway path |
ANTHROPIC_CUSTOM_HEADERS | claude.gateway.custom_headers, plus the attribution lines when configured |
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS | 1 when disable_experimental_betas: true |
ANTHROPIC_DEFAULT_{OPUS,SONNET,HAIKU}_MODEL | claude.gateway.pin_models.{opus,sonnet,haiku} |
Colony sets no timeout or retry variable. Set API_TIMEOUT_MS and CLAUDE_CODE_MAX_RETRIES in the container environment if the defaults do not suit your gateway.
Claude Code sends x-claude-code-session-id on every request. Colony persists the same id as execution_telemetry.session_id, so a gateway log line can be joined to Colony’s per-invocation telemetry.
Explicit model ids vs pins
Section titled “Explicit model ids vs pins”Colony always passes --model <id> using claude.models.*, claude.scaling.*.model and model_routing.routes. Claude Code documents ANTHROPIC_DEFAULT_*_MODEL as the id the opus / sonnet / haiku alias resolves to, so a pin does not rewrite an explicit id such as claude-opus-5-5.
Behind a gateway that serves namespaced ids (for example claude-code/claude-opus), set the claude.models.*, scaling and routing ids to the ids your gateway serves. A config that sets only pin_models sends the unnamespaced Anthropic id to the gateway.
colony checkand worker startup warn, listing the ids, whenpin_modelsis set and anyclaude.models.*,scaling.*.modelormodel_routing.routesvalue is an unnamespacedclaude-*id.- Colony does not rewrite explicit ids to pins, because that would break gateways that serve Anthropic ids and pin only some families.
- The model-specific quota fallback substitutes a bare family alias. Under a gateway, Colony declines the fallback when the substitute family has no
pin_modelsentry (and no matchingmodel_routingroute or scaling model), because the alias would reach the gateway as a Claude Code default id. - Cost attribution: a namespaced id whose bare segment is a known Anthropic id (for example
acme/claude-opus-5-5) is priced as that exact model. Other ids need a tier token — see Model naming rules for cost attribution.
Attribution headers
Section titled “Attribution headers”Off by default. Set claude.gateway.attribution and Colony composes a header per invocation and merges it with custom_headers:
claude: provider: gateway gateway: base_url: https://llm-gateway.example.com/api/llm attribution: header: X-TFY-METADATA # TrueFoundry format: json fields: [tenant, repo, issue, station, task]| Field | Value |
|---|---|
tenant | Tenant id |
repo | owner/repo |
issue | Issue number; omitted for repo-scoped tasks (for example strategize) |
station | Worker tasks use the task type. Other sites use a fixed name: mayor, blocked-retrospective, conflict-resolution, semantic-conflict, health-probe, colonize, scan-conventions |
task | Task id. A retry is a new task with a new id, so task distinguishes attempts |
format: json(default) emits one header whose value is a JSON object of string values:X-TFY-METADATA: {"tenant":"acme","repo":"acme/app","issue":"42","station":"develop","task":"..."}.format: headersemits one header per field, named<header>-<field>.- Portkey takes the same shape:
header: x-portkey-metadata,format: json. - TrueFoundry and Portkey cap each metadata value at 128 characters. Colony sanitizes every value to printable ASCII (CR, LF and other control or non-ASCII characters are dropped) and caps it at 128 characters, so a value can never add a header line.
- Config validation rejects an attribution header name that also appears in
custom_headers(compared case-insensitively). - Colony has no vendor defaults: the header name and format are always yours to set.
Multi-tenant deployments: tenants[].provider: gateway is honored by workers, the sprint-master Mayor and the blocked-issue retrospective, and those calls carry attribution too.
Budget, throttle and 403 behavior
Section titled “Budget, throttle and 403 behavior”Throttle (429). Claude Code retries temporary 429 throttles up to 10 times on its own. It skips the retries only when the response carries x-should-retry: false or a retry-after above 60. After those retries, Colony treats a 429 as quota: it does not re-spawn, and the worker enters a degraded pause (non-LLM tasks continue). Under a gateway, the pause comment and log line name the LLM gateway rather than “Anthropic API quota”.
Budget exhausted. A gateway budget block is usually also a 429, so Colony cannot tell it from a throttle without help. Tell it what your gateway’s budget block looks like with budget_exhausted_match. TrueFoundry returns HTTP 429 with "type": "RateLimitError", "error_origin_level": "rate_limit_budget", the message Budget exceeded for model … and an x-tfy-applied-rules response header:
claude: gateway: budget_exhausted_match: - { status: 429, text: 'rate_limit_budget' }A failed result whose HTTP status equals status and whose error text contains text (case-insensitive) classifies as gateway_budget:
- no Colony-level retry and no quota model fallback;
- the worker pauses LLM tasks with pause reason
gateway_budgetand the maximum probe backoff (30 minutes), then resumes when a health probe succeeds; - the issue comment names the gateway host and that its spend budget is exhausted. The gateway’s own message goes to the worker log and the operator alert only, never the issue comment, because it can contain user identifiers.
Without rules, the same response is treated as quota.
401 / 403. Under provider: gateway, a 401 or 403 is the gateway rejecting the request (an invalid token, or a model-permission or policy decision), not an expired OAuth token. Colony takes the fatal path with a health probe and logs and alerts LLM gateway rejected the request (HTTP <status>). It does not enter the token-expiry pause.
5xx and resets. 5xx responses are transient. A connection reset (ECONNRESET, socket hang up) with no API status is also transient, since proxies reset long-lived connections. ENOTFOUND and ECONNREFUSED stay fatal network errors.
Streaming requirements
Section titled “Streaming requirements”Claude Code streams responses. The gateway must not buffer server-sent events, and must forward ping events so idle connections are not cut. A buffering proxy shows up as long stalls followed by connection resets.
Timeouts
Section titled “Timeouts”Set API_TIMEOUT_MS in the container environment to match the gateway’s own request timeout. Colony does not set it.
Traffic that bypasses the gateway
Section titled “Traffic that bypasses the gateway”Some Claude Code features call api.anthropic.com directly regardless of ANTHROPIC_BASE_URL: the fast-mode and WebFetch domain checks. Set CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 in the container environment to turn off nonessential traffic. If egress to api.anthropic.com is blocked, allow these calls to fail or disable the features.
Verifying against a real gateway
Section titled “Verifying against a real gateway”Colony’s gateway behavior is covered by a network-free conformance suite (packages/core/src/__tests__/gateway-conformance.test.ts) that runs a loopback stub gateway. Verification against a live gateway is a manual operator step: run colony check. Its gateway probe sends your custom_headers and reports a matched budget_exhausted_match rule as “gateway budget exhausted”.
Not covered
Section titled “Not covered”The Codex engine (engine: codex) is not routed through the gateway; Codex gateway support is tracked as a follow-up.