Skip to content

Running Colony behind an LLM gateway

Many platform and InfoSec teams require every LLM call to go through an approved gateway (TrueFoundry, LiteLLM, Portkey, Kong AI Gateway, Azure API Management). The gateway owns credentials, budgets, rate limits and the audit log. Colony is a workload inside that platform: it surfaces the gateway’s decisions, does not fight them, and labels its traffic so the platform team can see what it spends.

This page covers claude.provider: gateway for the Claude Code engine. For the config keys themselves, see ClaudeGatewayConfig.

For every Claude Code invocation under provider: gateway, Colony sets:

Environment variableSource
ANTHROPIC_BASE_URLclaude.gateway.base_url
ANTHROPIC_AUTH_TOKENprocess.env[claude.gateway.auth_token_env] (default name ANTHROPIC_AUTH_TOKEN). Claude Code sends it as Authorization: Bearer; there is no x-api-key option on the gateway path
ANTHROPIC_CUSTOM_HEADERSclaude.gateway.custom_headers, plus the attribution lines when configured
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS1 when disable_experimental_betas: true
ANTHROPIC_DEFAULT_{OPUS,SONNET,HAIKU}_MODELclaude.gateway.pin_models.{opus,sonnet,haiku}

Colony sets no timeout or retry variable. Set API_TIMEOUT_MS and CLAUDE_CODE_MAX_RETRIES in the container environment if the defaults do not suit your gateway.

Claude Code sends x-claude-code-session-id on every request. Colony persists the same id as execution_telemetry.session_id, so a gateway log line can be joined to Colony’s per-invocation telemetry.

Colony always passes --model <id> using claude.models.*, claude.scaling.*.model and model_routing.routes. Claude Code documents ANTHROPIC_DEFAULT_*_MODEL as the id the opus / sonnet / haiku alias resolves to, so a pin does not rewrite an explicit id such as claude-opus-5-5.

Behind a gateway that serves namespaced ids (for example claude-code/claude-opus), set the claude.models.*, scaling and routing ids to the ids your gateway serves. A config that sets only pin_models sends the unnamespaced Anthropic id to the gateway.

  • colony check and worker startup warn, listing the ids, when pin_models is set and any claude.models.*, scaling.*.model or model_routing.routes value is an unnamespaced claude-* id.
  • Colony does not rewrite explicit ids to pins, because that would break gateways that serve Anthropic ids and pin only some families.
  • The model-specific quota fallback substitutes a bare family alias. Under a gateway, Colony declines the fallback when the substitute family has no pin_models entry (and no matching model_routing route or scaling model), because the alias would reach the gateway as a Claude Code default id.
  • Cost attribution: a namespaced id whose bare segment is a known Anthropic id (for example acme/claude-opus-5-5) is priced as that exact model. Other ids need a tier token — see Model naming rules for cost attribution.

Off by default. Set claude.gateway.attribution and Colony composes a header per invocation and merges it with custom_headers:

claude:
provider: gateway
gateway:
base_url: https://llm-gateway.example.com/api/llm
attribution:
header: X-TFY-METADATA # TrueFoundry
format: json
fields: [tenant, repo, issue, station, task]
FieldValue
tenantTenant id
repoowner/repo
issueIssue number; omitted for repo-scoped tasks (for example strategize)
stationWorker tasks use the task type. Other sites use a fixed name: mayor, blocked-retrospective, conflict-resolution, semantic-conflict, health-probe, colonize, scan-conventions
taskTask id. A retry is a new task with a new id, so task distinguishes attempts
  • format: json (default) emits one header whose value is a JSON object of string values: X-TFY-METADATA: {"tenant":"acme","repo":"acme/app","issue":"42","station":"develop","task":"..."}.
  • format: headers emits one header per field, named <header>-<field>.
  • Portkey takes the same shape: header: x-portkey-metadata, format: json.
  • TrueFoundry and Portkey cap each metadata value at 128 characters. Colony sanitizes every value to printable ASCII (CR, LF and other control or non-ASCII characters are dropped) and caps it at 128 characters, so a value can never add a header line.
  • Config validation rejects an attribution header name that also appears in custom_headers (compared case-insensitively).
  • Colony has no vendor defaults: the header name and format are always yours to set.

Multi-tenant deployments: tenants[].provider: gateway is honored by workers, the sprint-master Mayor and the blocked-issue retrospective, and those calls carry attribution too.

Throttle (429). Claude Code retries temporary 429 throttles up to 10 times on its own. It skips the retries only when the response carries x-should-retry: false or a retry-after above 60. After those retries, Colony treats a 429 as quota: it does not re-spawn, and the worker enters a degraded pause (non-LLM tasks continue). Under a gateway, the pause comment and log line name the LLM gateway rather than “Anthropic API quota”.

Budget exhausted. A gateway budget block is usually also a 429, so Colony cannot tell it from a throttle without help. Tell it what your gateway’s budget block looks like with budget_exhausted_match. TrueFoundry returns HTTP 429 with "type": "RateLimitError", "error_origin_level": "rate_limit_budget", the message Budget exceeded for model … and an x-tfy-applied-rules response header:

claude:
gateway:
budget_exhausted_match:
- { status: 429, text: 'rate_limit_budget' }

A failed result whose HTTP status equals status and whose error text contains text (case-insensitive) classifies as gateway_budget:

  • no Colony-level retry and no quota model fallback;
  • the worker pauses LLM tasks with pause reason gateway_budget and the maximum probe backoff (30 minutes), then resumes when a health probe succeeds;
  • the issue comment names the gateway host and that its spend budget is exhausted. The gateway’s own message goes to the worker log and the operator alert only, never the issue comment, because it can contain user identifiers.

Without rules, the same response is treated as quota.

401 / 403. Under provider: gateway, a 401 or 403 is the gateway rejecting the request (an invalid token, or a model-permission or policy decision), not an expired OAuth token. Colony takes the fatal path with a health probe and logs and alerts LLM gateway rejected the request (HTTP <status>). It does not enter the token-expiry pause.

5xx and resets. 5xx responses are transient. A connection reset (ECONNRESET, socket hang up) with no API status is also transient, since proxies reset long-lived connections. ENOTFOUND and ECONNREFUSED stay fatal network errors.

Claude Code streams responses. The gateway must not buffer server-sent events, and must forward ping events so idle connections are not cut. A buffering proxy shows up as long stalls followed by connection resets.

Set API_TIMEOUT_MS in the container environment to match the gateway’s own request timeout. Colony does not set it.

Some Claude Code features call api.anthropic.com directly regardless of ANTHROPIC_BASE_URL: the fast-mode and WebFetch domain checks. Set CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 in the container environment to turn off nonessential traffic. If egress to api.anthropic.com is blocked, allow these calls to fail or disable the features.

Colony’s gateway behavior is covered by a network-free conformance suite (packages/core/src/__tests__/gateway-conformance.test.ts) that runs a loopback stub gateway. Verification against a live gateway is a manual operator step: run colony check. Its gateway probe sends your custom_headers and reports a matched budget_exhausted_match rule as “gateway budget exhausted”.

The Codex engine (engine: codex) is not routed through the gateway; Codex gateway support is tracked as a follow-up.