Skip to content

Operate and Scale Workers

Workers claim tasks from Colony’s durable queue and run analyzer, planner, developer, reviewer, merger, and sweep executors. A Worker executes one claimed pipeline task at a time; add Workers for concurrency rather than only adding CPU to one process.

Before scaling, record:

  • online, stale, and draining Worker counts;
  • pending and claimed tasks by type and repository;
  • queue wait time and task duration;
  • CPU, memory, disk, and checkout/setup time;
  • provider and LLM rate-limit errors;
  • daily spend and concurrent-issue limits.

More Workers do not help when work is blocked on clarification, dependencies, provider checks, budget, or a misconfigured repository.

Managed Workers require no registration. Add BYO Workers when code execution must reach private dependencies, requires specific hardware, or needs separately operated capacity.

BYO Workers authenticate to Colony Cloud with a generated Worker token and continue to depend on the Cloud API, queue, configuration, and GitHub App credentials. They are not an air-gapped mode.

  1. Open Operations → Fleet (/operations/fleet), generate a token for one logical Worker group, and copy it while it is visible.
  2. Save the token in a colony.env file (COLONY_CLOUD_TOKEN=<token>) and deploy the supported Colony Worker image with --env-file colony.env, --restart unless-stopped, a unique --name and the same value as COLONY_WORKER_NAME, the Cloud URL when non-default, and an explicit repository scope when required by the image. The dashboard generates these commands for you.
  3. Wait for registration and a fresh heartbeat in Operations → Fleet.
  4. Confirm the Worker’s Role, Status, Host and Last Heartbeat in the Workers table before adding more replicas.
  5. Keep separate tokens for environments or Worker groups so one compromised token can be rotated independently.

Cloud re-mints short-lived provider credentials during registration and heartbeat. Do not inject a long-lived GitHub PAT into a Cloud BYO Worker as a replacement for the Cloud registration flow.

Scale out when eligible pending tasks wait materially longer than normal while existing Workers remain busy and healthy. Scale down when Workers are persistently idle and no recurring schedule depends on immediate capacity.

SignalInterpretationAction
Pending queue grows; Workers busyCapacity-bound.Add a Worker for the affected repository or eligible pool.
Pending queue grows; Workers idleRouting, repo scope, workflow executor, budget, or scheduling problem.Diagnose configuration; do not add capacity.
Claimed tasks have stale heartbeatsWorker crashed or lost connectivity.Inspect, then reclaim or let the stale-task policy recover.
Tasks fail during setupEnvironment, credentials, dependency service, disk, or memory problem.Fix the Worker image/configuration first.
Provider/LLM throttling risesExternal rate limit.Reduce concurrency or obtain the appropriate service capacity.

Remember that SI tracks, external PR review, planning, review loops, and sweep work use the same Worker capacity as ordinary implementation.

In Operations → Fleet, request Drain for the Worker. Draining prevents new claims and allows its current task to finish. Wait for the Worker to report drained or inactive before stopping the container.

Replace one Worker at a time and confirm another healthy Worker can serve the repository. Generate and deploy a replacement token before revoking the old token during credential rotation.

A worker’s identity comes from its name. Without COLONY_WORKER_NAME, the name follows the container hostname, and Docker gives every new container a random hostname. Recreating the container then registers a new worker, leaves the old one offline, and its claims wait for orphan recovery. The commands generated in Operations → Fleet set COLONY_WORKER_NAME for you.

To replace or upgrade a Worker:

  1. Drain it, as described above.
  2. Run docker stop <name>, then docker rm <name>. docker run --name fails while a stopped container still holds the name.
  3. Run the same command again with the new image tag and the same COLONY_WORKER_NAME.

Keep names unique per token (for single-repo Workers, per repository). Two live containers with the same name share one worker row and overwrite each other, so change the name when you copy a command to a second host.

Only one sprint master runs per org. A different sprint master is refused with 409 SPRINT_MASTER_EXISTS until the incumbent has missed heartbeats for 5 minutes. A sprint master restarted under the same COLONY_WORKER_NAME has the same worker id as the incumbent, so it takes over immediately.

In Cloud, use Reclaim task for a Worker that cannot finish its claim. In self-hosted Colony, inspect first, then run:

Terminal window
colony workers
colony tasks --status claimed
colony workers reclaim <workerId>

After reclaim, confirm the old Worker is stopped or fenced, the task returned to pending, and another Worker claimed it once. Workers and the monitor also reclaim stale tasks automatically according to heartbeat and task-age policy.

  1. Confirm every intended Worker has a fresh heartbeat and unique identity.
  2. Verify repository scoping and that the expected executor types can be claimed.
  3. Submit one small issue and follow its tasks through analyze, develop, and review.
  4. Compare queue wait, task duration, failures, and cost with the pre-change baseline.
  5. Remove old tokens or containers only after the replacement has completed real work.