Operate and Scale Workers
Workers claim tasks from Colony’s durable queue and run analyzer, planner, developer, reviewer, merger, and sweep executors. A Worker executes one claimed pipeline task at a time; add Workers for concurrency rather than only adding CPU to one process.
Establish a baseline
Section titled “Establish a baseline”Before scaling, record:
- online, stale, and draining Worker counts;
- pending and claimed tasks by type and repository;
- queue wait time and task duration;
- CPU, memory, disk, and checkout/setup time;
- provider and LLM rate-limit errors;
- daily spend and concurrent-issue limits.
More Workers do not help when work is blocked on clarification, dependencies, provider checks, budget, or a misconfigured repository.
Choose a deployment model
Section titled “Choose a deployment model”Managed Workers require no registration. Add BYO Workers when code execution must reach private dependencies, requires specific hardware, or needs separately operated capacity.
BYO Workers authenticate to Colony Cloud with a generated Worker token and continue to depend on the Cloud API, queue, configuration, and GitHub App credentials. They are not an air-gapped mode.
Configure capacity per repository:
repos: - owner: acme repo: payments workers: pool_size: 2 memory: 6g health_port_start: 9200Each process needs a unique internal Worker ID and health port. In multi-repository containers, set COLONY_REPO=owner/repo for every Worker service; an unscoped Worker can start without claiming repository tasks.
Register Cloud BYO Workers
Section titled “Register Cloud BYO Workers”- Open Operations → Fleet (
/operations/fleet), generate a token for one logical Worker group, and copy it while it is visible. - Save the token in a
colony.envfile (COLONY_CLOUD_TOKEN=<token>) and deploy the supported Colony Worker image with--env-file colony.env,--restart unless-stopped, a unique--nameand the same value asCOLONY_WORKER_NAME, the Cloud URL when non-default, and an explicit repository scope when required by the image. The dashboard generates these commands for you. - Wait for registration and a fresh heartbeat in Operations → Fleet.
- Confirm the Worker’s Role, Status, Host and Last Heartbeat in the Workers table before adding more replicas.
- Keep separate tokens for environments or Worker groups so one compromised token can be rotated independently.
Cloud re-mints short-lived provider credentials during registration and heartbeat. Do not inject a long-lived GitHub PAT into a Cloud BYO Worker as a replacement for the Cloud registration flow.
Scale from queue evidence
Section titled “Scale from queue evidence”Scale out when eligible pending tasks wait materially longer than normal while existing Workers remain busy and healthy. Scale down when Workers are persistently idle and no recurring schedule depends on immediate capacity.
| Signal | Interpretation | Action |
|---|---|---|
| Pending queue grows; Workers busy | Capacity-bound. | Add a Worker for the affected repository or eligible pool. |
| Pending queue grows; Workers idle | Routing, repo scope, workflow executor, budget, or scheduling problem. | Diagnose configuration; do not add capacity. |
| Claimed tasks have stale heartbeats | Worker crashed or lost connectivity. | Inspect, then reclaim or let the stale-task policy recover. |
| Tasks fail during setup | Environment, credentials, dependency service, disk, or memory problem. | Fix the Worker image/configuration first. |
| Provider/LLM throttling rises | External rate limit. | Reduce concurrency or obtain the appropriate service capacity. |
Remember that SI tracks, external PR review, planning, review loops, and sweep work use the same Worker capacity as ordinary implementation.
Drain before replacement
Section titled “Drain before replacement”In Operations → Fleet, request Drain for the Worker. Draining prevents new claims and allows its current task to finish. Wait for the Worker to report drained or inactive before stopping the container.
Replace one Worker at a time and confirm another healthy Worker can serve the repository. Generate and deploy a replacement token before revoking the old token during credential rotation.
colony safestop gracefully drains and stops the installation; it is the correct boundary for an upgrade that affects all processes.
For a rolling Worker-only replacement, you need at least two Workers for the repository and orchestration that stops new claims on the retiring process. Verify its claimed task completed before terminating it. The self-host CLI does not expose the same per-Worker drain control as the Cloud dashboard.
Replacing or upgrading workers
Section titled “Replacing or upgrading workers”A worker’s identity comes from its name. Without COLONY_WORKER_NAME, the name follows the container hostname, and Docker gives every new container a random hostname. Recreating the container then registers a new worker, leaves the old one offline, and its claims wait for orphan recovery. The commands generated in Operations → Fleet set COLONY_WORKER_NAME for you.
To replace or upgrade a Worker:
- Drain it, as described above.
- Run
docker stop <name>, thendocker rm <name>.docker run --namefails while a stopped container still holds the name. - Run the same command again with the new image tag and the same
COLONY_WORKER_NAME.
Keep names unique per token (for single-repo Workers, per repository). Two live containers with the same name share one worker row and overwrite each other, so change the name when you copy a command to a second host.
Sprint master succession
Section titled “Sprint master succession”Only one sprint master runs per org. A different sprint master is refused with 409 SPRINT_MASTER_EXISTS until the incumbent has missed heartbeats for 5 minutes. A sprint master restarted under the same COLONY_WORKER_NAME has the same worker id as the incumbent, so it takes over immediately.
Reclaim only abandoned tasks
Section titled “Reclaim only abandoned tasks”In Cloud, use Reclaim task for a Worker that cannot finish its claim. In self-hosted Colony, inspect first, then run:
colony workerscolony tasks --status claimedcolony workers reclaim <workerId>After reclaim, confirm the old Worker is stopped or fenced, the task returned to pending, and another Worker claimed it once. Workers and the monitor also reclaim stale tasks automatically according to heartbeat and task-age policy.
Verify health after a change
Section titled “Verify health after a change”- Confirm every intended Worker has a fresh heartbeat and unique identity.
- Verify repository scoping and that the expected executor types can be claimed.
- Submit one small issue and follow its tasks through analyze, develop, and review.
- Compare queue wait, task duration, failures, and cost with the pre-change baseline.
- Remove old tokens or containers only after the replacement has completed real work.