Operate and Scale Workers

Workers claim tasks from Colony’s durable queue and run analyzer, planner, developer, reviewer, merger, and sweep executors. A Worker executes one claimed pipeline task at a time; add Workers for concurrency rather than only adding CPU to one process.

Establish a baseline

Before scaling, record:

• online, stale, and draining Worker counts;
• pending and claimed tasks by type and repository;
• queue wait time and task duration;
• CPU, memory, disk, and checkout/setup time;
• provider and LLM rate-limit errors;
• daily spend and concurrent-issue limits.

More Workers do not help when work is blocked on clarification, dependencies, provider checks, budget, or a misconfigured repository.

Choose a deployment model

Colony Cloud

Managed Workers require no registration. Add BYO Workers when code execution must reach private dependencies, requires specific hardware, or needs separately operated capacity.

BYO Workers authenticate to Colony Cloud with a generated Worker token and continue to depend on the Cloud API, queue, configuration, and GitHub App credentials. They are not an air-gapped mode.

Self-Host

Configure capacity per repository:

repos:
  - owner: acme
    repo: payments
    workers:
      pool_size: 2
      memory: 6g
      health_port_start: 9200

Each process needs a unique internal Worker ID and health port. In multi-repository containers, set COLONY_REPO=owner/repo for every Worker service; an unscoped Worker can start without claiming repository tasks.

Register Cloud BYO Workers

1. Open Settings → Workers, generate a token for one logical Worker group, and copy it while it is visible.
2. Deploy the supported Colony Worker image with COLONY_CLOUD_TOKEN, the Cloud URL when non-default, and an explicit repository scope when required by the image.
3. Wait for registration and a fresh heartbeat in Settings → Workers.
4. Confirm the Worker reports the intended role, repository, version, and scheduling status before adding more replicas.
5. Keep separate tokens for environments or Worker groups so one compromised token can be rotated independently.

Cloud re-mints short-lived provider credentials during registration and heartbeat. Do not inject a long-lived GitHub PAT into a Cloud BYO Worker as a replacement for the Cloud registration flow.

Scale from queue evidence

Scale out when eligible pending tasks wait materially longer than normal while existing Workers remain busy and healthy. Scale down when Workers are persistently idle and no recurring schedule depends on immediate capacity.

Signal	Interpretation	Action
Pending queue grows; Workers busy	Capacity-bound.	Add a Worker for the affected repository or eligible pool.
Pending queue grows; Workers idle	Routing, repo scope, workflow executor, budget, or scheduling problem.	Diagnose configuration; do not add capacity.
Claimed tasks have stale heartbeats	Worker crashed or lost connectivity.	Inspect, then reclaim or let the stale-task policy recover.
Tasks fail during setup	Environment, credentials, dependency service, disk, or memory problem.	Fix the Worker image/configuration first.
Provider/LLM throttling rises	External rate limit.	Reduce concurrency or obtain the appropriate service capacity.

Remember that SI tracks, external PR review, planning, review loops, and sweep work use the same Worker capacity as ordinary implementation.

Drain before replacement

Colony Cloud

In Settings → Workers, request Drain for the Worker. Draining prevents new claims and allows its current task to finish. Wait for the Worker to report drained or inactive before stopping the container.

Replace one Worker at a time and confirm another healthy Worker can serve the repository. Generate and deploy a replacement token before revoking the old token during credential rotation.

Self-Host

colony safestop gracefully drains and stops the installation; it is the correct boundary for an upgrade that affects all processes.

For a rolling Worker-only replacement, you need at least two Workers for the repository and orchestration that stops new claims on the retiring process. Verify its claimed task completed before terminating it. The self-host CLI does not expose the same per-Worker drain control as the Cloud dashboard.

Caution

Do not use task reclaim as a normal drain mechanism. Reclaim returns a claimed task to the queue; if the original process is still running, two Workers may execute the same task until lease and staleness checks stop one of them.

Reclaim only abandoned tasks

In Cloud, use Reclaim task for a Worker that cannot finish its claim. In self-hosted Colony, inspect first, then run:

colony workers
colony tasks --status claimed
colony workers reclaim <workerId>

After reclaim, confirm the old Worker is stopped or fenced, the task returned to pending, and another Worker claimed it once. Workers and the monitor also reclaim stale tasks automatically according to heartbeat and task-age policy.

Verify health after a change

1. Confirm every intended Worker has a fresh heartbeat and unique identity.
2. Verify repository scoping and that the expected executor types can be claimed.
3. Submit one small issue and follow its tasks through analyze, develop, and review.
4. Compare queue wait, task duration, failures, and cost with the pre-change baseline.
5. Remove old tokens or containers only after the replacement has completed real work.

What next

• Triage a Stuck Issue
• Rotate Credentials Without Stopping the Pipeline
• Tune Model Routing and Cost Caps