Top Failure Modes
Top Failure Modes
Section titled “Top Failure Modes”Operator-facing diagnosis and fix guides for the most common Colony failure modes. Each section covers symptoms, root cause, diagnosis steps, fix, and prevention.
Postgres is the authority for pipeline state; GitHub labels are a projection for human visibility. Always check Postgres first when diagnosing pipeline issues.
For the underlying engineering patterns that explain many of the specific symptoms here (retry-without-terminal-classification, silent-fallback at trust boundaries, operator-surface inconsistency), see docs/production-learnings.md § Cross-Cutting Reliability Patterns. When a symptom doesn’t match any single entry below, it’s often an instance of one of those patterns.
Quick Decision Tree
Section titled “Quick Decision Tree”Setup issues? → colony check --fix → confirm prompts → doneIssue not picked up? → colony status → colony check --stage runtime → check intake_modeAgent won't start? → colony check --fix → check .colony/logs/<agent>.logIssue stuck in a state? → colony why <N> → colony tasks --issue <N>Worker stuck / one worker pinned? → colony workers → colony workers reclaim <workerId>Merger blocked on CI-red PR? → colony why <N> → fix failing check → push → /colony:retryBranch CI red on files diff doesn't touch? → branch is stale; merge `main` in (or open a fresh branch)Config confusion? → colony check --stage config → review effective configWorker logs show "Git authentication failed"? → colony check --stage credentials → rotate GITHUB_TOKENReviewer says a check failed then passed / quarantined? → dashboard Review Queue tab → Flaky checks panel → colony metrics → fix the test1. Stale is_blocked Flag
Section titled “1. Stale is_blocked Flag”Symptoms
Section titled “Symptoms”- Issue stuck in a state with no worker activity
work_taskstable has no pending tasks for the issue- Dashboard shows the issue as blocked, but there is no obvious reason
Root Cause
Section titled “Root Cause”is_blocked was set by a transient failure (e.g., push conflict, OOM, subprocess timeout) and never cleared. The sprint-master’s auto_unblock_transient may be disabled or the issue has exceeded max_auto_unblocks_per_issue (default: 3).
Diagnosis Steps
Section titled “Diagnosis Steps”Get a quick read on why the issue is blocked:
colony why <N>Check which issues are blocked:
colony issues --blockedSQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.is_paused, r.owner, r.nameFROM pipeline_issues piJOIN repos r ON r.id = pi.repo_idWHERE pi.is_blocked = true;Confirm worker liveness — verify a worker is registered, healthy, and not pinned to a different issue:
colony workersCheck whether there are any pending tasks for the issue:
colony tasks --issue <N>SQL fallback (no CLI access)
SELECT id, task_type, status, created_atFROM work_tasksWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')ORDER BY created_at DESCLIMIT 10;Check the monitor logs for auto-unblock activity — the self-healing module logs when it unblocks issues and when it skips issues that have exceeded the unblock cap.
Use one of the supported recovery paths, in order of preference:
-
Run the unblock CLI command:
Terminal window colony unblock <N>This transitions the issue out of the blocked state through its workflow snapshot (writing an audit row), resets the station-attempt count, enqueues a work task, and projects the label change through the outbox.
-
In container deployments, run it inside the sprint-master container:
Terminal window docker exec colony-sprint-master colony unblock <N> -
Click Unblock on the dashboard’s Blocked tab, or in the issue’s actions panel.
-
Comment
/colony:retryon the GitHub issue to retry with current settings.
To confirm the current flag state without CLI access, use a read-only query:
SELECT issue_number, state, is_blocked, block_reason, blocked_from_stateFROM pipeline_issuesWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>');Do not clear the flag with a direct UPDATE: it records no state transition, enqueues no work task, and leaves stale block_reason, block_context and blocked_from_state values, so the issue stays stranded in its old state.
Prevention
Section titled “Prevention”Automatic unblocking of transient failures is on by default (auto_unblock_transient: true, max_auto_unblocks_per_issue: 3). Use the config below to raise the cap or to confirm it has not been disabled:
agents: sprint_master: auto_unblock_transient: true max_auto_unblocks_per_issue: 5Each cycle the sprint-master matches the issue’s blockReason (or a strategy’s appliesTo predicate) against its registered recovery strategies. If one matches, the issue is unblocked and moved to a retry state, subject to three limits: a per-issue cooldown (auto_unblock_cooldown_minutes, default 10), a per-cycle cap (max_auto_unblocks_per_cycle, default 3) and a lifetime per-issue cap (max_auto_unblocks_per_issue). Once the lifetime cap is reached, the sprint-master adds the needs-human label and posts a comment instead of retrying.
Relevant code: packages/pipeline-store/src/pipeline-store.ts (is_blocked column), packages/sprint-master/src/self-healing.ts (auto-unblock logic)
2. Head Branch Out of Date on Merge
Section titled “2. Head Branch Out of Date on Merge”Symptoms
Section titled “Symptoms”- Merge fails — merger logs show rebase failures or drift assessment
- Multiple PRs targeting main at the same time
work_taskstable shows failedmergetasks
Root Cause
Section titled “Root Cause”Concurrent PRs — when PR A merges into main, PR B’s branch is stale. The merger attempts to rebase PR B onto the updated main branch. If the rebase produces conflicts, the merger runs a drift assessment to determine whether the conflicts are resolvable.
Diagnosis Steps
Section titled “Diagnosis Steps”Check for failed merge tasks:
colony tasks --type merge --status failedSQL fallback (no CLI access)
SELECT id, issue_number, status, created_at, updated_atFROM work_tasksWHERE task_type = 'merge' AND status = 'failed' AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')ORDER BY created_at DESCLIMIT 10;Check merger logs for rebase failure indicators:
"Rebase failed"— initial rebase attempt failed"Rebase failed, running drift assessment"— merger is evaluating conflict severity"Unable to rebase automatically"— escalation marker indicating conflicts require intervention
The merger computes a “drift overlap” metric (percentage of line overlap between conflicting code) to assess whether auto-resolution is feasible.
The sprint-master automatically re-enqueues merge tasks, and the merger retries the rebase after the conflicting PR has merged. In most cases, the retry succeeds without intervention.
If the issue is stuck:
- Comment
/colony:retryon the GitHub issue to re-enqueue the merge task - For complex conflicts, the merger may escalate with a comment containing conflict details, drift estimates, and a recommendation (manual rebase vs. re-implementation)
- As a last resort, manually rebase the branch:
git rebase origin/mainand force-push
Prevention
Section titled “Prevention”- Before review checks run, the base branch (
origin/<merge target>) is always merged into a detached review worktree so checks observe the same integrated tree the hosted CI builds, reducing stale-branch scenarios at merge time — this step is skipped for epic reviews, since the target merge is already present on the epic branch, and a conflict routes the issue to the merger for a branch update rather than failing the review - The merger has built-in retry logic with drift assessment — most concurrent-PR conflicts resolve automatically on the next attempt
- For repos with high PR throughput, ensure the sprint-master poll interval is short enough to quickly re-enqueue failed merge tasks
Relevant code: packages/merger/src/executor.ts (rebase and drift assessment logic), packages/sprint-master/src/sprint-master.ts (task re-enqueue)
3. Label Projection Failures
Section titled “3. Label Projection Failures”Symptoms
Section titled “Symptoms”- GitHub issue shows the wrong or missing
colony:label - Pipeline is actually progressing — dashboard or Postgres shows the correct state
- Sprint-master logs show label sync errors (rate limit, network timeout)
Root Cause
Section titled “Root Cause”GitHub API rate limit or transient network error during label sync. Labels are a write-only projection of Postgres state, not the source of truth. A failed label update does not affect pipeline processing.
Important: Manually adding or removing colony: labels on GitHub does NOT affect pipeline state. The pipeline reads state exclusively from Postgres. The only exceptions are three supported label commands (see Operator Command Labels): colony:enqueue (seeds a new issue into the pipeline), colony:paused (pauses or resumes a running issue) and colony:ignore (opts an untracked issue out of automatic intake). For all other pipeline control, use slash commands (e.g., /colony:retry, /colony:state <target>) rather than label manipulation.
Diagnosis Steps
Section titled “Diagnosis Steps”Check for issues where Postgres state and stored labels have diverged:
colony issues --label-driftSQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.is_paused, pi.labelsFROM pipeline_issues piWHERE pi.issue_number = <N> AND pi.repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>');Compare the state column (authoritative) with the labels array (projection). If they diverge, the label sync will auto-correct on the next poll cycle.
Check sprint-master logs for label sync errors — the syncLabelsFromPostgres() function runs on every poll cycle and reconciles managed labels (state labels, colony:blocked, colony:paused) against GitHub.
Wait one poll cycle — syncLabelsFromPostgres() auto-corrects within the sprint-master’s poll interval (default 30s). The function compares expected labels (derived from Postgres state and is_blocked/is_paused flags) against actual GitHub labels and issues the necessary add/remove calls.
Manually adding or removing colony:blocked (or any other state label) changes nothing in Postgres: the label is restored on the next poll and the sprint-master posts a one-time hint comment. To unblock an issue, use /colony:retry or colony unblock <n>; to hold it, use /colony:pause. Only colony:enqueue, colony:paused and colony:ignore are inbound label commands (see Operator Command Labels).
To confirm the drift, list affected issues:
colony issues --label-driftDo not use colony issue transition to fix a label: it performs a real pipeline state transition (and requires --reason), not a label re-sync.
Prevention
Section titled “Prevention”This is cosmetic — no action needed. Postgres state is authoritative. The sprint-master’s label sync is self-healing by design and catches up automatically. The label sync processes up to 25 issues per cycle (configurable via label_sync_limit).
Relevant code: packages/sprint-master/src/label-sync.ts (syncLabelsFromPostgres()), packages/core/src/state-transition.ts
4. Worker OOM During npm install
Section titled “4. Worker OOM During npm install”Symptoms
Section titled “Symptoms”- Worker container crashes or is killed during workspace setup
- Container logs show
KilledorOOMKilled - The issue gets blocked after repeated setup failures
Root Cause
Section titled “Root Cause”Default container memory limit is too low for large node_modules trees. npm install can spike memory significantly for repos with many dependencies.
Diagnosis Steps
Section titled “Diagnosis Steps”Check if the container was OOM-killed:
docker inspect <container> | grep OOMKilledOr check container logs:
docker compose logs worker | grep -i killedCheck worker health to confirm whether the container restarted or is still running:
colony workersA dead or stale freshness reading for the worker that was processing the issue confirms it was killed or lost its heartbeat around the time of the failure.
Check the work task failure history for the affected issue:
colony tasks --issue <N> --status failedSQL fallback (no CLI access)
SELECT id, task_type, status, created_atFROM work_tasksWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>') AND status = 'failed'ORDER BY created_at DESCLIMIT 10;Increase the worker memory limit in your config:
repos: - owner: my-org name: my-repo workers: memory: '6g'For very large repos (thousands of dependencies), use '8g' or higher. After updating the config, rebuild and restart the worker containers.
Prevention
Section titled “Prevention”- Set
repos[].workers.memorybased on the target repo’s dependency tree size - Monitor container memory usage during initial workspace setup to establish a baseline
- If using a custom
workspace.setup_command(e.g.,bundle installfor Ruby), the same memory considerations apply
Relevant code: docs/user-guide/configuration.md (repos[].workers.memory field)
5. Planning Timeout Loops
Section titled “5. Planning Timeout Loops”Symptoms
Section titled “Symptoms”- Issue stuck in
ready-for-devwith a high turn count - Multiple failed
developtasks inwork_tasks— sprint-master keeps re-enqueueing - Developer logs show max turns being hit (look for
maxTurnsin structured log output)
Root Cause
Section titled “Root Cause”The issue is too complex or ambiguous for the configured turn limit. The developer exhausts developer_max_turns without completing the task, gets blocked, and the sprint-master retries — creating a loop.
Diagnosis Steps
Section titled “Diagnosis Steps”Use colony why for a quick diagnosis of why the issue is stuck:
colony why <N>Confirm workers are running and not stuck on a different issue before examining task history:
colony workersCheck for repeated develop task failures:
colony tasks --issue <N>SQL fallback (no CLI access)
SELECT id, task_type, status, created_at, updated_atFROM work_tasksWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')ORDER BY created_at DESCLIMIT 10;Check developer logs for max turns indicators:
- The developer executor logs
maxTurnsin its structured output when starting development - When the turn limit is hit, the result includes
isMaxTurns: true - The sprint-master’s self-healing sweep (
autoUnblockRecoverableIssues()) tracks a lifetime auto-unblock counter per issue; once it exceedsmax_auto_unblocks_per_issue(default: 3), the issue is escalated with aneeds-humanlabel instead of retried again
Stop the retry loop:
/colony:cancelComment this on the GitHub issue to close it and stop re-enqueue.
Alternatively, decompose the issue into smaller sub-issues:
/colony:decomposeThis sends the issue to the planner, which breaks it into smaller, more tractable sub-issues.
If the issue is close to completion and just needs more turns, bump the limits:
claude: scaling: large: developer_max_turns: 350Complexity tiers (small, medium, large) each have their own developer_max_turns setting. The defaults are 80 (small), 150 (medium) and 250 (large), so choose a value above the default for the tier.
Prevention
Section titled “Prevention”- Use the planner for large issues — comment
/colony:decomposebefore the issue enters development - Configure progress detection windows to catch stalls early:
(75 is the default for theclaude:scaling:large:no_progress_window: 75
largetier; use a smaller value to catch stalls sooner.) - Write well-scoped issues with clear acceptance criteria — ambiguous issues are the primary driver of timeout loops
- The self-healing auto-unblock counter is a lifetime cap per issue, not a sliding window; once
max_auto_unblocks_per_issueis exceeded, the issue stops being auto-retried and escalates toneeds-humanfor an operator to break the loop manually
Relevant code: packages/developer/src/executor.ts (turn limit logic), packages/sprint-master/src/self-healing.ts (autoUnblockRecoverableIssues() — lifetime auto-unblock cap and needs-human escalation), packages/sprint-master/src/slash-commands/handlers/cancel.ts (handleCancel()) and packages/sprint-master/src/slash-commands/handlers/decompose.ts (handleDecompose()), docs/user-guide/configuration.md (claude.scaling)
6. PR Blocked by Hard CI Failure
Section titled “6. PR Blocked by Hard CI Failure”Symptoms
Section titled “Symptoms”Primary (common case): Issue blocked with block_reason=ci_hard_failure:
colony why <N>reportsci_hard_failureas the block reasoncolony issues --blockedlists the issue- PR has a
FAILURECI check that won’t recover without a code or infra fix - A single bot comment on the PR identifies the failing check(s) and the HEAD SHA they were detected on
Secondary (rare — orphaned claim): A stale or dead worker that held a merge task claim before the failure classification landed:
colony workersshowsstaleordeadfreshness for the worker last seen on this issue- No new tasks are being created for the issue despite the block not being set
Root Cause
Section titled “Root Cause”When the merger detects a definitive CI failure, compareCiChecks() (packages/merger/src/ci-comparison.ts) classifies the failure as PR-introduced (not pre-existing on the base branch). Before reporting a PR-introduced failure, it reruns the failed checks up to review.ci_flaky_reruns times (default 1; 0 disables, and code hosts without rerun support skip it); checks that pass on rerun are recorded as flakes, annotated as passed on rerun (flaky), and do not trigger ci_hard_failure. Pre-existing failures are not rerun. The executor then returns block_reason: 'ci_hard_failure', which causes the worker to block the issue rather than re-enqueue. The status comment is SHA-scoped via postCommentOnce({ kind, key: newHeadSha }), so it is posted once per HEAD SHA and not repeated until the author pushes a new commit.
This is the resolved instance of the cross-cutting pattern “every retry path needs an explicit terminal classification” (see production-learnings § Cross-Cutting Reliability Patterns).
A cancelled check conclusion counts as incomplete, not failed. It never triggers ci_hard_failure or CI repair, and it never counts as green, so repositories may use cancel-in-progress concurrency. A cancelled required check is treated like a run still in progress: Colony waits until it is re-run or superseded by a newer commit, or the CI poll times out.
Diagnosis Steps
Section titled “Diagnosis Steps”Get a quick read on the block reason:
colony why <N>A ci_hard_failure block reason confirms the merger detected a definitive CI failure and blocked the issue. Check which issues are currently blocked:
colony issues --blockedSQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.block_reason, r.owner, r.nameFROM pipeline_issues piJOIN repos r ON r.id = pi.repo_idWHERE pi.is_blocked = true;Check which CI checks are failing on the PR to identify what needs fixing:
gh pr checks <pr-number> --repo <owner>/<repo>For the orphaned-claim sub-case, check worker liveness:
colony workersIf the worker’s freshness shows stale or dead, the work_tasks claim is held by a dead worker — the issue will not make progress until the claim is released.
Check the recent task history to confirm the issue is blocked and not looping:
colony tasks --issue <N>SQL fallback (no CLI access)
SELECT id, task_type, status, claimed_by, created_at, completed_atFROM work_tasksWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')ORDER BY created_at DESCLIMIT 15;Scenario A — stale or dead worker (orphaned claim):
If colony workers shows stale or dead freshness for the worker that last held a task on this issue, free the lease:
colony workers reclaim <workerId>This returns the claimed task to pending so any available worker can pick it up. On success:
Reclaimed <taskType> task for issue #<N> (task <taskId>) from worker <workerId>Scenario B — issue blocked with block_reason=ci_hard_failure:
-
Fix the failing CI check (push a code fix, repair the CI workflow, rotate a broken secret — whatever the failing check requires).
-
Push the fix to the PR branch.
-
Comment
/colony:retryon the GitHub issue:/colony:retryThis clears the block and re-enqueues the merge task. The merger will re-run
compareCiChecks()against the new HEAD SHA; if all required checks now pass, it proceeds to merge.
No manual loop-breaking is needed — the block state prevents re-enqueueing until the retry is explicitly requested.
Prevention
Section titled “Prevention”- Triage flaky tests on
mainaggressively — a flake that lands onmaincauses hard failures on every open PR branch until fixed or worked around. - For repos with custom CI jobs (Cloudflare Pages deploys, content validation, security scans), keep
review.checksconfig aligned with the CI workflow’s check-run names so the reviewer catches breakage locally before the merger sees it.
Relevant code: packages/merger/src/ci-comparison.ts (compareCiChecks() — FAILURE vs transient distinction), packages/merger/src/executor.ts (the ciResult.prIntroduced branch that returns blockReason: 'ci_hard_failure', with SHA-scoped postCommentOnce dedup).
7. Stale Branch Failing CI After main Moved
Section titled “7. Stale Branch Failing CI After main Moved”Symptoms
Section titled “Symptoms”- A PR’s CI was passing yesterday and is failing today, with no changes pushed in between
- The failing check points at a file that isn’t in the branch’s diff (e.g.,
.gitmodules,package-lock.json, a CI workflow YAML, an SSH/secrets setup step) - Multiple unrelated PRs all fail the same way at roughly the same time
- The error often mentions submodules, missing dependencies, secret/auth setup, or workflow steps the branch never touched
Root Cause
Section titled “Root Cause”A structural change landed on main — submodule schema change, lockfile bump, CI workflow change, secret rotation — that every open branch must absorb before its CI can pass again. Branches forked before that commit don’t yet have the new shape; their CI re-runs against a checkout that is internally inconsistent (e.g., a gitlink with no matching .gitmodules entry, or a workflow expecting a secret only the new main knows how to fetch).
This is the operator-visible surface of the cross-cutting pattern “trunk drift has O(open branches) blast radius” (see production-learnings § Cross-Cutting Reliability Patterns). The pattern shows up across pipeline-initiated work because there are typically 10+ branches open at any time, all individually exposed to the same main advance.
Diagnosis Steps
Section titled “Diagnosis Steps”Confirm the failing check references a file the branch doesn’t touch:
gh pr diff <pr-number> --repo <owner>/<repo> --name-onlygh run view <run-id> --repo <owner>/<repo> --log-failed | head -100If the failed step references a file not in the diff (commonly .gitmodules, .github/workflows/*.yml, package-lock.json, ssh config, secret-fetch steps), the breakage is on main, not in the branch.
Cross-check by looking at recent landings on main for structural changes:
git log --oneline origin/main -20 -- .gitmodules .github/workflows/ package-lock.jsonA recent commit touching any of these is almost certainly the trigger.
Merge main into the branch and resolve any conflicts that surface:
git checkout <branch>git fetch origin maingit merge origin/main# resolve conflicts (commonly tests with expanded mock chains, submodule pointers)git push origin <branch>If the branch is one you opened manually, this is straightforward. If the branch is owned by Colony (a feature branch or epic branch), merging main in is still safe — push the merge commit and the next pipeline cycle will re-run CI with the absorbed change.
If the branch is far behind and conflicts are unrelated to the feature, consider abandoning the branch and re-creating from the current main (the issue may transition back to ready-for-dev to regenerate the work).
Prevention
Section titled “Prevention”- After landing a structural change on
main(submodule add/remove, lockfile bump, CI workflow rewrite, secret rotation), expect every open PR to need amainmerge — plan a sweep, not one-by-one repair. - Auto-merge-
main-into-open-branches whenmainadvances is the structural fix (tracked as engineering work; until then, treat post-structural-change as an “all branches need touchups” event). - Distinguish “branch CI red on files not in diff” (stale-vs-main) from “branch CI red on files in diff” (feature bug) in operator triage — they need opposite responses (merge-main vs. fix-the-code).
Relevant code: packages/merger/src/ (where merge-main automation would live), .github/workflows/ (the workflow files most often involved in trunk-drift cascades).
8. Git Authentication Failed During Workspace Setup
Section titled “8. Git Authentication Failed During Workspace Setup”Symptoms
Section titled “Symptoms”- Worker logs show
Git authentication failed for <owner>/<repo>with no code changes attempted - The issue never leaves its current state — no analysis, no development, no review activity
work_tasksshows a failed task of any type, with the failure occurring at workspace-setup time- Raw git output visible in logs:
Authentication failed,Invalid username or token,Password authentication is not supported for Git operations, orcould not read Username
Root Cause
Section titled “Root Cause”Colony’s workspace manager calls git clone or git fetch using an HTTPS remote URL. GitHub requires a valid credential (Personal Access Token or GitHub App installation token) embedded in the URL or supplied by a git credential helper. The failure occurs when:
GITHUB_TOKEN(or the env var named bygithub.token_env) is missing from the worker’s environment- The token has expired (fine-grained PATs have a mandatory expiry; classic PATs can expire)
- The token lacks the required scopes:
repofor classic PATs, or Contents + Issues + Pull-requests read-write (plus Checks, Commit statuses, Actions read) for fine-grained PATs - The git credential helper is configured to send a username/password instead of a token (GitHub disabled password auth for git operations in 2021)
- A GitHub App’s installation token was not minted correctly for the target repository
Diagnosis Steps
Section titled “Diagnosis Steps”Run the Colony credential check:
colony check --stage credentialsThis validates that the configured token is present, well-formed, and can authenticate to the GitHub API. It does not test git-over-HTTPS directly, but a passing credential check combined with a passing repository check (colony check --stage repository) covers the most common causes.
For container deployments, confirm the token is visible inside the worker container:
docker compose exec worker printenv GITHUB_TOKENIf this prints nothing, the .env file is missing or the compose file does not forward the variable.
Check worker logs for the raw git error:
docker compose logs worker | grep -i "authentication\|username\|token"Confirm the git remote URL includes the token (the workspace manager should inject x-access-token:<TOKEN>@github.com):
docker compose exec worker git -C .colony/workspaces/issue-<N> remote get-url originIf the URL shows https://github.com/... without credentials, the token injection step failed.
-
Rotate or re-export the token:
Terminal window # Edit your .env file and set the correct token valueGITHUB_TOKEN=ghp_... -
Restart the worker containers so they pick up the new environment:
Terminal window docker compose up -d workerWorkers read environment variables at startup — a running container does not see changes to
.envuntil it is restarted. -
Verify the fix:
Terminal window colony check --stage credentials -
Retry the blocked issue:
/colony:retryComment this on the GitHub issue to re-enqueue it. Because the failure was environmental (no code changes were attempted), no analysis or re-decomposition is needed.
Prevention
Section titled “Prevention”- Token rotation reminders: Set calendar reminders for PAT expiry dates. Fine-grained PATs have a maximum 366-day lifetime; classic PATs default to no expiry but should be rotated on a schedule.
- GitHub App tokens: Consider switching to a GitHub App for zero-manual-expiry installation tokens. The App’s installation token is minted per-request by Colony’s auth layer (
packages/core/src/github-auth.ts) and does not need manual rotation. - Credential helper pattern: Ensure the git remote URL uses the
https://x-access-token:<token>@github.com/<owner>/<repo>.gitformat, not a barehttps://URL that relies on a system credential helper. - Run
colony checkafter any token rotation before re-deploying workers to catch scope or expiry issues before they block live issues.
Relevant code: packages/core/src/clone.ts (cloneRepo and cloneRepoAsync — git auth failures are detected and surfaced as a clear error), packages/core/src/github-auth.ts (token injection for GitHub App), docs/user-guide/configuration.md (github.token_env)
9. Runtime Not Found in bash -lc setup_command or review.checks
Section titled “9. Runtime Not Found in bash -lc setup_command or review.checks”Symptoms
Section titled “Symptoms”python: command not found,node: command not found,ruby: command not found, or similar errors during workspace setup or review checks- The failure occurs only when the command is wrapped in a login shell (
bash -lc '...') — the same command viabash -c '...'succeeds - Running
docker compose exec worker bash -c 'python --version'works, butdocker compose exec worker bash -lc 'python --version'fails
Root Cause
Section titled “Root Cause”The production image adds language runtimes to PATH via Docker ENV directives (mise shims at /usr/local/share/mise/shims, dotnet tools, mssql-tools18, and workspace bin scripts). These ENV layers are inherited by the worker process and any non-login child shells it spawns.
When a login shell is invoked (bash -l or bash -lc), Bash sources /etc/profile. On Debian, /etc/profile resets PATH to a fixed minimal default that does not include the image’s custom directories. Login shells therefore cannot find mise-managed runtimes like Python, Node, or Ruby — even though those runtimes are installed and resolve fine in the worker’s own (non-login) environment.
Related but distinct: task-installed CLI entry points. The image also puts the mise installs/<runtime>/<version>/bin directories for the pinned Python (3.12, 3.11) and Node (20, 22) versions on PATH, alongside the shims. This lets a CLI entry point installed at task runtime — pip install, npm install -g — resolve by name immediately, in both login and non-login shells, without waiting for a mise reshim. A repo that pins a Python or Node version other than the four listed above (via .python-version, .node-version, or .tool-versions) is not covered by this static list: a task-time install under that version’s bin directory will still produce command not found immediately after the install reports success, since only the versions known at image build time are on PATH.
Diagnosis Steps
Section titled “Diagnosis Steps”Confirm the symptom is login-shell-specific by running both variants inside the container:
# This should succeed (non-login shell inherits worker PATH)docker compose exec worker bash -c 'python --version && node --version'
# This fails without the fix (login shell resets PATH)docker compose exec worker bash -lc 'python --version && node --version'If the non-login form succeeds and the login form fails, this is the issue. Check whether setup_command or a review.checks entry in your config wraps the command in bash -lc '...'.
Option A — use a non-login shell wrapper (recommended):
Replace bash -lc '...' with bash -c '...' in your setup_command or review.checks config. Non-login shells inherit the worker’s PATH and resolve all mise-managed runtimes without any extra configuration:
# Before (broken)workspace: setup_command: "bash -lc 'bundle install && rails db:migrate'"
# After (fixed)workspace: setup_command: "bash -c 'bundle install && rails db:migrate'"
# Or just omit the shell wrapper entirely for single commandsworkspace: setup_command: "bundle install"Option B — upgrade to an image that includes /etc/profile.d/colony-path.sh:
Production images built after this fix include /etc/profile.d/colony-path.sh, which re-prepends the custom PATH segments after /etc/profile resets them. Login shells (bash -lc) in those images resolve mise runtimes identically to non-login shells. No config change is needed — rebuild or pull the updated image. Note that colony-path.sh no longer adds /colony/node_modules/.bin, so login shells do not see Colony’s CLI or tools; use bash -c or the full path /colony/node_modules/.bin/colony.
Verify the file is present in your image:
docker compose exec worker cat /etc/profile.d/colony-path.shPrevention
Section titled “Prevention”- Prefer plain commands or
bash -c '...'wrappers insetup_commandandreview.checks— non-login shells always inherit the worker’s PATH. - Avoid
bash -lcin these config values; it provides no benefit overbash -cin a container environment where.bashrcand/etc/profile.d/are not customized for interactive use.
Relevant code: Dockerfile (/etc/profile.d/colony-path.sh RUN step, ENV PATH=... layers), packages/core/src/workspace.ts (execSync calls for setup_command), packages/reviewer/src/checks.ts (execSync calls for review.checks)
10. A Deterministic Review Check Is Flaky
Section titled “10. A Deterministic Review Check Is Flaky”Symptoms
Section titled “Symptoms”- The reviewer comment says a check failed and then passed when re-run on the PR tree
- The reviewer comment says a check is quarantined
- Before flaky detection, the developer agent was repeatedly asked to fix code that wasn’t broken, burning review cycles on an intermittent test
Root Cause
Section titled “Root Cause”How Colony handles it:
- A failed blocking check is re-run on the PR’s own tree up to
review.flaky_rerunstimes (default1) before the base-branch comparison. - A passing rerun marks the check
flaky. This is advisory and does not request changes. - Every attempt is recorded in the
review_check_runstable. - A check with at least
review.flaky_quarantine_thresholdflakes in the last 14 days is quarantined. Its failures are advisory, and the review comment says so. - A quarantined check is still re-run on failure, so its flakes keep being recorded. Quarantine lifts only once the check stops flaking for long enough that its 14-day flake count falls below the threshold.
- Quarantine only applies to the reviewer. Developer self-validation and merger conflict-resolution checks still treat the check as blocking.
review_check_runsrows older than 90 days are pruned by the sprint-master.
Diagnosis Steps
Section titled “Diagnosis Steps”- Open the Flaky checks panel on the dashboard’s Review Queue tab (see dashboard.md).
- Query the
colony_review_check_flake_rateandcolony_review_check_flakesPrometheus gauges (see metrics.md). - Run
colony metricsand read the Flaky checks section (see cli-reference.md). - Call the
colony_flaky_checksMCP tool (see mcp.md).
- Fix or stabilise the test in the target repo (timeouts, ordering, shared state).
- Set
review.flaky_reruns: 0to turn off reruns. - Raise or unset
review.flaky_quarantine_thresholdto stop quarantining.
Quarantine is advisory, so real regressions in a quarantined check won’t block review until its flake count drops back under the threshold.
Prevention
Section titled “Prevention”- Fix flaky tests promptly rather than relying on quarantine.
- Keep
flaky_quarantine_thresholdhigh enough that a real regression isn’t masked by a few unrelated flakes.
CI checks are rerun once too
Section titled “CI checks are rerun once too”A failing CI check (the GitHub check runs on the PR, not the reviewer’s local checks) is also rerun once on the same head before the reviewer routes back:
- The reviewer asks the code host to rerun the failed check runs and waits up to
review.ci_check_timeoutfor the new results. If the PR head moved in the meantime, no rerun happens. - If every failed check passes on rerun, the reviewer treats CI as passed, records the flake (source
ci) and posts a short flaky-CI notice. If a check fails again, the issue goes back to the developer as before. - The route-back comment keeps its first line and adds one line per check, e.g.
`Database-backed tests` — failed again on rerun; flaked 7 times in the last 14 days (see review.ci_flaky_reruns). The count is the number of CI flakes in the rolling 14-day window; a high count suggests the failure is the check, not the change. The same count is appended to the matching action item’s required action (the finding title stays stable). - Rerunning needs
actions:writeon the token. Without it, or on a code host that cannot rerun checks, the reviewer routes back immediately. - Set
review.ci_flaky_reruns: 0to disable CI reruns.
See the review section of configuration.md for the config keys.