Skip to content

Top Failure Modes

Operator-facing diagnosis and fix guides for the most common Colony failure modes. Each section covers symptoms, root cause, diagnosis steps, fix, and prevention.

Postgres is the authority for pipeline state; GitHub labels are a projection for human visibility. Always check Postgres first when diagnosing pipeline issues.

For the underlying engineering patterns that explain many of the specific symptoms here (retry-without-terminal-classification, silent-fallback at trust boundaries, operator-surface inconsistency), see docs/production-learnings.md § Cross-Cutting Reliability Patterns. When a symptom doesn’t match any single entry below, it’s often an instance of one of those patterns.

Setup issues? → colony check --fix → confirm prompts → done
Issue not picked up? → colony status → colony check --stage runtime → check intake_mode
Agent won't start? → colony check --fix → check .colony/logs/<agent>.log
Issue stuck in a state? → colony why <N> → colony tasks --issue <N>
Worker stuck / one worker pinned? → colony workers → colony workers reclaim <workerId>
Merger blocked on CI-red PR? → colony why <N> → fix failing check → push → /colony:retry
Branch CI red on files diff doesn't touch? → branch is stale; merge `main` in (or open a fresh branch)
Config confusion? → colony check --stage config → review effective config
Worker logs show "Git authentication failed"? → colony check --stage credentials → rotate GITHUB_TOKEN
Reviewer says a check failed then passed / quarantined? → dashboard Review Queue tab → Flaky checks panel → colony metrics → fix the test

  • Issue stuck in a state with no worker activity
  • work_tasks table has no pending tasks for the issue
  • Dashboard shows the issue as blocked, but there is no obvious reason

is_blocked was set by a transient failure (e.g., push conflict, OOM, subprocess timeout) and never cleared. The sprint-master’s auto_unblock_transient may be disabled or the issue has exceeded max_auto_unblocks_per_issue (default: 3).

Get a quick read on why the issue is blocked:

Terminal window
colony why <N>

Check which issues are blocked:

Terminal window
colony issues --blocked
SQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.is_paused, r.owner, r.name
FROM pipeline_issues pi
JOIN repos r ON r.id = pi.repo_id
WHERE pi.is_blocked = true;

Confirm worker liveness — verify a worker is registered, healthy, and not pinned to a different issue:

Terminal window
colony workers

Check whether there are any pending tasks for the issue:

Terminal window
colony tasks --issue <N>
SQL fallback (no CLI access)
SELECT id, task_type, status, created_at
FROM work_tasks
WHERE issue_number = <N>
AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')
ORDER BY created_at DESC
LIMIT 10;

Check the monitor logs for auto-unblock activity — the self-healing module logs when it unblocks issues and when it skips issues that have exceeded the unblock cap.

Use one of the supported recovery paths, in order of preference:

  1. Run the unblock CLI command:

    Terminal window
    colony unblock <N>

    This transitions the issue out of the blocked state through its workflow snapshot (writing an audit row), resets the station-attempt count, enqueues a work task, and projects the label change through the outbox.

  2. In container deployments, run it inside the sprint-master container:

    Terminal window
    docker exec colony-sprint-master colony unblock <N>
  3. Click Unblock on the dashboard’s Blocked tab, or in the issue’s actions panel.

  4. Comment /colony:retry on the GitHub issue to retry with current settings.

To confirm the current flag state without CLI access, use a read-only query:

SELECT issue_number, state, is_blocked, block_reason, blocked_from_state
FROM pipeline_issues
WHERE issue_number = <N>
AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>');

Do not clear the flag with a direct UPDATE: it records no state transition, enqueues no work task, and leaves stale block_reason, block_context and blocked_from_state values, so the issue stays stranded in its old state.

Automatic unblocking of transient failures is on by default (auto_unblock_transient: true, max_auto_unblocks_per_issue: 3). Use the config below to raise the cap or to confirm it has not been disabled:

agents:
sprint_master:
auto_unblock_transient: true
max_auto_unblocks_per_issue: 5

Each cycle the sprint-master matches the issue’s blockReason (or a strategy’s appliesTo predicate) against its registered recovery strategies. If one matches, the issue is unblocked and moved to a retry state, subject to three limits: a per-issue cooldown (auto_unblock_cooldown_minutes, default 10), a per-cycle cap (max_auto_unblocks_per_cycle, default 3) and a lifetime per-issue cap (max_auto_unblocks_per_issue). Once the lifetime cap is reached, the sprint-master adds the needs-human label and posts a comment instead of retrying.

Relevant code: packages/pipeline-store/src/pipeline-store.ts (is_blocked column), packages/sprint-master/src/self-healing.ts (auto-unblock logic)


  • Merge fails — merger logs show rebase failures or drift assessment
  • Multiple PRs targeting main at the same time
  • work_tasks table shows failed merge tasks

Concurrent PRs — when PR A merges into main, PR B’s branch is stale. The merger attempts to rebase PR B onto the updated main branch. If the rebase produces conflicts, the merger runs a drift assessment to determine whether the conflicts are resolvable.

Check for failed merge tasks:

Terminal window
colony tasks --type merge --status failed
SQL fallback (no CLI access)
SELECT id, issue_number, status, created_at, updated_at
FROM work_tasks
WHERE task_type = 'merge'
AND status = 'failed'
AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')
ORDER BY created_at DESC
LIMIT 10;

Check merger logs for rebase failure indicators:

  • "Rebase failed" — initial rebase attempt failed
  • "Rebase failed, running drift assessment" — merger is evaluating conflict severity
  • "Unable to rebase automatically" — escalation marker indicating conflicts require intervention

The merger computes a “drift overlap” metric (percentage of line overlap between conflicting code) to assess whether auto-resolution is feasible.

The sprint-master automatically re-enqueues merge tasks, and the merger retries the rebase after the conflicting PR has merged. In most cases, the retry succeeds without intervention.

If the issue is stuck:

  • Comment /colony:retry on the GitHub issue to re-enqueue the merge task
  • For complex conflicts, the merger may escalate with a comment containing conflict details, drift estimates, and a recommendation (manual rebase vs. re-implementation)
  • As a last resort, manually rebase the branch: git rebase origin/main and force-push
  • Before review checks run, the base branch (origin/<merge target>) is always merged into a detached review worktree so checks observe the same integrated tree the hosted CI builds, reducing stale-branch scenarios at merge time — this step is skipped for epic reviews, since the target merge is already present on the epic branch, and a conflict routes the issue to the merger for a branch update rather than failing the review
  • The merger has built-in retry logic with drift assessment — most concurrent-PR conflicts resolve automatically on the next attempt
  • For repos with high PR throughput, ensure the sprint-master poll interval is short enough to quickly re-enqueue failed merge tasks

Relevant code: packages/merger/src/executor.ts (rebase and drift assessment logic), packages/sprint-master/src/sprint-master.ts (task re-enqueue)


  • GitHub issue shows the wrong or missing colony: label
  • Pipeline is actually progressing — dashboard or Postgres shows the correct state
  • Sprint-master logs show label sync errors (rate limit, network timeout)

GitHub API rate limit or transient network error during label sync. Labels are a write-only projection of Postgres state, not the source of truth. A failed label update does not affect pipeline processing.

Important: Manually adding or removing colony: labels on GitHub does NOT affect pipeline state. The pipeline reads state exclusively from Postgres. The only exceptions are three supported label commands (see Operator Command Labels): colony:enqueue (seeds a new issue into the pipeline), colony:paused (pauses or resumes a running issue) and colony:ignore (opts an untracked issue out of automatic intake). For all other pipeline control, use slash commands (e.g., /colony:retry, /colony:state <target>) rather than label manipulation.

Check for issues where Postgres state and stored labels have diverged:

Terminal window
colony issues --label-drift
SQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.is_paused, pi.labels
FROM pipeline_issues pi
WHERE pi.issue_number = <N>
AND pi.repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>');

Compare the state column (authoritative) with the labels array (projection). If they diverge, the label sync will auto-correct on the next poll cycle.

Check sprint-master logs for label sync errors — the syncLabelsFromPostgres() function runs on every poll cycle and reconciles managed labels (state labels, colony:blocked, colony:paused) against GitHub.

Wait one poll cycle — syncLabelsFromPostgres() auto-corrects within the sprint-master’s poll interval (default 30s). The function compares expected labels (derived from Postgres state and is_blocked/is_paused flags) against actual GitHub labels and issues the necessary add/remove calls.

Manually adding or removing colony:blocked (or any other state label) changes nothing in Postgres: the label is restored on the next poll and the sprint-master posts a one-time hint comment. To unblock an issue, use /colony:retry or colony unblock <n>; to hold it, use /colony:pause. Only colony:enqueue, colony:paused and colony:ignore are inbound label commands (see Operator Command Labels).

To confirm the drift, list affected issues:

Terminal window
colony issues --label-drift

Do not use colony issue transition to fix a label: it performs a real pipeline state transition (and requires --reason), not a label re-sync.

This is cosmetic — no action needed. Postgres state is authoritative. The sprint-master’s label sync is self-healing by design and catches up automatically. The label sync processes up to 25 issues per cycle (configurable via label_sync_limit).

Relevant code: packages/sprint-master/src/label-sync.ts (syncLabelsFromPostgres()), packages/core/src/state-transition.ts


  • Worker container crashes or is killed during workspace setup
  • Container logs show Killed or OOMKilled
  • The issue gets blocked after repeated setup failures

Default container memory limit is too low for large node_modules trees. npm install can spike memory significantly for repos with many dependencies.

Check if the container was OOM-killed:

Terminal window
docker inspect <container> | grep OOMKilled

Or check container logs:

Terminal window
docker compose logs worker | grep -i killed

Check worker health to confirm whether the container restarted or is still running:

Terminal window
colony workers

A dead or stale freshness reading for the worker that was processing the issue confirms it was killed or lost its heartbeat around the time of the failure.

Check the work task failure history for the affected issue:

Terminal window
colony tasks --issue <N> --status failed
SQL fallback (no CLI access)
SELECT id, task_type, status, created_at
FROM work_tasks
WHERE issue_number = <N>
AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')
AND status = 'failed'
ORDER BY created_at DESC
LIMIT 10;

Increase the worker memory limit in your config:

repos:
- owner: my-org
name: my-repo
workers:
memory: '6g'

For very large repos (thousands of dependencies), use '8g' or higher. After updating the config, rebuild and restart the worker containers.

  • Set repos[].workers.memory based on the target repo’s dependency tree size
  • Monitor container memory usage during initial workspace setup to establish a baseline
  • If using a custom workspace.setup_command (e.g., bundle install for Ruby), the same memory considerations apply

Relevant code: docs/user-guide/configuration.md (repos[].workers.memory field)


  • Issue stuck in ready-for-dev with a high turn count
  • Multiple failed develop tasks in work_tasks — sprint-master keeps re-enqueueing
  • Developer logs show max turns being hit (look for maxTurns in structured log output)

The issue is too complex or ambiguous for the configured turn limit. The developer exhausts developer_max_turns without completing the task, gets blocked, and the sprint-master retries — creating a loop.

Use colony why for a quick diagnosis of why the issue is stuck:

Terminal window
colony why <N>

Confirm workers are running and not stuck on a different issue before examining task history:

Terminal window
colony workers

Check for repeated develop task failures:

Terminal window
colony tasks --issue <N>
SQL fallback (no CLI access)
SELECT id, task_type, status, created_at, updated_at
FROM work_tasks
WHERE issue_number = <N>
AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')
ORDER BY created_at DESC
LIMIT 10;

Check developer logs for max turns indicators:

  • The developer executor logs maxTurns in its structured output when starting development
  • When the turn limit is hit, the result includes isMaxTurns: true
  • The sprint-master’s self-healing sweep (autoUnblockRecoverableIssues()) tracks a lifetime auto-unblock counter per issue; once it exceeds max_auto_unblocks_per_issue (default: 3), the issue is escalated with a needs-human label instead of retried again

Stop the retry loop:

/colony:cancel

Comment this on the GitHub issue to close it and stop re-enqueue.

Alternatively, decompose the issue into smaller sub-issues:

/colony:decompose

This sends the issue to the planner, which breaks it into smaller, more tractable sub-issues.

If the issue is close to completion and just needs more turns, bump the limits:

claude:
scaling:
large:
developer_max_turns: 350

Complexity tiers (small, medium, large) each have their own developer_max_turns setting. The defaults are 80 (small), 150 (medium) and 250 (large), so choose a value above the default for the tier.

  • Use the planner for large issues — comment /colony:decompose before the issue enters development
  • Configure progress detection windows to catch stalls early:
    claude:
    scaling:
    large:
    no_progress_window: 75
    (75 is the default for the large tier; use a smaller value to catch stalls sooner.)
  • Write well-scoped issues with clear acceptance criteria — ambiguous issues are the primary driver of timeout loops
  • The self-healing auto-unblock counter is a lifetime cap per issue, not a sliding window; once max_auto_unblocks_per_issue is exceeded, the issue stops being auto-retried and escalates to needs-human for an operator to break the loop manually

Relevant code: packages/developer/src/executor.ts (turn limit logic), packages/sprint-master/src/self-healing.ts (autoUnblockRecoverableIssues() — lifetime auto-unblock cap and needs-human escalation), packages/sprint-master/src/slash-commands/handlers/cancel.ts (handleCancel()) and packages/sprint-master/src/slash-commands/handlers/decompose.ts (handleDecompose()), docs/user-guide/configuration.md (claude.scaling)


Primary (common case): Issue blocked with block_reason=ci_hard_failure:

  • colony why <N> reports ci_hard_failure as the block reason
  • colony issues --blocked lists the issue
  • PR has a FAILURE CI check that won’t recover without a code or infra fix
  • A single bot comment on the PR identifies the failing check(s) and the HEAD SHA they were detected on

Secondary (rare — orphaned claim): A stale or dead worker that held a merge task claim before the failure classification landed:

  • colony workers shows stale or dead freshness for the worker last seen on this issue
  • No new tasks are being created for the issue despite the block not being set

When the merger detects a definitive CI failure, compareCiChecks() (packages/merger/src/ci-comparison.ts) classifies the failure as PR-introduced (not pre-existing on the base branch). Before reporting a PR-introduced failure, it reruns the failed checks up to review.ci_flaky_reruns times (default 1; 0 disables, and code hosts without rerun support skip it); checks that pass on rerun are recorded as flakes, annotated as passed on rerun (flaky), and do not trigger ci_hard_failure. Pre-existing failures are not rerun. The executor then returns block_reason: 'ci_hard_failure', which causes the worker to block the issue rather than re-enqueue. The status comment is SHA-scoped via postCommentOnce({ kind, key: newHeadSha }), so it is posted once per HEAD SHA and not repeated until the author pushes a new commit.

This is the resolved instance of the cross-cutting pattern “every retry path needs an explicit terminal classification” (see production-learnings § Cross-Cutting Reliability Patterns).

A cancelled check conclusion counts as incomplete, not failed. It never triggers ci_hard_failure or CI repair, and it never counts as green, so repositories may use cancel-in-progress concurrency. A cancelled required check is treated like a run still in progress: Colony waits until it is re-run or superseded by a newer commit, or the CI poll times out.

Get a quick read on the block reason:

Terminal window
colony why <N>

A ci_hard_failure block reason confirms the merger detected a definitive CI failure and blocked the issue. Check which issues are currently blocked:

Terminal window
colony issues --blocked
SQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.block_reason, r.owner, r.name
FROM pipeline_issues pi
JOIN repos r ON r.id = pi.repo_id
WHERE pi.is_blocked = true;

Check which CI checks are failing on the PR to identify what needs fixing:

Terminal window
gh pr checks <pr-number> --repo <owner>/<repo>

For the orphaned-claim sub-case, check worker liveness:

Terminal window
colony workers

If the worker’s freshness shows stale or dead, the work_tasks claim is held by a dead worker — the issue will not make progress until the claim is released.

Check the recent task history to confirm the issue is blocked and not looping:

Terminal window
colony tasks --issue <N>
SQL fallback (no CLI access)
SELECT id, task_type, status, claimed_by, created_at, completed_at
FROM work_tasks
WHERE issue_number = <N>
AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')
ORDER BY created_at DESC
LIMIT 15;

Scenario A — stale or dead worker (orphaned claim):

If colony workers shows stale or dead freshness for the worker that last held a task on this issue, free the lease:

Terminal window
colony workers reclaim <workerId>

This returns the claimed task to pending so any available worker can pick it up. On success:

Reclaimed <taskType> task for issue #<N> (task <taskId>) from worker <workerId>

Scenario B — issue blocked with block_reason=ci_hard_failure:

  1. Fix the failing CI check (push a code fix, repair the CI workflow, rotate a broken secret — whatever the failing check requires).

  2. Push the fix to the PR branch.

  3. Comment /colony:retry on the GitHub issue:

    /colony:retry

    This clears the block and re-enqueues the merge task. The merger will re-run compareCiChecks() against the new HEAD SHA; if all required checks now pass, it proceeds to merge.

No manual loop-breaking is needed — the block state prevents re-enqueueing until the retry is explicitly requested.

  • Triage flaky tests on main aggressively — a flake that lands on main causes hard failures on every open PR branch until fixed or worked around.
  • For repos with custom CI jobs (Cloudflare Pages deploys, content validation, security scans), keep review.checks config aligned with the CI workflow’s check-run names so the reviewer catches breakage locally before the merger sees it.

Relevant code: packages/merger/src/ci-comparison.ts (compareCiChecks() — FAILURE vs transient distinction), packages/merger/src/executor.ts (the ciResult.prIntroduced branch that returns blockReason: 'ci_hard_failure', with SHA-scoped postCommentOnce dedup).


7. Stale Branch Failing CI After main Moved

Section titled “7. Stale Branch Failing CI After main Moved”
  • A PR’s CI was passing yesterday and is failing today, with no changes pushed in between
  • The failing check points at a file that isn’t in the branch’s diff (e.g., .gitmodules, package-lock.json, a CI workflow YAML, an SSH/secrets setup step)
  • Multiple unrelated PRs all fail the same way at roughly the same time
  • The error often mentions submodules, missing dependencies, secret/auth setup, or workflow steps the branch never touched

A structural change landed on main — submodule schema change, lockfile bump, CI workflow change, secret rotation — that every open branch must absorb before its CI can pass again. Branches forked before that commit don’t yet have the new shape; their CI re-runs against a checkout that is internally inconsistent (e.g., a gitlink with no matching .gitmodules entry, or a workflow expecting a secret only the new main knows how to fetch).

This is the operator-visible surface of the cross-cutting pattern “trunk drift has O(open branches) blast radius” (see production-learnings § Cross-Cutting Reliability Patterns). The pattern shows up across pipeline-initiated work because there are typically 10+ branches open at any time, all individually exposed to the same main advance.

Confirm the failing check references a file the branch doesn’t touch:

Terminal window
gh pr diff <pr-number> --repo <owner>/<repo> --name-only
gh run view <run-id> --repo <owner>/<repo> --log-failed | head -100

If the failed step references a file not in the diff (commonly .gitmodules, .github/workflows/*.yml, package-lock.json, ssh config, secret-fetch steps), the breakage is on main, not in the branch.

Cross-check by looking at recent landings on main for structural changes:

Terminal window
git log --oneline origin/main -20 -- .gitmodules .github/workflows/ package-lock.json

A recent commit touching any of these is almost certainly the trigger.

Merge main into the branch and resolve any conflicts that surface:

Terminal window
git checkout <branch>
git fetch origin main
git merge origin/main
# resolve conflicts (commonly tests with expanded mock chains, submodule pointers)
git push origin <branch>

If the branch is one you opened manually, this is straightforward. If the branch is owned by Colony (a feature branch or epic branch), merging main in is still safe — push the merge commit and the next pipeline cycle will re-run CI with the absorbed change.

If the branch is far behind and conflicts are unrelated to the feature, consider abandoning the branch and re-creating from the current main (the issue may transition back to ready-for-dev to regenerate the work).

  • After landing a structural change on main (submodule add/remove, lockfile bump, CI workflow rewrite, secret rotation), expect every open PR to need a main merge — plan a sweep, not one-by-one repair.
  • Auto-merge-main-into-open-branches when main advances is the structural fix (tracked as engineering work; until then, treat post-structural-change as an “all branches need touchups” event).
  • Distinguish “branch CI red on files not in diff” (stale-vs-main) from “branch CI red on files in diff” (feature bug) in operator triage — they need opposite responses (merge-main vs. fix-the-code).

Relevant code: packages/merger/src/ (where merge-main automation would live), .github/workflows/ (the workflow files most often involved in trunk-drift cascades).


8. Git Authentication Failed During Workspace Setup

Section titled “8. Git Authentication Failed During Workspace Setup”
  • Worker logs show Git authentication failed for <owner>/<repo> with no code changes attempted
  • The issue never leaves its current state — no analysis, no development, no review activity
  • work_tasks shows a failed task of any type, with the failure occurring at workspace-setup time
  • Raw git output visible in logs: Authentication failed, Invalid username or token, Password authentication is not supported for Git operations, or could not read Username

Colony’s workspace manager calls git clone or git fetch using an HTTPS remote URL. GitHub requires a valid credential (Personal Access Token or GitHub App installation token) embedded in the URL or supplied by a git credential helper. The failure occurs when:

  • GITHUB_TOKEN (or the env var named by github.token_env) is missing from the worker’s environment
  • The token has expired (fine-grained PATs have a mandatory expiry; classic PATs can expire)
  • The token lacks the required scopes: repo for classic PATs, or Contents + Issues + Pull-requests read-write (plus Checks, Commit statuses, Actions read) for fine-grained PATs
  • The git credential helper is configured to send a username/password instead of a token (GitHub disabled password auth for git operations in 2021)
  • A GitHub App’s installation token was not minted correctly for the target repository

Run the Colony credential check:

Terminal window
colony check --stage credentials

This validates that the configured token is present, well-formed, and can authenticate to the GitHub API. It does not test git-over-HTTPS directly, but a passing credential check combined with a passing repository check (colony check --stage repository) covers the most common causes.

For container deployments, confirm the token is visible inside the worker container:

Terminal window
docker compose exec worker printenv GITHUB_TOKEN

If this prints nothing, the .env file is missing or the compose file does not forward the variable.

Check worker logs for the raw git error:

Terminal window
docker compose logs worker | grep -i "authentication\|username\|token"

Confirm the git remote URL includes the token (the workspace manager should inject x-access-token:<TOKEN>@github.com):

Terminal window
docker compose exec worker git -C .colony/workspaces/issue-<N> remote get-url origin

If the URL shows https://github.com/... without credentials, the token injection step failed.

  1. Rotate or re-export the token:

    Terminal window
    # Edit your .env file and set the correct token value
    GITHUB_TOKEN=ghp_...
  2. Restart the worker containers so they pick up the new environment:

    Terminal window
    docker compose up -d worker

    Workers read environment variables at startup — a running container does not see changes to .env until it is restarted.

  3. Verify the fix:

    Terminal window
    colony check --stage credentials
  4. Retry the blocked issue:

    /colony:retry

    Comment this on the GitHub issue to re-enqueue it. Because the failure was environmental (no code changes were attempted), no analysis or re-decomposition is needed.

  • Token rotation reminders: Set calendar reminders for PAT expiry dates. Fine-grained PATs have a maximum 366-day lifetime; classic PATs default to no expiry but should be rotated on a schedule.
  • GitHub App tokens: Consider switching to a GitHub App for zero-manual-expiry installation tokens. The App’s installation token is minted per-request by Colony’s auth layer (packages/core/src/github-auth.ts) and does not need manual rotation.
  • Credential helper pattern: Ensure the git remote URL uses the https://x-access-token:<token>@github.com/<owner>/<repo>.git format, not a bare https:// URL that relies on a system credential helper.
  • Run colony check after any token rotation before re-deploying workers to catch scope or expiry issues before they block live issues.

Relevant code: packages/core/src/clone.ts (cloneRepo and cloneRepoAsync — git auth failures are detected and surfaced as a clear error), packages/core/src/github-auth.ts (token injection for GitHub App), docs/user-guide/configuration.md (github.token_env)


9. Runtime Not Found in bash -lc setup_command or review.checks

Section titled “9. Runtime Not Found in bash -lc setup_command or review.checks”
  • python: command not found, node: command not found, ruby: command not found, or similar errors during workspace setup or review checks
  • The failure occurs only when the command is wrapped in a login shell (bash -lc '...') — the same command via bash -c '...' succeeds
  • Running docker compose exec worker bash -c 'python --version' works, but docker compose exec worker bash -lc 'python --version' fails

The production image adds language runtimes to PATH via Docker ENV directives (mise shims at /usr/local/share/mise/shims, dotnet tools, mssql-tools18, and workspace bin scripts). These ENV layers are inherited by the worker process and any non-login child shells it spawns.

When a login shell is invoked (bash -l or bash -lc), Bash sources /etc/profile. On Debian, /etc/profile resets PATH to a fixed minimal default that does not include the image’s custom directories. Login shells therefore cannot find mise-managed runtimes like Python, Node, or Ruby — even though those runtimes are installed and resolve fine in the worker’s own (non-login) environment.

Related but distinct: task-installed CLI entry points. The image also puts the mise installs/<runtime>/<version>/bin directories for the pinned Python (3.12, 3.11) and Node (20, 22) versions on PATH, alongside the shims. This lets a CLI entry point installed at task runtime — pip install, npm install -g — resolve by name immediately, in both login and non-login shells, without waiting for a mise reshim. A repo that pins a Python or Node version other than the four listed above (via .python-version, .node-version, or .tool-versions) is not covered by this static list: a task-time install under that version’s bin directory will still produce command not found immediately after the install reports success, since only the versions known at image build time are on PATH.

Confirm the symptom is login-shell-specific by running both variants inside the container:

Terminal window
# This should succeed (non-login shell inherits worker PATH)
docker compose exec worker bash -c 'python --version && node --version'
# This fails without the fix (login shell resets PATH)
docker compose exec worker bash -lc 'python --version && node --version'

If the non-login form succeeds and the login form fails, this is the issue. Check whether setup_command or a review.checks entry in your config wraps the command in bash -lc '...'.

Option A — use a non-login shell wrapper (recommended):

Replace bash -lc '...' with bash -c '...' in your setup_command or review.checks config. Non-login shells inherit the worker’s PATH and resolve all mise-managed runtimes without any extra configuration:

# Before (broken)
workspace:
setup_command: "bash -lc 'bundle install && rails db:migrate'"
# After (fixed)
workspace:
setup_command: "bash -c 'bundle install && rails db:migrate'"
# Or just omit the shell wrapper entirely for single commands
workspace:
setup_command: "bundle install"

Option B — upgrade to an image that includes /etc/profile.d/colony-path.sh:

Production images built after this fix include /etc/profile.d/colony-path.sh, which re-prepends the custom PATH segments after /etc/profile resets them. Login shells (bash -lc) in those images resolve mise runtimes identically to non-login shells. No config change is needed — rebuild or pull the updated image. Note that colony-path.sh no longer adds /colony/node_modules/.bin, so login shells do not see Colony’s CLI or tools; use bash -c or the full path /colony/node_modules/.bin/colony.

Verify the file is present in your image:

Terminal window
docker compose exec worker cat /etc/profile.d/colony-path.sh
  • Prefer plain commands or bash -c '...' wrappers in setup_command and review.checks — non-login shells always inherit the worker’s PATH.
  • Avoid bash -lc in these config values; it provides no benefit over bash -c in a container environment where .bashrc and /etc/profile.d/ are not customized for interactive use.

Relevant code: Dockerfile (/etc/profile.d/colony-path.sh RUN step, ENV PATH=... layers), packages/core/src/workspace.ts (execSync calls for setup_command), packages/reviewer/src/checks.ts (execSync calls for review.checks)


  • The reviewer comment says a check failed and then passed when re-run on the PR tree
  • The reviewer comment says a check is quarantined
  • Before flaky detection, the developer agent was repeatedly asked to fix code that wasn’t broken, burning review cycles on an intermittent test

How Colony handles it:

  • A failed blocking check is re-run on the PR’s own tree up to review.flaky_reruns times (default 1) before the base-branch comparison.
  • A passing rerun marks the check flaky. This is advisory and does not request changes.
  • Every attempt is recorded in the review_check_runs table.
  • A check with at least review.flaky_quarantine_threshold flakes in the last 14 days is quarantined. Its failures are advisory, and the review comment says so.
  • A quarantined check is still re-run on failure, so its flakes keep being recorded. Quarantine lifts only once the check stops flaking for long enough that its 14-day flake count falls below the threshold.
  • Quarantine only applies to the reviewer. Developer self-validation and merger conflict-resolution checks still treat the check as blocking.
  • review_check_runs rows older than 90 days are pruned by the sprint-master.
  • Open the Flaky checks panel on the dashboard’s Review Queue tab (see dashboard.md).
  • Query the colony_review_check_flake_rate and colony_review_check_flakes Prometheus gauges (see metrics.md).
  • Run colony metrics and read the Flaky checks section (see cli-reference.md).
  • Call the colony_flaky_checks MCP tool (see mcp.md).
  • Fix or stabilise the test in the target repo (timeouts, ordering, shared state).
  • Set review.flaky_reruns: 0 to turn off reruns.
  • Raise or unset review.flaky_quarantine_threshold to stop quarantining.

Quarantine is advisory, so real regressions in a quarantined check won’t block review until its flake count drops back under the threshold.

  • Fix flaky tests promptly rather than relying on quarantine.
  • Keep flaky_quarantine_threshold high enough that a real regression isn’t masked by a few unrelated flakes.

A failing CI check (the GitHub check runs on the PR, not the reviewer’s local checks) is also rerun once on the same head before the reviewer routes back:

  • The reviewer asks the code host to rerun the failed check runs and waits up to review.ci_check_timeout for the new results. If the PR head moved in the meantime, no rerun happens.
  • If every failed check passes on rerun, the reviewer treats CI as passed, records the flake (source ci) and posts a short flaky-CI notice. If a check fails again, the issue goes back to the developer as before.
  • The route-back comment keeps its first line and adds one line per check, e.g. `Database-backed tests` — failed again on rerun; flaked 7 times in the last 14 days (see review.ci_flaky_reruns). The count is the number of CI flakes in the rolling 14-day window; a high count suggests the failure is the check, not the change. The same count is appended to the matching action item’s required action (the finding title stays stable).
  • Rerunning needs actions:write on the token. Without it, or on a code host that cannot rerun checks, the reviewer routes back immediately.
  • Set review.ci_flaky_reruns: 0 to disable CI reruns.

See the review section of configuration.md for the config keys.