Top Failure Modes
Top Failure Modes
Section titled “Top Failure Modes”Operator-facing diagnosis and fix guides for the most common Colony failure modes. Each section covers symptoms, root cause, diagnosis steps, fix, and prevention.
Postgres is the authority for pipeline state; GitHub labels are a projection for human visibility. Always check Postgres first when diagnosing pipeline issues.
For the underlying engineering patterns that explain many of the specific symptoms here (retry-without-terminal-classification, silent-fallback at trust boundaries, operator-surface inconsistency), see docs/production-learnings.md § Cross-Cutting Reliability Patterns. When a symptom doesn’t match any single entry below, it’s often an instance of one of those patterns.
Quick Decision Tree
Section titled “Quick Decision Tree”Setup issues? → colony check --fix → confirm prompts → doneIssue not picked up? → colony status → colony check --stage runtime → check intake_modeAgent won't start? → colony check --fix → check .colony/logs/<agent>.logIssue stuck in a state? → colony why <N> → colony tasks --issue <N>Worker stuck / one worker pinned? → colony workers → colony workers reclaim <workerId>Merger blocked on CI-red PR? → colony why <N> → fix failing check → push → /colony:retryBranch CI red on files diff doesn't touch? → branch is stale; merge `main` in (or open a fresh branch)Config confusion? → colony check --stage config → review effective configWorker logs show "Git authentication failed"? → colony check --stage credentials → rotate GITHUB_TOKEN1. Stale is_blocked Flag
Section titled “1. Stale is_blocked Flag”Symptoms
Section titled “Symptoms”- Issue stuck in a state with no worker activity
work_taskstable has no pending tasks for the issue- Dashboard shows the issue as blocked, but there is no obvious reason
Root Cause
Section titled “Root Cause”is_blocked was set by a transient failure (e.g., push conflict, OOM, subprocess timeout) and never cleared. The sprint-master’s auto_unblock_transient may be disabled or the issue has exceeded max_auto_unblocks_per_issue (default: 3).
Diagnosis Steps
Section titled “Diagnosis Steps”Get a quick read on why the issue is blocked:
colony why <N>Check which issues are blocked:
colony issues --blockedSQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.is_paused, r.owner, r.nameFROM pipeline_issues piJOIN repos r ON r.id = pi.repo_idWHERE pi.is_blocked = true;Confirm worker liveness — verify a worker is registered, healthy, and not pinned to a different issue:
colony workersCheck whether there are any pending tasks for the issue:
colony tasks --issue <N>SQL fallback (no CLI access)
SELECT id, task_type, status, created_atFROM work_tasksWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')ORDER BY created_at DESCLIMIT 10;Check the monitor logs for auto-unblock activity — the self-healing module logs when it unblocks issues and when it skips issues that have exceeded the unblock cap.
Run the unblock CLI command:
colony unblock <N>This clears the is_blocked flag, removes the colony:blocked label, and re-queues the issue in one step.
Alternatively, comment /colony:retry on the GitHub issue to retry with current settings.
If you do not have CLI access, clear the flag manually with SQL:
UPDATE pipeline_issuesSET is_blocked = falseWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>');Prevention
Section titled “Prevention”Enable automatic unblocking of transient failures in your config:
agents: sprint_master: auto_unblock_transient: true max_auto_unblocks_per_issue: 5The sprint-master classifies blocking reasons by analyzing the most recent Colony bot comment on the issue. If the reason is transient and the responsible agent is healthy, it automatically clears the is_blocked flag and transitions the issue to a retry state.
Relevant code: packages/pipeline-store/src/pipeline-store.ts (is_blocked column), packages/sprint-master/src/self-healing.ts (auto-unblock logic)
2. Head Branch Out of Date on Merge
Section titled “2. Head Branch Out of Date on Merge”Symptoms
Section titled “Symptoms”- Merge fails — merger logs show rebase failures or drift assessment
- Multiple PRs targeting main at the same time
work_taskstable shows failedmergetasks
Root Cause
Section titled “Root Cause”Concurrent PRs — when PR A merges into main, PR B’s branch is stale. The merger attempts to rebase PR B onto the updated main branch. If the rebase produces conflicts, the merger runs a drift assessment to determine whether the conflicts are resolvable.
Diagnosis Steps
Section titled “Diagnosis Steps”Check for failed merge tasks:
colony tasks --type merge --status failedSQL fallback (no CLI access)
SELECT id, issue_number, status, created_at, updated_atFROM work_tasksWHERE task_type = 'merge' AND status = 'failed' AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')ORDER BY created_at DESCLIMIT 10;Check merger logs for rebase failure indicators:
"Rebase failed"— initial rebase attempt failed"Rebase failed, running drift assessment"— merger is evaluating conflict severity"Unable to rebase automatically"— escalation marker indicating conflicts require intervention
The merger computes a “drift overlap” metric (percentage of line overlap between conflicting code) to assess whether auto-resolution is feasible.
The sprint-master automatically re-enqueues merge tasks, and the merger retries the rebase after the conflicting PR has merged. In most cases, the retry succeeds without intervention.
If the issue is stuck:
- Comment
/colony:retryon the GitHub issue to re-enqueue the merge task - For complex conflicts, the merger may escalate with a comment containing conflict details, drift estimates, and a recommendation (manual rebase vs. re-implementation)
- As a last resort, manually rebase the branch:
git rebase origin/mainand force-push
Prevention
Section titled “Prevention”review.rebase_before_check: true(default) ensures the branch is rebased before review checks run, reducing stale-branch scenarios at merge time- The merger has built-in retry logic with drift assessment — most concurrent-PR conflicts resolve automatically on the next attempt
- For repos with high PR throughput, ensure the sprint-master poll interval is short enough to quickly re-enqueue failed merge tasks
Relevant code: packages/merger/src/executor.ts (rebase and drift assessment logic), packages/sprint-master/src/sprint-master.ts (task re-enqueue)
3. Label Projection Failures
Section titled “3. Label Projection Failures”Symptoms
Section titled “Symptoms”- GitHub issue shows the wrong or missing
colony:label - Pipeline is actually progressing — dashboard or Postgres shows the correct state
- Sprint-master logs show label sync errors (rate limit, network timeout)
Root Cause
Section titled “Root Cause”GitHub API rate limit or transient network error during label sync. Labels are a write-only projection of Postgres state, not the source of truth. A failed label update does not affect pipeline processing.
Important: Manually adding or removing colony: labels on GitHub does NOT affect pipeline state. The pipeline reads state exclusively from Postgres. The only exceptions are two supported label commands: colony:enqueue (seeds a new issue into the pipeline) and colony:paused (pauses or resumes a running issue). For all other pipeline control, use slash commands (e.g., /colony:retry, /colony:state <target>) rather than label manipulation.
Diagnosis Steps
Section titled “Diagnosis Steps”Check for issues where Postgres state and stored labels have diverged:
colony issues --label-driftSQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.is_paused, pi.labelsFROM pipeline_issues piWHERE pi.issue_number = <N> AND pi.repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>');Compare the state column (authoritative) with the labels array (projection). If they diverge, the label sync will auto-correct on the next poll cycle.
Check sprint-master logs for label sync errors — the syncLabelsFromPostgres() function runs on every poll cycle and reconciles managed labels (state labels, colony:blocked, colony:paused) against GitHub.
Wait one poll cycle — syncLabelsFromPostgres() auto-corrects within the sprint-master’s poll interval (default 30s). The function compares expected labels (derived from Postgres state and is_blocked/is_paused flags) against actual GitHub labels and issues the necessary add/remove calls.
For immediate correction, use the CLI:
npx colony issue transition <issue-number> --state <state>Prevention
Section titled “Prevention”This is cosmetic — no action needed. Postgres state is authoritative. The sprint-master’s label sync is self-healing by design and catches up automatically. The label sync processes up to 25 issues per cycle (configurable via label_sync_limit).
Relevant code: packages/sprint-master/src/label-sync.ts (syncLabelsFromPostgres()), packages/core/src/state-transition.ts
4. Worker OOM During npm install
Section titled “4. Worker OOM During npm install”Symptoms
Section titled “Symptoms”- Worker container crashes or is killed during workspace setup
- Container logs show
KilledorOOMKilled - The issue gets blocked after repeated setup failures
Root Cause
Section titled “Root Cause”Default container memory limit is too low for large node_modules trees. npm install can spike memory significantly for repos with many dependencies.
Diagnosis Steps
Section titled “Diagnosis Steps”Check if the container was OOM-killed:
docker inspect <container> | grep OOMKilledOr check container logs:
docker compose logs worker | grep -i killedCheck worker health to confirm whether the container restarted or is still running:
colony workersA dead or stale freshness reading for the worker that was processing the issue confirms it was killed or lost its heartbeat around the time of the failure.
Check the work task failure history for the affected issue:
colony tasks --issue <N> --status failedSQL fallback (no CLI access)
SELECT id, task_type, status, created_atFROM work_tasksWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>') AND status = 'failed'ORDER BY created_at DESCLIMIT 10;Increase the worker memory limit in your config:
repos: - owner: my-org name: my-repo workers: memory: '6g'For very large repos (thousands of dependencies), use '8g' or higher. After updating the config, rebuild and restart the worker containers.
Prevention
Section titled “Prevention”- Set
repos[].workers.memorybased on the target repo’s dependency tree size - Monitor container memory usage during initial workspace setup to establish a baseline
- If using a custom
workspace.setup_command(e.g.,bundle installfor Ruby), the same memory considerations apply
Relevant code: docs/user-guide/configuration.md (repos[].workers.memory field)
5. Planning Timeout Loops
Section titled “5. Planning Timeout Loops”Symptoms
Section titled “Symptoms”- Issue stuck in
ready-for-devwith a high turn count - Multiple failed
developtasks inwork_tasks— sprint-master keeps re-enqueueing - Developer logs show max turns being hit (look for
maxTurnsin structured log output)
Root Cause
Section titled “Root Cause”The issue is too complex or ambiguous for the configured turn limit. The developer exhausts developer_max_turns without completing the task, gets blocked, and the sprint-master retries — creating a loop.
Diagnosis Steps
Section titled “Diagnosis Steps”Use colony why for a quick diagnosis of why the issue is stuck:
colony why <N>Confirm workers are running and not stuck on a different issue before examining task history:
colony workersCheck for repeated develop task failures:
colony tasks --issue <N>SQL fallback (no CLI access)
SELECT id, task_type, status, created_at, updated_atFROM work_tasksWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')ORDER BY created_at DESCLIMIT 10;Check developer logs for max turns indicators:
- The developer executor logs
maxTurnsin its structured output when starting development - When the turn limit is hit, the result includes
isMaxTurns: true - The failure tracker (
packages/core/src/failure-tracker.ts) counts consecutive failures per issue
Stop the retry loop:
/colony:cancelComment this on the GitHub issue to close it and stop re-enqueue.
Alternatively, decompose the issue into smaller sub-issues:
/colony:decomposeThis sends the issue to the planner, which breaks it into smaller, more tractable sub-issues.
If the issue is close to completion and just needs more turns, bump the limits:
claude: scaling: large: developer_max_turns: 50Complexity tiers (small, medium, large) each have their own developer_max_turns setting.
Prevention
Section titled “Prevention”- Use the planner for large issues — comment
/colony:decomposebefore the issue enters development - Configure progress detection windows to catch stalls early:
claude:scaling:large:no_progress_window: 75
- Write well-scoped issues with clear acceptance criteria — ambiguous issues are the primary driver of timeout loops
- The failure tracker counts consecutive failures per issue key; after the threshold is exceeded, the issue is blocked to prevent unbounded retries
Relevant code: packages/developer/src/executor.ts (turn limit logic), packages/core/src/state-transition.ts (slash commands), packages/core/src/failure-tracker.ts (failure counting), docs/user-guide/configuration.md (claude.scaling)
6. PR Blocked by Hard CI Failure
Section titled “6. PR Blocked by Hard CI Failure”Symptoms
Section titled “Symptoms”Primary (common case): Issue blocked with block_reason=ci_hard_failure:
colony why <N>reportsci_hard_failureas the block reasoncolony issues --blockedlists the issue- PR has a
FAILURECI check that won’t recover without a code or infra fix - A single bot comment on the PR identifies the failing check(s) and the HEAD SHA they were detected on
Secondary (rare — orphaned claim): A stale or dead worker that held a merge task claim before the failure classification landed:
colony workersshowsstaleordeadfreshness for the worker last seen on this issue- No new tasks are being created for the issue despite the block not being set
Root Cause
Section titled “Root Cause”When the merger detects a definitive CI failure, compareCiChecks() (packages/merger/src/ci-comparison.ts) classifies the failure as PR-introduced (not pre-existing on the base branch). The executor then returns block_reason: 'ci_hard_failure', which causes the worker to block the issue rather than re-enqueue. The status comment is SHA-scoped via postCommentOnce({ kind, key: headSha }), so it is posted once per HEAD SHA and not repeated until the author pushes a new commit.
This is the resolved instance of the cross-cutting pattern “every retry path needs an explicit terminal classification” (see production-learnings § Cross-Cutting Reliability Patterns).
Diagnosis Steps
Section titled “Diagnosis Steps”Get a quick read on the block reason:
colony why <N>A ci_hard_failure block reason confirms the merger detected a definitive CI failure and blocked the issue. Check which issues are currently blocked:
colony issues --blockedSQL fallback (no CLI access)
SELECT pi.issue_number, pi.state, pi.is_blocked, pi.block_reason, r.owner, r.nameFROM pipeline_issues piJOIN repos r ON r.id = pi.repo_idWHERE pi.is_blocked = true;Check which CI checks are failing on the PR to identify what needs fixing:
gh pr checks <pr-number> --repo <owner>/<repo>For the orphaned-claim sub-case, check worker liveness:
colony workersIf the worker’s freshness shows stale or dead, the work_tasks claim is held by a dead worker — the issue will not make progress until the claim is released.
Check the recent task history to confirm the issue is blocked and not looping:
colony tasks --issue <N>SQL fallback (no CLI access)
SELECT id, task_type, status, claimed_by, created_at, completed_atFROM work_tasksWHERE issue_number = <N> AND repo_id = (SELECT id FROM repos WHERE owner = '<owner>' AND name = '<repo>')ORDER BY created_at DESCLIMIT 15;Scenario A — stale or dead worker (orphaned claim):
If colony workers shows stale or dead freshness for the worker that last held a task on this issue, free the lease:
colony workers reclaim <workerId>This returns the claimed task to pending so any available worker can pick it up. On success:
Reclaimed <taskType> task for issue #<N> (task <taskId>) from worker <workerId>Scenario B — issue blocked with block_reason=ci_hard_failure:
-
Fix the failing CI check (push a code fix, repair the CI workflow, rotate a broken secret — whatever the failing check requires).
-
Push the fix to the PR branch.
-
Comment
/colony:retryon the GitHub issue:/colony:retryThis clears the block and re-enqueues the merge task. The merger will re-run
compareCiChecks()against the new HEAD SHA; if all required checks now pass, it proceeds to merge.
No manual loop-breaking is needed — the block state prevents re-enqueueing until the retry is explicitly requested.
Prevention
Section titled “Prevention”- Triage flaky tests on
mainaggressively — a flake that lands onmaincauses hard failures on every open PR branch until fixed or worked around. - For repos with custom CI jobs (Cloudflare Pages deploys, content validation, security scans), keep
review.checksconfig aligned with the CI workflow’s check-run names so the reviewer catches breakage locally before the merger sees it.
Relevant code: packages/merger/src/ci-comparison.ts (compareCiChecks() — FAILURE vs transient distinction), packages/merger/src/executor.ts (lines 1276–1318, ci_hard_failure block path and SHA-scoped postCommentOnce dedup).
7. Stale Branch Failing CI After main Moved
Section titled “7. Stale Branch Failing CI After main Moved”Symptoms
Section titled “Symptoms”- A PR’s CI was passing yesterday and is failing today, with no changes pushed in between
- The failing check points at a file that isn’t in the branch’s diff (e.g.,
.gitmodules,package-lock.json, a CI workflow YAML, an SSH/secrets setup step) - Multiple unrelated PRs all fail the same way at roughly the same time
- The error often mentions submodules, missing dependencies, secret/auth setup, or workflow steps the branch never touched
Root Cause
Section titled “Root Cause”A structural change landed on main — submodule schema change, lockfile bump, CI workflow change, secret rotation — that every open branch must absorb before its CI can pass again. Branches forked before that commit don’t yet have the new shape; their CI re-runs against a checkout that is internally inconsistent (e.g., a gitlink with no matching .gitmodules entry, or a workflow expecting a secret only the new main knows how to fetch).
This is the operator-visible surface of the cross-cutting pattern “trunk drift has O(open branches) blast radius” (see production-learnings § Cross-Cutting Reliability Patterns). The pattern shows up across pipeline-initiated work because there are typically 10+ branches open at any time, all individually exposed to the same main advance.
Diagnosis Steps
Section titled “Diagnosis Steps”Confirm the failing check references a file the branch doesn’t touch:
gh pr diff <pr-number> --repo <owner>/<repo> --name-onlygh run view <run-id> --repo <owner>/<repo> --log-failed | head -100If the failed step references a file not in the diff (commonly .gitmodules, .github/workflows/*.yml, package-lock.json, ssh config, secret-fetch steps), the breakage is on main, not in the branch.
Cross-check by looking at recent landings on main for structural changes:
git log --oneline origin/main -20 -- .gitmodules .github/workflows/ package-lock.jsonA recent commit touching any of these is almost certainly the trigger.
Merge main into the branch and resolve any conflicts that surface:
git checkout <branch>git fetch origin maingit merge origin/main# resolve conflicts (commonly tests with expanded mock chains, submodule pointers)git push origin <branch>If the branch is one you opened manually, this is straightforward. If the branch is owned by Colony (a feature branch or epic branch), merging main in is still safe — push the merge commit and the next pipeline cycle will re-run CI with the absorbed change.
If the branch is far behind and conflicts are unrelated to the feature, consider abandoning the branch and re-creating from the current main (the issue may transition back to ready-for-dev to regenerate the work).
Prevention
Section titled “Prevention”- After landing a structural change on
main(submodule add/remove, lockfile bump, CI workflow rewrite, secret rotation), expect every open PR to need amainmerge — plan a sweep, not one-by-one repair. - Auto-merge-
main-into-open-branches whenmainadvances is the structural fix (tracked as engineering work; until then, treat post-structural-change as an “all branches need touchups” event). - Distinguish “branch CI red on files not in diff” (stale-vs-main) from “branch CI red on files in diff” (feature bug) in operator triage — they need opposite responses (merge-main vs. fix-the-code).
Relevant code: packages/merger/src/ (where merge-main automation would live), .github/workflows/ (the workflow files most often involved in trunk-drift cascades).
8. Git Authentication Failed During Workspace Setup {#git-authentication-failed-during-workspace-setup}
Section titled “8. Git Authentication Failed During Workspace Setup {#git-authentication-failed-during-workspace-setup}”Symptoms
Section titled “Symptoms”- Worker logs show
Git authentication failed for <owner>/<repo>with no code changes attempted - The issue never leaves its current state — no analysis, no development, no review activity
work_tasksshows a failed task of any type, with the failure occurring at workspace-setup time- Raw git output visible in logs:
Authentication failed,Invalid username or token,Password authentication is not supported for Git operations, orcould not read Username
Root Cause
Section titled “Root Cause”Colony’s workspace manager calls git clone or git fetch using an HTTPS remote URL. GitHub requires a valid credential (Personal Access Token or GitHub App installation token) embedded in the URL or supplied by a git credential helper. The failure occurs when:
GITHUB_TOKEN(or the env var named bygithub.token_env) is missing from the worker’s environment- The token has expired (fine-grained PATs have a mandatory expiry; classic PATs can expire)
- The token lacks the required scopes:
repofor classic PATs, or Contents + Issues + Pull-requests read-write for fine-grained PATs - The git credential helper is configured to send a username/password instead of a token (GitHub disabled password auth for git operations in 2021)
- A GitHub App’s installation token was not minted correctly for the target repository
Diagnosis Steps
Section titled “Diagnosis Steps”Run the Colony credential check:
colony check --stage credentialsThis validates that the configured token is present, well-formed, and can authenticate to the GitHub API. It does not test git-over-HTTPS directly, but a passing credential check combined with a passing repository check (colony check --stage repository) covers the most common causes.
For container deployments, confirm the token is visible inside the worker container:
docker compose exec worker printenv GITHUB_TOKENIf this prints nothing, the .env file is missing or the compose file does not forward the variable.
Check worker logs for the raw git error:
docker compose logs worker | grep -i "authentication\|username\|token"Confirm the git remote URL includes the token (the workspace manager should inject x-access-token:<TOKEN>@github.com):
docker compose exec worker git -C .colony/workspaces/issue-<N> remote get-url originIf the URL shows https://github.com/... without credentials, the token injection step failed.
-
Rotate or re-export the token:
Terminal window # Edit your .env file and set the correct token valueGITHUB_TOKEN=ghp_... -
Restart the worker containers so they pick up the new environment:
Terminal window docker compose up -d workerWorkers read environment variables at startup — a running container does not see changes to
.envuntil it is restarted. -
Verify the fix:
Terminal window colony check --stage credentials -
Retry the blocked issue:
/colony:retryComment this on the GitHub issue to re-enqueue it. Because the failure was environmental (no code changes were attempted), no analysis or re-decomposition is needed.
Prevention
Section titled “Prevention”- Token rotation reminders: Set calendar reminders for PAT expiry dates. Fine-grained PATs have a maximum 366-day lifetime; classic PATs default to no expiry but should be rotated on a schedule.
- GitHub App tokens: Consider switching to a GitHub App for zero-manual-expiry installation tokens. The App’s installation token is minted per-request by Colony’s auth layer (
packages/core/src/github-auth.ts) and does not need manual rotation. - Credential helper pattern: Ensure the git remote URL uses the
https://x-access-token:<token>@github.com/<owner>/<repo>.gitformat, not a barehttps://URL that relies on a system credential helper. - Run
colony checkafter any token rotation before re-deploying workers to catch scope or expiry issues before they block live issues.
Relevant code: packages/core/src/clone.ts (cloneRepo, isGitAuthFailure, makeGitAuthError), packages/core/src/github-auth.ts (token injection for GitHub App), docs/user-guide/configuration.md (github.token_env)
9. Runtime Not Found in bash -lc setup_command or review.checks
Section titled “9. Runtime Not Found in bash -lc setup_command or review.checks”Symptoms
Section titled “Symptoms”python: command not found,node: command not found,ruby: command not found, or similar errors during workspace setup or review checks- The failure occurs only when the command is wrapped in a login shell (
bash -lc '...') — the same command viabash -c '...'succeeds - Running
docker compose exec worker bash -c 'python --version'works, butdocker compose exec worker bash -lc 'python --version'fails
Root Cause
Section titled “Root Cause”The production image adds language runtimes to PATH via Docker ENV directives (mise shims at /usr/local/share/mise/shims, dotnet tools, mssql-tools18, and workspace bin scripts). These ENV layers are inherited by the worker process and any non-login child shells it spawns.
When a login shell is invoked (bash -l or bash -lc), Bash sources /etc/profile. On Debian, /etc/profile resets PATH to a fixed minimal default that does not include the image’s custom directories. Login shells therefore cannot find mise-managed runtimes like Python, Node, or Ruby — even though those runtimes are installed and resolve fine in the worker’s own (non-login) environment.
Related but distinct: task-installed CLI entry points. The image also puts the mise installs/<runtime>/<version>/bin directories for the pinned Python (3.12, 3.11) and Node (20, 22) versions on PATH, alongside the shims. This lets a CLI entry point installed at task runtime — pip install, npm install -g — resolve by name immediately, in both login and non-login shells, without waiting for a mise reshim. A repo that pins a Python or Node version other than the four listed above (via .python-version, .node-version, or .tool-versions) is not covered by this static list: a task-time install under that version’s bin directory will still produce command not found immediately after the install reports success, since only the versions known at image build time are on PATH.
Diagnosis Steps
Section titled “Diagnosis Steps”Confirm the symptom is login-shell-specific by running both variants inside the container:
# This should succeed (non-login shell inherits worker PATH)docker compose exec worker bash -c 'python --version && node --version'
# This fails without the fix (login shell resets PATH)docker compose exec worker bash -lc 'python --version && node --version'If the non-login form succeeds and the login form fails, this is the issue. Check whether setup_command or a review.checks entry in your config wraps the command in bash -lc '...'.
Option A — use a non-login shell wrapper (recommended):
Replace bash -lc '...' with bash -c '...' in your setup_command or review.checks config. Non-login shells inherit the worker’s PATH and resolve all mise-managed runtimes without any extra configuration:
# Before (broken)workspace: setup_command: "bash -lc 'bundle install && rails db:migrate'"
# After (fixed)workspace: setup_command: "bash -c 'bundle install && rails db:migrate'"
# Or just omit the shell wrapper entirely for single commandsworkspace: setup_command: "bundle install"Option B — upgrade to an image that includes /etc/profile.d/colony-path.sh:
Production images built after this fix include /etc/profile.d/colony-path.sh, which re-prepends the custom PATH segments after /etc/profile resets them. Login shells (bash -lc) in those images resolve mise runtimes identically to non-login shells. No config change is needed — rebuild or pull the updated image.
Verify the file is present in your image:
docker compose exec worker cat /etc/profile.d/colony-path.shPrevention
Section titled “Prevention”- Prefer plain commands or
bash -c '...'wrappers insetup_commandandreview.checks— non-login shells always inherit the worker’s PATH. - Avoid
bash -lcin these config values; it provides no benefit overbash -cin a container environment where.bashrcand/etc/profile.d/are not customized for interactive use.
Relevant code: Dockerfile (/etc/profile.d/colony-path.sh RUN step, ENV PATH=... layers), packages/core/src/workspace.ts (execSync calls for setup_command), packages/reviewer/src/checks.ts (execSync calls for review.checks)