Skip to content

External PR Review

Colony can review human-authored pull requests — PRs that are not part of its own pipeline — using the same review engine it applies to its own work: deterministic checks (build, test, lint) run in a real worktree, followed by an LLM code review that understands the repo’s conventions.

This is configured via review.external_prs in your Colony config.

Colony never edits the title or body of an external PR. For the review format it posts, see What Colony puts on a pull request.


There are three modes, ordered from safest to most aggressive:

ModeTriggerBest for
on_requestA user with write access comments /colony:review on the PRGetting started; explicit opt-in per PR
on_readyPR is opened or transitions from draft to ready-for-reviewTeams that want automatic review without auto-approval
autoSame as on_ready, plus the auto_approve flag is availableFully automated review pipelines

The feature is off by default (mode: off). You must opt in.


Section titled “Mode 1 — On Request (Recommended Starting Point)”

Enable by setting mode: on_request. Colony only reviews a PR when a human explicitly requests it.

review:
external_prs:
mode: on_request
  1. Open any non-pipeline PR (non-draft, unless exclude_drafts: false) in your repo.
  2. Comment /colony:review on the PR.
  3. Colony adds a 👍 reaction to acknowledge the request.
  4. Colony runs deterministic checks and the LLM review, then posts the results as a GitHub PR review.

Under the default commands.authorization: write, the commenter must have write access to the repo; a commenter without it gets a one-time reply explaining the refusal. Set commands.authorization: any to allow any account (see commands). Bot comments are ignored.

The author_filter, label_filter, exclude_drafts and exclude_bots filters (described under Mode 2) also apply to /colony:review requests; a request on a filtered-out PR is skipped silently.

After you address feedback and push new commits, comment /colony:review again to request a fresh review. Colony will run against the updated head commit and post a new review, dismissing the previous one.


Colony automatically reviews a PR when it is opened (non-draft) or transitions from draft to ready-for-review.

review:
external_prs:
mode: on_ready

The filters in this section and the next apply in every mode other than off, including on_request. A /colony:review request on a filtered-out PR is skipped silently.

Use author_filter to limit automatic reviews to a specific allowlist of GitHub usernames. The default ['*'] means all authors are eligible.

review:
external_prs:
mode: on_ready
author_filter: ['alice', 'bob', 'carol']

Use label_filter to require specific labels on the PR before reviewing. Multiple labels use AND semantics — all listed labels must be present.

review:
external_prs:
mode: on_ready
label_filter: ['review-please']

By default, Colony skips:

  • Draft PRs (exclude_drafts: true) — set to false to review drafts
  • Bot-authored PRs (exclude_bots: true) — set to false to review bot PRs
review:
external_prs:
mode: on_ready
exclude_drafts: false # review drafts too

When a contributor pushes new commits to a PR that Colony has already reviewed, Colony automatically re-reviews the updated PR. This only fires if Colony has previously reviewed that PR — new pushes on PRs that have never been reviewed do not trigger an unsolicited initial review.


Same as Mode 2 in terms of triggers — Colony reviews every eligible PR when it is opened or transitions to ready. The auto_approve config flag is only meaningful with this mode (Colony logs a startup warning if auto_approve: true is set without mode: auto).

review:
external_prs:
mode: auto
auto_approve: true

Note: Colony always posts a Request Changes event when the verdict is REQUEST_CHANGES, regardless of mode or auto_approve. For a passing (APPROVE) verdict, the event depends on auto_approve: with mode: auto and auto_approve: true, Colony posts a formal GitHub Approve review event; otherwise (any other mode, or auto_approve unset/false), Colony posts a neutral Comment review carrying the same summary instead. This is independent of review.empty_verdict_auto_approve, which controls a separate auto-approve path for internal (Colony-authored) PR reviews.


Regardless of mode, Colony always skips:

  • Colony’s own pipeline PRs — branches matching colony/issue-N or other agent branch patterns are never reviewed
  • Colony’s bot username — PRs opened by Colony’s configured bot_username are skipped
  • PRs carrying any colony: label — pipeline-managed PRs are excluded

These exclusions cannot be overridden.


In multi-repo deployments, you can set a global default and override it per repo:

# Global default — off for all repos
review:
external_prs:
mode: off
repos:
- owner: myorg
repo: my-app
workspace:
repo_dir: /repos/my-app
base_dir: ~/.colony/workspaces/myorg/my-app
review:
external_prs:
mode: on_request # only this repo gets external review

Each field set in the per-repo review.external_prs block overrides the corresponding global value; fields not set in the per-repo block fall back to the global review.external_prs value (or the built-in default if neither is set).


Besides the formal review, Colony writes one Colony Review check run on the reviewed head commit, so branch protection can enforce the verdict. The check passes only when Colony reviewed that head and found nothing blocking; no outcome reports neutral, because both GitHub and Azure DevOps treat it as passing.

OutcomeFormal reviewCheck conclusion
Spawn-level environment errorCommentfailure (not reviewed)
Deterministic checks failedRequest changesfailure
Checks failed on the base tooCommentfailure (not reviewed)
Unparseable LLM outputCommentfailure (not reviewed)
Approve, but production files unaccountedCommentfailure (review incomplete)
Approve with auto_approveApprovesuccess
Approve without auto_approveCommentsuccess
Request changes verdictRequest changesfailure

To make Colony’s verdict mandatory, add Colony Review as a required status check in your branch protection rules. No check is written when no review is posted (merge conflicts, PR not found) or when the head SHA cannot be read.

For full field documentation, see ExternalPrReviewConfig in the configuration reference.

FieldDefaultDescription
mode'off'off, on_request, on_ready, or auto
author_filter['*']Allowlist of GitHub usernames (['*'] = all)
label_filter[]Require all listed labels (AND semantics)
exclude_botstrueSkip bot-authored PRs
exclude_draftstrueSkip draft PRs
auto_approvefalseWith mode: auto, a passing (APPROVE) verdict posts a formal GitHub Approve review event; otherwise a passing verdict posts a neutral Comment review. Startup warning if set without mode: auto. An APPROVE that leaves a changed production file unaccounted for posts as a Comment regardless