External PR Review
External PR Review
Section titled “External PR Review”Colony can review human-authored pull requests — PRs that are not part of its own pipeline — using the same review engine it applies to its own work: deterministic checks (build, test, lint) run in a real worktree, followed by an LLM code review that understands the repo’s conventions.
This is configured via review.external_prs in your Colony config.
Colony never edits the title or body of an external PR. For the review format it posts, see What Colony puts on a pull request.
There are three modes, ordered from safest to most aggressive:
| Mode | Trigger | Best for |
|---|---|---|
on_request | A user with write access comments /colony:review on the PR | Getting started; explicit opt-in per PR |
on_ready | PR is opened or transitions from draft to ready-for-review | Teams that want automatic review without auto-approval |
auto | Same as on_ready, plus the auto_approve flag is available | Fully automated review pipelines |
The feature is off by default (mode: off). You must opt in.
Mode 1 — On Request (Recommended Starting Point)
Section titled “Mode 1 — On Request (Recommended Starting Point)”Enable by setting mode: on_request. Colony only reviews a PR when a human explicitly requests it.
review: external_prs: mode: on_requestHow to trigger a review
Section titled “How to trigger a review”- Open any non-pipeline PR (non-draft, unless
exclude_drafts: false) in your repo. - Comment
/colony:reviewon the PR. - Colony adds a 👍 reaction to acknowledge the request.
- Colony runs deterministic checks and the LLM review, then posts the results as a GitHub PR review.
Under the default commands.authorization: write, the commenter must have write access to the repo; a commenter without it gets a one-time reply explaining the refusal. Set commands.authorization: any to allow any account (see commands). Bot comments are ignored.
The author_filter, label_filter, exclude_drafts and exclude_bots filters (described under Mode 2) also apply to /colony:review requests; a request on a filtered-out PR is skipped silently.
Re-triggering a review
Section titled “Re-triggering a review”After you address feedback and push new commits, comment /colony:review again to request a fresh review. Colony will run against the updated head commit and post a new review, dismissing the previous one.
Mode 2 — On Ready
Section titled “Mode 2 — On Ready”Colony automatically reviews a PR when it is opened (non-draft) or transitions from draft to ready-for-review.
review: external_prs: mode: on_readyFiltering who gets reviewed
Section titled “Filtering who gets reviewed”The filters in this section and the next apply in every mode other than off, including on_request. A /colony:review request on a filtered-out PR is skipped silently.
Use author_filter to limit automatic reviews to a specific allowlist of GitHub usernames. The default ['*'] means all authors are eligible.
review: external_prs: mode: on_ready author_filter: ['alice', 'bob', 'carol']Use label_filter to require specific labels on the PR before reviewing. Multiple labels use AND semantics — all listed labels must be present.
review: external_prs: mode: on_ready label_filter: ['review-please']Bot and draft exclusions
Section titled “Bot and draft exclusions”By default, Colony skips:
- Draft PRs (
exclude_drafts: true) — set tofalseto review drafts - Bot-authored PRs (
exclude_bots: true) — set tofalseto review bot PRs
review: external_prs: mode: on_ready exclude_drafts: false # review drafts tooRe-review on push
Section titled “Re-review on push”When a contributor pushes new commits to a PR that Colony has already reviewed, Colony automatically re-reviews the updated PR. This only fires if Colony has previously reviewed that PR — new pushes on PRs that have never been reviewed do not trigger an unsolicited initial review.
Mode 3 — Auto
Section titled “Mode 3 — Auto”Same as Mode 2 in terms of triggers — Colony reviews every eligible PR when it is opened or transitions to ready. The auto_approve config flag is only meaningful with this mode (Colony logs a startup warning if auto_approve: true is set without mode: auto).
review: external_prs: mode: auto auto_approve: trueNote: Colony always posts a Request Changes event when the verdict is
REQUEST_CHANGES, regardless of mode orauto_approve. For a passing (APPROVE) verdict, the event depends onauto_approve: withmode: autoandauto_approve: true, Colony posts a formal GitHub Approve review event; otherwise (any other mode, orauto_approveunset/false), Colony posts a neutral Comment review carrying the same summary instead. This is independent ofreview.empty_verdict_auto_approve, which controls a separate auto-approve path for internal (Colony-authored) PR reviews.
Automatic Exclusions
Section titled “Automatic Exclusions”Regardless of mode, Colony always skips:
- Colony’s own pipeline PRs — branches matching
colony/issue-Nor other agent branch patterns are never reviewed - Colony’s bot username — PRs opened by Colony’s configured
bot_usernameare skipped - PRs carrying any
colony:label — pipeline-managed PRs are excluded
These exclusions cannot be overridden.
Per-Repo Overrides
Section titled “Per-Repo Overrides”In multi-repo deployments, you can set a global default and override it per repo:
# Global default — off for all reposreview: external_prs: mode: off
repos: - owner: myorg repo: my-app workspace: repo_dir: /repos/my-app base_dir: ~/.colony/workspaces/myorg/my-app review: external_prs: mode: on_request # only this repo gets external reviewEach field set in the per-repo review.external_prs block overrides the corresponding global value; fields not set in the per-repo block fall back to the global review.external_prs value (or the built-in default if neither is set).
Colony Review check
Section titled “Colony Review check”Besides the formal review, Colony writes one Colony Review check run on the reviewed head commit, so branch protection can enforce the verdict. The check passes only when Colony reviewed that head and found nothing blocking; no outcome reports neutral, because both GitHub and Azure DevOps treat it as passing.
| Outcome | Formal review | Check conclusion |
|---|---|---|
| Spawn-level environment error | Comment | failure (not reviewed) |
| Deterministic checks failed | Request changes | failure |
| Checks failed on the base too | Comment | failure (not reviewed) |
| Unparseable LLM output | Comment | failure (not reviewed) |
| Approve, but production files unaccounted | Comment | failure (review incomplete) |
Approve with auto_approve | Approve | success |
Approve without auto_approve | Comment | success |
| Request changes verdict | Request changes | failure |
To make Colony’s verdict mandatory, add Colony Review as a required status check in your branch protection rules. No check is written when no review is posted (merge conflicts, PR not found) or when the head SHA cannot be read.
Configuration Reference
Section titled “Configuration Reference”For full field documentation, see ExternalPrReviewConfig in the configuration reference.
| Field | Default | Description |
|---|---|---|
mode | 'off' | off, on_request, on_ready, or auto |
author_filter | ['*'] | Allowlist of GitHub usernames (['*'] = all) |
label_filter | [] | Require all listed labels (AND semantics) |
exclude_bots | true | Skip bot-authored PRs |
exclude_drafts | true | Skip draft PRs |
auto_approve | false | With mode: auto, a passing (APPROVE) verdict posts a formal GitHub Approve review event; otherwise a passing verdict posts a neutral Comment review. Startup warning if set without mode: auto. An APPROVE that leaves a changed production file unaccounted for posts as a Comment regardless |