GitHub App Setup Guide
GitHub App Setup Guide
Section titled “GitHub App Setup Guide”This guide walks through setting up a GitHub App for Colony authentication. Using a GitHub App (instead of a Personal Access Token) provides fine-grained permissions, higher rate limits, and bot-attributed activity.
Prerequisites
Section titled “Prerequisites”- Admin access to your GitHub organization (or repository owner access for personal repos)
- Colony installed and running with a PAT-based config (to verify the pipeline works first)
Step 1: Create the GitHub App
Section titled “Step 1: Create the GitHub App”-
Go to your Organization Settings > Developer settings > GitHub Apps > New GitHub App
- For personal repos: Settings > Developer settings > GitHub Apps > New GitHub App
-
Fill in the app details:
- GitHub App name: e.g.,
colony-bot(must be globally unique) - Homepage URL: your org’s URL or
https://github.com/<org>/<repo> - Webhook: check “Active” and set the URL to your Colony webhook receiver endpoint. If you are not using webhooks (polling-only mode), uncheck “Active” (slash commands are unavailable in polling-only mode). When webhooks are enabled, subscribe to exactly these events (under Subscribe to events; each needs the listed repository permission, set in the next item):
Event What it drives Required permission Issues Issue intake, label commands, close Issues: Read & write Issue comment Slash commands and clarification replies Issues: Read & write Pull request PR lifecycle and drift detection Pull requests: Read & write Pull request review review_body(top-level review submission) ingestionPull requests: Read & write Pull request review comment review_inline(inline/line-level comment) ingestionPull requests: Read & write Check run CI-gated auto-merge wake Checks: Read-only or higher Check suite CI-gated auto-merge wake Checks: Read-only or higher Status CI-gated auto-merge wake for commit-status CI Commit statuses: Read-only Workflow run CI-gated auto-merge wake for workflow-run CI Actions: Read-only Without Status and Workflow run, CI that reports through commit statuses or workflow runs only reaches Colony on the next full poll. Without the Pull request review comment subscription, inline (
pull_request_review_comment) ingestion silently never fires — operators get no error, just no inline command processing or attribution. - GitHub App name: e.g.,
-
Set Repository permissions:
Permission Access Purpose Contents Read & write Push branches, read repo files Issues Read & write Manage issue labels, post comments Pull requests Read & write Open PRs, post reviews, manage PR lifecycle Checks Read & write check_run/check_suitewebhook events (CI status gating for auto-merge), plus writing the Reviewer’s check-run status on every review verdictMetadata Read-only Required (automatically granted) Commit statuses Read-only Receive statuswebhook events (CI status gating for auto-merge)Actions Read-only Receive workflow_runwebhook events (CI status gating for auto-merge)The Pull requests: Read & write permission above already covers the Pull request review and Pull request review comment webhook events — no additional permission is required to receive them or to react/comment on inline review comments.
-
Under Where can this GitHub App be installed? select “Only on this account”.
-
Click Create GitHub App.
-
Note the App ID displayed on the app’s settings page.
Step 2: Generate a Private Key
Section titled “Step 2: Generate a Private Key”- On the app’s settings page, scroll to Private keys.
- Click Generate a private key.
- A
.pemfile will be downloaded automatically. - Move the PEM file to a secure location:
Terminal window mkdir -p ~/.colony/keysmv ~/Downloads/<app-name>.*.private-key.pem ~/.colony/keys/github-app.pemchmod 600 ~/.colony/keys/github-app.pem
Step 3: Install the App on Target Repositories
Section titled “Step 3: Install the App on Target Repositories”- On the app’s settings page, click Install App in the left sidebar.
- Select your organization.
- Choose Only select repositories and pick the repos Colony will manage.
- Click Install.
- Note the Installation ID from the URL:
https://github.com/settings/installations/<INSTALLATION_ID>.
Step 4: Update Colony Configuration
Section titled “Step 4: Update Colony Configuration”Edit colony.config.yaml to use the App credentials:
github: owner: your-org repo: your-repo # Remove or comment out token_env when using App auth: # token_env: GITHUB_TOKEN app: app_id: 123456 # From Step 1 private_key_path: ~/.colony/keys/github-app.pem # From Step 2 installation_id: 78901234 # From Step 3 bot_username: "colony-bot[bot]" # See Step 5You can keep token_env alongside app — Colony will prefer the PAT when both are present, falling back to App auth if the PAT environment variable is not set. For the full github field reference (all app/ops_app sub-fields, token_env, ops_token_env, bot_username), see configuration.md#github and configuration.md#githubappconfig.
Step 5: Set the Bot Username
Section titled “Step 5: Set the Bot Username”The bot_username tells Colony which comments are its own (to avoid processing its own messages as human input). The format is:
<app-slug>[bot]Where <app-slug> is the lowercased, hyphenated version of your app name. For example:
- App name “Colony Bot” → bot_username:
colony-bot[bot] - App name “My Colony App” → bot_username:
my-colony-app[bot]
You can verify the exact slug by checking a comment posted by the app — the commenter login will show the [bot] suffix.
Step 6: Verify
Section titled “Step 6: Verify”Start Colony and check the startup logs:
colony startLook for:
GitHub App auth active owner=your-org repo=your-repoIf you see PAT auth active instead, the App credentials were not loaded — double-check the config and PEM path.
To test end-to-end, create a test issue. Colony should post a comment with the bot avatar and <app-slug>[bot] as the author.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Fix |
|---|---|---|
Authentication failed: token is invalid or expired | Bad PEM file or wrong app_id | Re-download the PEM; verify app_id matches your app |
Authorization failed: token lacks required permissions | Missing repo permissions | Check the app’s permission settings (Step 1.3) |
Repository not found or not accessible | App not installed on the repo | Install the app on the target repo (Step 3) |
Failed to read private key | Wrong path or permissions | Verify the path in config; check chmod 600 |
| Comments show wrong author | bot_username mismatch | Match the exact <slug>[bot] format (Step 5) |
Dual App Setup (Autonomous Merging)
Section titled “Dual App Setup (Autonomous Merging)”Colony can split its GitHub identity across two Apps: App A (colony-coder) authors code, and App B (colony-ops) reviews and approves it. This mirrors how healthy human teams operate on GitHub — the person who writes the code cannot approve their own PR.
With a single App, GitHub rejects the Reviewer’s APPROVED review because the Reviewer and Developer share the same identity (“cannot approve your own pull request”). The Reviewer falls back to posting a plain comment, and Sprint Master’s auto-merge never fires. A human must always press Approve.
With two Apps, the Reviewer posts a formal APPROVED review via App B’s identity, and Sprint Master’s checkAutoMerge can merge the PR without waiting for a human — closing the autonomous loop.
This setup is optional. Single-App deployments continue to work with no changes. Only configure a second App if you want auto_merge_on_approval: true to work without a human approving each PR.
App A — colony-coder
Section titled “App A — colony-coder”This is the App created in Steps 1–5 above. If you followed the single-App guide, you already have App A.
Required permissions for App A:
| Permission | Access | Purpose |
|---|---|---|
| Contents | Read & write | Push branches, read repo files, create worktrees |
| Issues | Read & write | Manage issue labels, post comments |
| Pull requests | Read & write | Open PRs, post analyzer/developer comments |
| Metadata | Read-only | Required (automatically granted) |
| Checks | Read-only | Receive check_run/check_suite webhook events (CI status gating for auto-merge) |
| Commit statuses | Read-only | Receive status webhook events (CI status gating for auto-merge) |
| Actions | Read-only | Receive workflow_run webhook events (CI status gating for auto-merge) |
App A is the webhook-receiving identity: subscribe it to the nine events listed in Step 1.
Used by: Analyzer, Developer
App B — colony-ops
Section titled “App B — colony-ops”Create a second GitHub App following the same steps (Steps 1–5), but with a different name (e.g., colony-ops). This App must be a different GitHub identity than App A — GitHub checks the identity of the reviewer, not just the token.
Required permissions for App B:
| Permission | Access | Purpose |
|---|---|---|
| Contents | Read-only | Read repo files (no push — App B never writes code) |
| Issues | Read & write | Transition labels, post sprint-master/reviewer/merger comments |
| Pull requests | Read & write | Post formal PR reviews, merge PRs |
| Checks | Read & write | Write the Reviewer’s check-run status; read CI status for auto-merge gating |
| Metadata | Read-only | Required (automatically granted) |
Used by: Sprint Master, Planner, Reviewer, Merger (API calls)
Merger edge case: Merger performs git rebase and force-push (requiring
contents: write) using App A’s git credentials, but all GitHub API calls (post comment, transition labels, merge PR) use App B’s identity. The git worktree is configured with App A credentials. No changes to Merger’s git setup are needed.
Step B1: Create App B
Section titled “Step B1: Create App B”Follow the same Steps 1–5 from this guide for the second App, using the permissions table above. Note:
- A different App name (e.g.,
colony-ops) — must be globally unique - Install on the same repositories as App A
- Save the PEM file at a different path (e.g.,
~/.colony/keys/github-ops-app.pem) - Note the App B
app_id,installation_id, andbot_username - Webhook: leave Active unchecked and subscribe to no events. App A’s subscriptions already deliver every event the receiver consumes; if App B subscribes too, every event reaches the receiver twice.
Step B2: Update Colony Configuration
Section titled “Step B2: Update Colony Configuration”ops_app, ops_token_env, and bot_username are github config fields — for the full field reference see configuration.md#github and configuration.md#githubappconfig. Add an ops_app block alongside the existing app block:
github: owner: your-org repo: your-repo app: app_id: 111111 # App A (colony-coder) private_key_path: ~/.colony/keys/github-app.pem installation_id: 11111111 ops_app: app_id: 222222 # App B (colony-ops) private_key_path: ~/.colony/keys/github-ops-app.pem installation_id: 22222222 bot_username: 'colony-coder[bot]' # App A's bot username
review: auto_merge_on_approval: trueAlternatively, use a PAT for App B instead of a full GitHub App:
github: ops_token_env: COLONY_OPS_TOKEN # env var holding App B's installation tokenNote: the Checks API GitHub uses for the review-verdict check-run is a GitHub-App-only surface, so a PAT-only ops identity cannot create check runs. This fails silently with a warning (Could not post check run — issue comment is the feedback record) rather than blocking the review — a PAT-only ops identity loses only the check-run status annotation, not the formal review/comment.
Per-repo override
Section titled “Per-repo override”For multi-repo deployments, you can set ops_app per repo in the repos array:
repos: - owner: your-org repo: your-repo app: app_id: 111111 private_key_path: ~/.colony/keys/github-app.pem installation_id: 11111111 ops_app: app_id: 222222 private_key_path: ~/.colony/keys/github-ops-app.pem installation_id: 22222222Git Push Credentials
Section titled “Git Push Credentials”Git operations (clone, fetch, push) use system-level authentication — SSH keys or a credential helper. These should be configured with App A’s identity since only App A has contents: write. App B cannot push code.
If you use SSH for git operations, the SSH key should correspond to App A. App B credentials are only used for GitHub API calls (reviews, comments, label transitions, merges).
Backward Compatibility
Section titled “Backward Compatibility”If only app is configured (no ops_app), Colony behaves exactly as before:
coderCodeHostandopsCodeHostboth point to the same App A client- The Reviewer will fall back to posting a plain comment when GitHub rejects the self-review
auto_merge_on_approvalstill works for human-approved PRs
No config changes are needed for existing single-App deployments.
Verification
Section titled “Verification”After configuring both Apps:
-
Enable
auto_merge_on_approval: truein your config -
Run
colony check— it should not warn about missing ops identity -
Create a test issue and let it flow through the pipeline
-
When the Reviewer approves, check the Reviewer logs for:
Posted formal PR review via ops identity repo=your-org/your-repo issue=N event=APPROVEIf you instead see a warning
Could not post formal PR review — issue comment is the feedback record(with anerrorfield describing a self-approval rejection), the Reviewer is still using a single identity — verify theops_appconfig and that App B is a different GitHub account than App A. -
If
auto_merge_on_approval: true, the Sprint Master should merge the PR automatically after the formal approval without waiting for a human.
Troubleshooting Dual App Setup
Section titled “Troubleshooting Dual App Setup”| Symptom | Cause | Fix |
|---|---|---|
Could not post formal PR review — issue comment is the feedback record warning in Reviewer logs | ops_app is not configured, or App B is the same identity as App A, so GitHub rejects the self-approval | Add ops_app config; ensure App B is a separate GitHub App |
colony check warns about ops_app | auto_merge_on_approval: true but no ops_app configured | Add ops_app config or disable auto_merge_on_approval |
| Sprint Master does not auto-merge after Reviewer approves | App B’s formal review was not posted | Check Reviewer logs for “Posted formal PR review via ops identity” |
Failed to read private key for ops_app | Wrong path in ops_app.private_key_path | Verify path; check chmod 600 on the PEM file |
| App B installation not found | App B not installed on the target repo | Install App B on the repo (same step as App A installation) |
Multi-Repo Setup
Section titled “Multi-Repo Setup”For managing multiple repositories with the same App, use the repos array in config:
github: owner: your-org repo: primary-repo app: app_id: 123456 private_key_path: ~/.colony/keys/github-app.pem installation_id: 78901234 bot_username: "colony-bot[bot]"
repos: - owner: your-org repo: primary-repo app: app_id: 123456 private_key_path: ~/.colony/keys/github-app.pem installation_id: 78901234 bot_username: "colony-bot[bot]" workspace: repo_dir: /path/to/primary-repo base_dir: ~/.colony/workspaces/{owner}/{repo}
- owner: your-org repo: secondary-repo app: app_id: 123456 private_key_path: ~/.colony/keys/github-app.pem installation_id: 78901234 # Same installation if both repos are in one install bot_username: "colony-bot[bot]" workspace: repo_dir: /path/to/secondary-repo base_dir: ~/.colony/workspaces/{owner}/{repo}