Upgrade, Back Up, and Roll Back Self-Hosted Colony

This guide applies only to self-hosted Colony. Colony Cloud upgrades and control-plane backups are operated by Colony.

An upgrade changes application code and may apply forward-only Postgres migrations at startup. A source or image rollback is not sufficient when the new version changed the schema incompatibly; keep a pre-upgrade database backup that can be restored with the previous version.

Inventory the deployment

Record these before changing anything:

• current Colony commit, tag, or image digest;
• deployment mode and Compose overrides;
• colony.config.yaml and any included configuration;
• .env variable names and the location of secret values;
• GitHub App private keys, mounted feed credentials, and custom CA files;
• Postgres server/version, database URL, extensions, and latest schema_migrations version;
• enabled repositories, Worker pool sizes, and current active tasks;
• local plugins or image customizations.

Do not copy live secret values into an unencrypted runbook. Back up the secret store or key files through its approved protected mechanism.

Check the current installation

colony check
colony status --db
colony workers
colony tasks --status claimed
colony migrate apply --dry-run

Resolve credential, database, repository, label, runtime, or migration errors before upgrading. Capture current health output and the latest migration number for comparison.

Back up durable state

Create a custom-format Postgres dump from a host with the matching client tools:

pg_dump --format=custom --file colony-before-upgrade.dump "$DATABASE_URL"

Back up configuration, Compose overrides, protected keys, custom certificates, and plugin manifests separately. Repository clones and Worker worktrees are disposable execution state; the provider repositories and Postgres pipeline records are the durable sources that matter.

Verify the dump before proceeding:

pg_restore --list colony-before-upgrade.dump

For a production recovery test, restore into a separate empty database and run read-only consistency checks against it. A backup that has never been restored is not a tested rollback plan.

Drain the current version

colony safestop --timeout 15

The command asks agents to finish active polls/tasks, prevents new claims, and stops the installation. Without --force, a process that cannot drain remains for investigation instead of being killed. Confirm no claimed tasks remain before replacing binaries or containers.

Caution

colony safestop intentionally pauses processing. Do not describe a whole-installation upgrade as zero downtime. Provider issues and durable queued tasks remain available and resume after restart.

Install the target version

Choose the path matching the original installation. Pin a tag, commit, or immutable image digest; do not use an unrecorded moving tag for a production upgrade.

Docker Compose from source

1. Fetch the target Colony release in the source checkout.
2. Review its changelog, configuration changes, and migration notes.
3. Install locked dependencies and rebuild the Colony image.
4. Preserve the existing .env, config, keys, volumes, and Compose overrides.
5. Start Postgres first, then the Colony services with the normal Compose command.

Native

1. Fetch and check out the target release.
2. Install locked dependencies and build the workspace.
3. Keep the previous checkout or artifact available for rollback.
4. Start Colony with the same config and environment used by the prior version.

Apple Container

Rebuild or fetch the pinned target image, then use the existing Apple Container deployment script and configuration. Keep the previous image reference until the new version passes health and pipeline checks.

Both the Mayor and Workers initialize the pipeline store and apply pending migrations before processing. To control that boundary explicitly, apply and inspect migrations before starting the full pool:

colony migrate apply --dry-run
colony migrate apply

Verify the upgraded installation

1. Inspect startup logs for migration, configuration, provider, and plugin errors.
2. Run colony check and compare it with the pre-upgrade result.
3. Run colony status --db, colony workers, and colony tasks.
4. Confirm the latest schema_migrations version matches the target release.
5. Verify the monitor dashboard and webhook receiver when enabled.
6. Enqueue one small canary issue, follow its complete workflow, and verify provider labels/tags, PR, checks, and final state.
7. Retain the backup and previous application version until the canary and normal workload remain healthy for the agreed observation window.

Roll back application-only changes

If the target version did not apply a schema or data migration incompatible with the previous version:

1. Run colony safestop on the target version.
2. Reinstall or restart the recorded previous commit, tag, or image digest.
3. Start with the previous configuration and verify health.
4. Re-run the canary before restoring normal intake.

Restore the database when required

When the previous version cannot use the upgraded database, do not run it against that database and hope it ignores new state.

1. Stop all Colony processes so no new writes occur.
2. Create a separate empty restore database.
3. Restore colony-before-upgrade.dump into it with pg_restore --no-owner --dbname "$RESTORE_DATABASE_URL" colony-before-upgrade.dump.
4. Point the previous Colony version at the restored database through the normal secret/config mechanism.
5. Start the previous version and verify migrations, repository registrations, queue state, and provider projections.
6. Reconcile provider activity that happened after the backup through normal full sync and operator review; it is not present in the restored database.

Keep the failed upgraded database intact until the incident is understood. It can contain evidence and post-backup work needed for reconciliation.

What next

• Rotate Credentials Without Stopping the Pipeline
• Operate and Scale Workers
• Triage a Stuck Issue