Rotate Credentials Without Stopping the Pipeline

A safe rotation follows add, test, switch, observe, revoke. Keep the old credential valid until every process that needs the replacement has proved it can work. “Without stopping the pipeline” means preserving queued work and maintaining capacity where the deployment supports a rolling change; some singleton and database rotations still require a brief controlled interruption.

Inventory consumers first

Credential	Typical consumers	Reload behavior
Cloud GitHub Apps	Colony Cloud control plane and Workers through short-lived installation tokens	Managed by Cloud; reconnect only when installation access changes.
Cloud LLM/API secrets	Managed and BYO Worker tasks	Update in Settings → Secrets; verify a new task before removing the old provider key.
Cloud Worker token	BYO Worker registration and heartbeat	Deploy replacement Workers with a new token, drain old Workers, then revoke.
Self-host GitHub/ADO provider identity	Mayor, Workers, monitor, webhook receiver as applicable	Startup-only; restart consumers.
Self-host LLM credential	Workers and any singleton that invokes the configured engine	Environment change requires process restart.
Postgres credential	Mayor, Workers, monitor, MCP, and operational CLI	Coordinate database and all clients; normally requires reconnect/restart.
Monitor login credential	Monitor dashboard/API	Restart monitor; active sessions may be invalidated.
Package-feed credential or custom CA	Worker entrypoint and package managers	Re-rendered at Worker startup; replace Workers one at a time.

Configuration hot reload does not reread arbitrary environment variables. Provider, database, tenant, repository identity, webhook, and command settings are explicitly redeploy-only.

Rotate a Cloud secret

Create the replacement credential at the upstream provider without revoking the current one.
Open Settings → Secrets and update the corresponding secret.
Start a small task that exercises the credential and inspect its Worker result.
Observe normal tasks through at least one complete analyze/develop/review path when the secret affects the LLM engine.
Revoke the old upstream credential only after new tasks consistently use the replacement.

Changing a package-feed credential for a repository updates what Workers receive on a later registration/config refresh. Validate with a clean dependency install, not a Worker that already has packages cached.

Rotate a Cloud BYO Worker token

Generate a new token under Settings → Workers for the same logical group.
Deploy at least one replacement Worker with the new token and confirm a fresh heartbeat.
Let the replacement complete a real task for each required repository or role.
Request Drain on Workers using the old token and wait for their claims to finish.
Stop the drained containers.
Revoke the old token and confirm no active Worker still depends on it.

Revoking first immediately blocks registration and heartbeat. Do that only for an active compromise, then inspect and reclaim any abandoned tasks.

Rotate self-host provider or LLM credentials

Create a new PAT/key with the same least-privilege access and leave the old one active.
Update the secret value behind the existing environment-variable name. Do not rename the variable and change config in the same rotation unless necessary.
Run colony check --stage credentials and colony check --stage repository from an environment using the replacement.
Replace Workers one at a time when at least two serve the repository; wait for each replacement heartbeat and task before continuing.
Restart the Mayor and other singleton consumers during a controlled window. The durable queue remains in Postgres, but intake may pause briefly.
Revoke the old credential after the replacement completes a full workflow.

Add a new private key to the same GitHub App before deleting the old key. Mount the new PEM, update private_key_path only when the path changes, and roll the consuming processes. Confirm App installation-token generation and repository access, then remove the old key from GitHub.

Rotate package-feed credentials or certificates

Update the protected host copy of /colony/keys/credentials.yaml or the mounted CA files. The Worker entrypoint renders package-manager configuration and trust stores from scratch at startup, so a running Worker does not adopt the change.

Add the new feed PAT or certificate while the old one remains accepted.
Replace one Worker and wait for startup rendering to succeed.
Run a clean setup that downloads from every affected feed or TLS endpoint.
Continue the rolling replacement across the pool.
Revoke the old PAT or remove the old certificate only after all Workers have restarted.

Rotate the database credential

Use the database provider’s dual-password, role handoff, or connection-proxy feature when available:

Add a new database role/password with equivalent least-privilege schema and runtime access.
Validate a new connection and backup with the replacement.
Update Colony’s protected DATABASE_URL value.
Drain/replace Workers, then restart the Mayor, monitor, MCP, and other long-lived clients so every pool reconnects.
Confirm migrations, queue reads/writes, heartbeats, and monitor queries.
Revoke the old database credential after checking active database sessions.

If the database cannot accept old and new credentials simultaneously, schedule a full colony safestop and treat the rotation as planned downtime rather than risking partial connectivity.

Verify and close the rotation

No Worker or singleton logs authentication failures.
Every intended Worker has a fresh heartbeat and the expected repository scope.
Provider write operations, formal review, and merge work under the correct identities.
Clean dependency installation succeeds without cached credentials.
The monitor and CLI can read and write Postgres.
The old credential is revoked at the provider, not merely removed from one environment file.
The rotation time, affected credential identifier, verification evidence, and rollback window are recorded without storing the secret itself.