Recover a Dead-Lettered Issue
A dead letter is a recorded pipeline transition that Colony could not complete permanently—for example, a failed state transition or task enqueue after retries were exhausted. It is not a synonym for the issue state failure-blocked, and one issue can have more than one historical dead-letter record.
List unresolved records
Section titled “List unresolved records”Open Operations → Throughput and choose Failed transitions (/operations/throughput?view=failed). See Dead Letters. The page lists every unresolved record for the tenant chosen in the sidebar tenant selector, with its repository, issue, transition, agent, error, and age. Select an issue number to open its work item. Each row has a Resolve button, and Resolve all acknowledges every unresolved record after you confirm. The page has no Retry control; retry is API-only.
To read the same records programmatically, call GET /pipeline/dead-letters with an authenticated Cloud session.
List the authoritative records with the CLI:
colony dead-letterscolony dead-letters --repo owner/repocolony dead-letters --jsonThe table includes the record ID, issue, source and target states, agent, error, and age. GitHub labels and log searches can help with diagnosis, but they cannot tell you whether the dead-letter record itself is unresolved.
Diagnose the failed transition
Section titled “Diagnose the failed transition”Read the dead-letter error together with the issue timeline and current state. The current state may have advanced by another route since the record was created.
Before retrying, verify:
- the target transition is still appropriate for the issue;
- the Worker pool and database are healthy;
- provider credentials and permissions are valid;
- any missing workflow definition, task executor, or queue dependency is now available;
- a permanent validation error has actually been corrected.
For a normal issue-stage failure rather than a dead-letter transition, use Triage a Stuck Issue.
Retry or acknowledge
Section titled “Retry or acknowledge”On the Dead Letters page, Resolve on a row acknowledges that record, and Resolve all acknowledges every unresolved record for the selected tenant after you confirm. Neither replays the transition; the record is marked resolved and Colony will not retry it.
Retry (POST /pipeline/dead-letters/{id}/retry) is API-only today, and it restarts the issue from new rather than replaying the transition. See the Colony Cloud API tab before you use it.
Retry the record:
POST /pipeline/dead-letters/{id}/retryCloud retry resolves the record and restarts the issue from new. It does not replay the recorded target state, and the response body is { "ok": true } with no resulting state. Restarting from intake can re-run analysis and incur extra cost.
If the recorded transition is still the right next step and you don’t want the issue to start over, use the issue’s workflow actions on the work-item page instead, then Resolve the record.
Or acknowledge it without retrying:
POST /pipeline/dead-letters/{id}/resolveUse resolve when the issue already recovered, was intentionally canceled, or the recorded transition is no longer appropriate. These endpoints require an authenticated Cloud session and are tenant-scoped.
Retry the exact recorded transition:
colony dead-letters retry <id>A successful retry applies the transition, enqueues any derived task, and resolves the record. To acknowledge without retrying:
colony dead-letters resolve <id>resolve-all is available, optionally with --repo, but use it only after reviewing the records; it acknowledges them without replaying their transitions.
Verify the result
Section titled “Verify the result”- Check that the record is gone: reload the Dead Letters page in Colony Cloud, or list unresolved dead letters again through the API or CLI.
- Inspect the issue’s current state and timeline.
- Confirm any task derived from the transition is pending, claimed, or completed. On Cloud the issue re-enters intake, so the derived task is the intake/analysis task rather than the recorded transition’s task.
- If replay fails again, diagnose the new error before another attempt.