Skip to content

Recover a Dead-Lettered Issue

A dead letter is a recorded pipeline transition that Colony could not complete permanently—for example, a failed state transition or task enqueue after retries were exhausted. It is not a synonym for the issue state failure-blocked, and one issue can have more than one historical dead-letter record.

Open Operations → Throughput and choose Failed transitions (/operations/throughput?view=failed). See Dead Letters. The page lists every unresolved record for the tenant chosen in the sidebar tenant selector, with its repository, issue, transition, agent, error, and age. Select an issue number to open its work item. Each row has a Resolve button, and Resolve all acknowledges every unresolved record after you confirm. The page has no Retry control; retry is API-only.

To read the same records programmatically, call GET /pipeline/dead-letters with an authenticated Cloud session.

Read the dead-letter error together with the issue timeline and current state. The current state may have advanced by another route since the record was created.

Before retrying, verify:

  • the target transition is still appropriate for the issue;
  • the Worker pool and database are healthy;
  • provider credentials and permissions are valid;
  • any missing workflow definition, task executor, or queue dependency is now available;
  • a permanent validation error has actually been corrected.

For a normal issue-stage failure rather than a dead-letter transition, use Triage a Stuck Issue.

On the Dead Letters page, Resolve on a row acknowledges that record, and Resolve all acknowledges every unresolved record for the selected tenant after you confirm. Neither replays the transition; the record is marked resolved and Colony will not retry it.

Retry (POST /pipeline/dead-letters/{id}/retry) is API-only today, and it restarts the issue from new rather than replaying the transition. See the Colony Cloud API tab before you use it.

  1. Check that the record is gone: reload the Dead Letters page in Colony Cloud, or list unresolved dead letters again through the API or CLI.
  2. Inspect the issue’s current state and timeline.
  3. Confirm any task derived from the transition is pending, claimed, or completed. On Cloud the issue re-enters intake, so the derived task is the intake/analysis task rather than the recorded transition’s task.
  4. If replay fails again, diagnose the new error before another attempt.