Metrics
This page gives the canonical definition, calculation method and unit for Colony Cloud’s metrics.
Pipeline Throughput
Section titled “Pipeline Throughput”Issue-level throughput, cycle time, and quality metrics reported across the selected dashboard window (7 d or 30 d).
| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| Open Issues | All pipeline issues currently being tracked, excluding terminal and paused states. | Count of issues in any state except done and paused. Source: live state counts polled at 3 s cadence. | count |
| In-flight | Issues actively progressing through the pipeline — all non-terminal, non-blocked, non-paused states. Canonical Pipeline scope. | Sum of counts for states: analyzing, planning, ready-for-dev, in-review, changes-requested, merge-pending, waiting-for-subtasks. | count |
| Blocked | Issues that cannot progress without external intervention. | Sum of counts for states: dependency-blocked, failure-blocked, needs-clarification. | count |
| Awaiting Review | Issues where Colony has opened a PR and is waiting for a human code review. | Count of issues in state human-review-ready. | count |
| Throughput | Number of issues completed (PR merged to the default branch) within the selected window. | Count of issues whose first PR merged within the window. Source: issuesCompletedByDay aggregated. | count |
| Cycle Time | Median elapsed time from issue creation to PR merge across issues completed in the selected window. | PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY (minutes_issue_to_first_pr + minutes_pr_to_merge)) for issues reaching done state in the window. | minutes |
| Success Rate | Percentage of resolved issues that merged successfully, where resolved means merged in the selected window, or opened in the window and currently paused. | success_count / total_resolved, where success_count is issues merged in the window and total_resolved adds issues opened in the window that are currently paused, expressed as a percentage. Source: weekOverWeek.successRate.current. | percent |
| Review Pass Rate | Fraction of merged PRs that passed code review on the first submission without a changes-requested cycle. | first_pass_approvals / total_merged_prs for the selected window. Source: reviewPassRate field. | ratio |
| Time to First PR | Median elapsed time from issue creation to the first PR opened for that issue. | PERCENTILE_CONT(0.5) of (first_pr_opened_at - issue_created_at) for issues in the window. Source: medianTimeToFirstPrMs field. | ms |
| Avg Review Cycles | Average number of review rounds (each changes-requested transition counts as one cycle) before a PR is approved. | AVG(changes_requested_count) per PR for merged PRs in the window. Source: avgReviewCyclesPerPr field. | count |
| PR Revert Rate | Fraction of merged PRs that were subsequently reverted within the tracking window. | reverted_pr_count / total_merged_prs for the selected window. Source: prRevertRate field. | ratio |
| Agent Attribution | Share of merged PRs handled by Colony agents versus human contributors in the selected window. | agent_pr_count / total_merged_prs for the window, split into agent % and human %. Source: /pipeline/attribution API. | ratio |
| Autonomous Merge Rate | Fraction of merged issues completed end-to-end by Colony with no human commits and no changes-requested review cycle. | autonomousIssues / mergedIssues for the selected window, where autonomous = first_pr_merged_at in window AND no changes-requested cycle (MAX(review_cycle_count) = 0 across its pull request events) AND all PRs agent-authored (BOOL_AND(effective_is_agent) across its agent attribution). Source: /pipeline/autonomy API, ratePct field. | percent |
LLM token spend tracked at org level for today or a selected window.
| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| Cost today | LLM token spend by Colony agents since midnight UTC today. | Sum of cost_usd across cost events for the current calendar day (UTC). Source: Pipeline Store, refreshed on each poll. | usd |
| Cost | Total LLM token spend across all Colony agents for the selected time window (24h / 7d / 30d / lifetime). | Sum of cost_usd across cost events within the selected window. Source: /pipeline/cost-summary API (last24hUsd, last7dUsd, last30dUsd, lifetimeUsd fields). | usd |
Queue / DLQ
Section titled “Queue / DLQ”Task queue depth and dead-letter queue health at the org level.
| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| DLQ depth | Number of unresolved entries in the dead-letter queue: state transitions or task enqueues that failed permanently and need manual resolution. | Count of unresolved dead-letter entries. Source: /pipeline/dead-letters endpoint. | count |
| Pending tasks | Number of pipeline tasks queued but not yet assigned to a Worker. | Count of work tasks with status pending. Source: /pipeline/snapshot endpoint, pendingTaskCount field. | count |
| Avg queue wait | Mean elapsed time since task creation for all current pending work tasks. | AVG(waitTimeMs) across all pending tasks of the tenant (or repo), as computed by GET /pipeline/task-queue. | ms |
| Oldest pending age | Elapsed time since the oldest pending task was enqueued. | MAX(waitTimeMs) for tasks with status='pending' across all pending tasks of the tenant (or repo). | ms |
| Est. duration | Median (p50) historical execution time for tasks of this type in this repository, based on completed work tasks from the last 7 days. | PERCENTILE_CONT(0.5) of (completed_at - claimed_at) in seconds from the history of completed work tasks, grouped by (repo_id, executor_ref) over a trailing 7-day window. Only shown when the sample count is ≥ 5. Source: GET /pipeline/task-queue, estimatedDurationMs field per task. Tasks with no history show —. | ms |
| Projected drain | Sum of estimated execution durations for all pending tasks that have historical duration data, giving a rough lower-bound on how long the queue will take to clear. | Sum of estimatedDurationMs across pending tasks where the estimate is non-null. Tasks with no history are excluded from the sum; the hint is hidden when no pending task has an estimate. Source: GET /pipeline/task-queue response. | ms |
| Write queue | Combined count of pending and failed VCS write projections (label swaps, comments, PR operations) across all repos in the org. | Count of VCS write projections with status 'pending' or 'failed' joined to repos belonging to this org. Source: /pipeline/operational-summary endpoint, vcsWriteQueueDepth and vcsWriteQueueFailed fields. | count |
Workers & SI Tracks
Section titled “Workers & SI Tracks”Point-in-time counts for Workers and self-improvement tracks.
| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| Worker count | Number of Colony Worker containers currently registered and reporting heartbeats to the Cloud API. | Count of registered Workers with a recent heartbeat. Each Worker handles one pipeline issue at a time. Source: /api/orgs/{orgId}/workers endpoint. | count |
| SI Tracks count | Number of self-improvement tracks configured across all repos in the org. | A track is overdue if now - last_run_at > cooldown_minutes. Source: count summed across the per-repo GET /api/repos/{owner}/{repo}/si-tracks responses. | count |
| Cadence / cooldown | How often a track may run — either a cooldown interval or a fixed cron schedule. | Cooldown tracks re-run once now - last_run_at > cooldown_minutes (stored as cooldownMinutes). Cron tracks run on a fixed schedule defined by a cron expression (stored as cronExpr). Source: summed across the per-repo GET /api/repos/{owner}/{repo}/si-tracks responses, cadenceKind and cooldownMinutes/cronExpr fields. | — |
Retrospectives
Section titled “Retrospectives”Planned-vs-actual analysis over completed issues.
| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| Plan Drift Rate | Fraction of completed issues in the window where the number of files Colony actually touched differed from the number it planned. | (touchedMore + touchedFewer) / totalRetrospectivesWithFileData over the window, from the planned-vs-actual file sets of each retrospective. | ratio |
| Outcome Distribution | Share of completed issues by outcome classification (e.g. success / partial / failure) over the window. | COUNT(*) GROUP BY outcome_classification over retrospectives in the window. | count |
Rolling Health KPIs
Section titled “Rolling Health KPIs”Regression-guard signals computed from state transitions, compared against a rolling baseline window. These are per-repo metrics.
| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| Block Rate | Fraction of issues reaching done, failure-blocked or dependency-blocked in the rolling window whose latest such move was to a blocked state. | blocked_count / total_issues over the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the rate rises — a higher block rate signals increasing pipeline impedance. | ratio |
| Failure-Blocked Rate | Fraction of those issues blocked for any reason other than a dependency. | failure_blocked_count / total_issues over the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the rate rises — more CI failures indicate systematic test or build instability. | ratio |
| Mean Cost / Issue | Mean LLM token spend (in USD) per issue completed in the rolling window, tracking cost efficiency of the autonomous pipeline. | AVG(cost_usd_per_issue) for completed issues in the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the mean rises — increasing per-issue cost indicates the pipeline requires more LLM work per deliverable. | usd |
| Mean Turns / Issue | Mean number of agent turns (LLM round-trips) consumed per issue completed in the rolling window, measuring implementation efficiency. | AVG(turns_per_issue) for completed issues in the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the mean rises — more turns per issue signals increased difficulty or iterative rework. | count |
| Reimplement Loop Rate | Fraction of those issues blocked because they hit the re-implementation loop limit. | reimplement_loop_count / total_issues over the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the rate rises — repeated re-implementations indicate systematic failure in planning or initial development quality. | ratio |
| Dead Letter Rate | New dead-letter entries in the rolling window, relative to issues reaching done, failure-blocked or dependency-blocked in that window. | dead_letter_count / terminal_count over the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the rate rises — more dead-lettered issues indicate the pipeline is unable to autonomously resolve an increasing share of work. | ratio |
Cost by Model
Section titled “Cost by Model”Per-model LLM spend and execution quality signals for the selected window.
| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| Model Spend Share | Fraction of total org LLM spend attributable to this model in the selected window. | model_cost / SUM(model_cost) OVER () for cost events in the window, grouped by model. Source: /pipeline/cost-by-model API, totalCostUsd relative to window total. | percent |
| Max-Turns Rate by Model | Fraction of invocations for this model that hit the agent turn limit, indicating tasks that exceeded the model’s context or reasoning budget. | COUNT(*) FILTER (WHERE is_max_turns = true) / COUNT(*) for execution telemetry records in the window, grouped by model. Source: /pipeline/cost-by-model API, maxTurnsRate field. | percent |
| No-Progress Rate by Model | Fraction of invocations for this model that produced no meaningful code progress, indicating effort consumed without output. | COUNT(*) FILTER (WHERE is_no_progress = true) / COUNT(*) for execution telemetry records in the window, grouped by model. Source: /pipeline/cost-by-model API, noProgressRate field. | percent |
Strategy
Section titled “Strategy”| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| Autonomy grant (max_auto_size) | The largest track size the Strategist may apply automatically without operator approval, as set in the repository autonomy grant. | Read from the max_auto_size field on GET /api/autonomy/:owner/:repo. Possible values: small, medium, large, epic. A repository with no grant configured defaults to small. The limit is enforced at approval time, against the grant as it stands when a snapshot is approved. Changes require an explicit grant update by an operator. | — |
| Strategy snapshot status | The lifecycle status of a Strategist-proposed strategy snapshot. | Read from the status field on StrategySnapshot returned by GET /api/strategy/:owner/:repo/snapshots. A snapshot starts proposed. Approving it applies it at once and makes it active (the previous active one becomes superseded); declining sets it rejected; a refused plain approve sets it invalid; a charter change or a newer active strategy sets it superseded. With automatic approval Colony applies a valid proposal itself. There is no resting approved state. | — |
Data Consistency
Section titled “Data Consistency”Cross-store audit findings that surface drift between Postgres pipeline state, dependency edges, SI linkage, and projected GitHub labels.
| Metric | Definition | Calculation | Unit |
|---|---|---|---|
| Data Consistency Findings | Count of unresolved data-consistency audit findings across the org — issues where Postgres pipeline state, dependency edges, SI linkage, or projected GitHub labels have drifted out of agreement. | Count of unresolved findings grouped by kind: stale_blocked, stale_subtask_edge, orphan_missing_from_pg, si_linkage_gap, label_drift, orphan_dependency_edge. Source: the data-consistency audit. | count |
| Stale Blocked | Issues with is_blocked = true and state != 'done' but no active depends_on dependency edge — the blocked flag is stranded with no actual dependency blocking the issue. | Queries issues for is_blocked = true AND state != 'done' with NOT EXISTS a dependency edge where dep_type = 'depends_on' AND status = 'active'. Source: the data-consistency audit, stale_blocked findings. | count |
| Stale Subtask Edge | Active subtask dependency edges whose parent epic has reached done state — dangling edges not cleaned up when the epic completed. | Queries dependency edges for dep_type = 'subtask' AND status = 'active', joined to the parent epic where state = 'done'. Grouped one finding per epic. Source: the data-consistency audit, stale_subtask_edge findings. | count |
| Orphan (missing from Postgres) | Open GitHub issues that have no corresponding pipeline issue — visible on GitHub but not yet ingested into the pipeline. | Compares open GitHub issue numbers against the pipeline issues for the repo; issues absent from Postgres are flagged. Requires a live GitHub API call — this detector is deferred/optional in Cloud. Source: the data-consistency audit, orphan_missing_from_pg findings. | count |
| SI Linkage Gap | Self-improvement issues whose pipeline issue reached done but completed_at is still NULL — the SI completion event was not recorded. | Queries self-improvement issues joined to their pipeline issues where pi.state = 'done' AND si.completed_at IS NULL. Source: the data-consistency audit, si_linkage_gap findings. | count |
| Label Drift | Issues where the Postgres authoritative state (state, is_blocked, is_paused) disagrees with the projected GitHub labels stored on the issue as labels — done issues excluded. | Queries issues where state != 'done' and at least one label disagrees: stateToLabel(state) not in labels, is_blocked mismatches labels.includes(BLOCKED_LABEL), or is_paused mismatches labels.includes(PAUSED_LABEL). Source: the data-consistency audit, label_drift findings. | count |
| Orphan Dependency Edge | Dependency edges whose parent or child endpoint has no pipeline issue — the edge points at an issue the pipeline has no record of. | Scans the repo’s dependency edges and flags any whose parent_issue or child_issue is absent from the pipeline issues, grouped by the missing endpoint. Source: the data-consistency audit, orphan_dependency_edge findings. | count |