Skip to content

Metrics

This page gives the canonical definition, calculation method and unit for Colony Cloud’s metrics.


Issue-level throughput, cycle time, and quality metrics reported across the selected dashboard window (7 d or 30 d).

MetricDefinitionCalculationUnit
Open IssuesAll pipeline issues currently being tracked, excluding terminal and paused states.Count of issues in any state except done and paused. Source: live state counts polled at 3 s cadence.count
In-flightIssues actively progressing through the pipeline — all non-terminal, non-blocked, non-paused states. Canonical Pipeline scope.Sum of counts for states: analyzing, planning, ready-for-dev, in-review, changes-requested, merge-pending, waiting-for-subtasks.count
BlockedIssues that cannot progress without external intervention.Sum of counts for states: dependency-blocked, failure-blocked, needs-clarification.count
Awaiting ReviewIssues where Colony has opened a PR and is waiting for a human code review.Count of issues in state human-review-ready.count
ThroughputNumber of issues completed (PR merged to the default branch) within the selected window.Count of issues whose first PR merged within the window. Source: issuesCompletedByDay aggregated.count
Cycle TimeMedian elapsed time from issue creation to PR merge across issues completed in the selected window.PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY (minutes_issue_to_first_pr + minutes_pr_to_merge)) for issues reaching done state in the window.minutes
Success RatePercentage of resolved issues that merged successfully, where resolved means merged in the selected window, or opened in the window and currently paused.success_count / total_resolved, where success_count is issues merged in the window and total_resolved adds issues opened in the window that are currently paused, expressed as a percentage. Source: weekOverWeek.successRate.current.percent
Review Pass RateFraction of merged PRs that passed code review on the first submission without a changes-requested cycle.first_pass_approvals / total_merged_prs for the selected window. Source: reviewPassRate field.ratio
Time to First PRMedian elapsed time from issue creation to the first PR opened for that issue.PERCENTILE_CONT(0.5) of (first_pr_opened_at - issue_created_at) for issues in the window. Source: medianTimeToFirstPrMs field.ms
Avg Review CyclesAverage number of review rounds (each changes-requested transition counts as one cycle) before a PR is approved.AVG(changes_requested_count) per PR for merged PRs in the window. Source: avgReviewCyclesPerPr field.count
PR Revert RateFraction of merged PRs that were subsequently reverted within the tracking window.reverted_pr_count / total_merged_prs for the selected window. Source: prRevertRate field.ratio
Agent AttributionShare of merged PRs handled by Colony agents versus human contributors in the selected window.agent_pr_count / total_merged_prs for the window, split into agent % and human %. Source: /pipeline/attribution API.ratio
Autonomous Merge RateFraction of merged issues completed end-to-end by Colony with no human commits and no changes-requested review cycle.autonomousIssues / mergedIssues for the selected window, where autonomous = first_pr_merged_at in window AND no changes-requested cycle (MAX(review_cycle_count) = 0 across its pull request events) AND all PRs agent-authored (BOOL_AND(effective_is_agent) across its agent attribution). Source: /pipeline/autonomy API, ratePct field.percent

LLM token spend tracked at org level for today or a selected window.

MetricDefinitionCalculationUnit
Cost todayLLM token spend by Colony agents since midnight UTC today.Sum of cost_usd across cost events for the current calendar day (UTC). Source: Pipeline Store, refreshed on each poll.usd
CostTotal LLM token spend across all Colony agents for the selected time window (24h / 7d / 30d / lifetime).Sum of cost_usd across cost events within the selected window. Source: /pipeline/cost-summary API (last24hUsd, last7dUsd, last30dUsd, lifetimeUsd fields).usd

Task queue depth and dead-letter queue health at the org level.

MetricDefinitionCalculationUnit
DLQ depthNumber of unresolved entries in the dead-letter queue: state transitions or task enqueues that failed permanently and need manual resolution.Count of unresolved dead-letter entries. Source: /pipeline/dead-letters endpoint.count
Pending tasksNumber of pipeline tasks queued but not yet assigned to a Worker.Count of work tasks with status pending. Source: /pipeline/snapshot endpoint, pendingTaskCount field.count
Avg queue waitMean elapsed time since task creation for all current pending work tasks.AVG(waitTimeMs) across all pending tasks of the tenant (or repo), as computed by GET /pipeline/task-queue.ms
Oldest pending ageElapsed time since the oldest pending task was enqueued.MAX(waitTimeMs) for tasks with status='pending' across all pending tasks of the tenant (or repo).ms
Est. durationMedian (p50) historical execution time for tasks of this type in this repository, based on completed work tasks from the last 7 days.PERCENTILE_CONT(0.5) of (completed_at - claimed_at) in seconds from the history of completed work tasks, grouped by (repo_id, executor_ref) over a trailing 7-day window. Only shown when the sample count is ≥ 5. Source: GET /pipeline/task-queue, estimatedDurationMs field per task. Tasks with no history show —.ms
Projected drainSum of estimated execution durations for all pending tasks that have historical duration data, giving a rough lower-bound on how long the queue will take to clear.Sum of estimatedDurationMs across pending tasks where the estimate is non-null. Tasks with no history are excluded from the sum; the hint is hidden when no pending task has an estimate. Source: GET /pipeline/task-queue response.ms
Write queueCombined count of pending and failed VCS write projections (label swaps, comments, PR operations) across all repos in the org.Count of VCS write projections with status 'pending' or 'failed' joined to repos belonging to this org. Source: /pipeline/operational-summary endpoint, vcsWriteQueueDepth and vcsWriteQueueFailed fields.count

Point-in-time counts for Workers and self-improvement tracks.

MetricDefinitionCalculationUnit
Worker countNumber of Colony Worker containers currently registered and reporting heartbeats to the Cloud API.Count of registered Workers with a recent heartbeat. Each Worker handles one pipeline issue at a time. Source: /api/orgs/{orgId}/workers endpoint.count
SI Tracks countNumber of self-improvement tracks configured across all repos in the org.A track is overdue if now - last_run_at > cooldown_minutes. Source: count summed across the per-repo GET /api/repos/{owner}/{repo}/si-tracks responses.count
Cadence / cooldownHow often a track may run — either a cooldown interval or a fixed cron schedule.Cooldown tracks re-run once now - last_run_at > cooldown_minutes (stored as cooldownMinutes). Cron tracks run on a fixed schedule defined by a cron expression (stored as cronExpr). Source: summed across the per-repo GET /api/repos/{owner}/{repo}/si-tracks responses, cadenceKind and cooldownMinutes/cronExpr fields.—

Planned-vs-actual analysis over completed issues.

MetricDefinitionCalculationUnit
Plan Drift RateFraction of completed issues in the window where the number of files Colony actually touched differed from the number it planned.(touchedMore + touchedFewer) / totalRetrospectivesWithFileData over the window, from the planned-vs-actual file sets of each retrospective.ratio
Outcome DistributionShare of completed issues by outcome classification (e.g. success / partial / failure) over the window.COUNT(*) GROUP BY outcome_classification over retrospectives in the window.count

Regression-guard signals computed from state transitions, compared against a rolling baseline window. These are per-repo metrics.

MetricDefinitionCalculationUnit
Block RateFraction of issues reaching done, failure-blocked or dependency-blocked in the rolling window whose latest such move was to a blocked state.blocked_count / total_issues over the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the rate rises — a higher block rate signals increasing pipeline impedance.ratio
Failure-Blocked RateFraction of those issues blocked for any reason other than a dependency.failure_blocked_count / total_issues over the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the rate rises — more CI failures indicate systematic test or build instability.ratio
Mean Cost / IssueMean LLM token spend (in USD) per issue completed in the rolling window, tracking cost efficiency of the autonomous pipeline.AVG(cost_usd_per_issue) for completed issues in the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the mean rises — increasing per-issue cost indicates the pipeline requires more LLM work per deliverable.usd
Mean Turns / IssueMean number of agent turns (LLM round-trips) consumed per issue completed in the rolling window, measuring implementation efficiency.AVG(turns_per_issue) for completed issues in the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the mean rises — more turns per issue signals increased difficulty or iterative rework.count
Reimplement Loop RateFraction of those issues blocked because they hit the re-implementation loop limit.reimplement_loop_count / total_issues over the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the rate rises — repeated re-implementations indicate systematic failure in planning or initial development quality.ratio
Dead Letter RateNew dead-letter entries in the rolling window, relative to issues reaching done, failure-blocked or dependency-blocked in that window.dead_letter_count / terminal_count over the rolling window computed from state transitions and stored per repository, compared against a baseline window mean. Regresses when the rate rises — more dead-lettered issues indicate the pipeline is unable to autonomously resolve an increasing share of work.ratio

Per-model LLM spend and execution quality signals for the selected window.

MetricDefinitionCalculationUnit
Model Spend ShareFraction of total org LLM spend attributable to this model in the selected window.model_cost / SUM(model_cost) OVER () for cost events in the window, grouped by model. Source: /pipeline/cost-by-model API, totalCostUsd relative to window total.percent
Max-Turns Rate by ModelFraction of invocations for this model that hit the agent turn limit, indicating tasks that exceeded the model’s context or reasoning budget.COUNT(*) FILTER (WHERE is_max_turns = true) / COUNT(*) for execution telemetry records in the window, grouped by model. Source: /pipeline/cost-by-model API, maxTurnsRate field.percent
No-Progress Rate by ModelFraction of invocations for this model that produced no meaningful code progress, indicating effort consumed without output.COUNT(*) FILTER (WHERE is_no_progress = true) / COUNT(*) for execution telemetry records in the window, grouped by model. Source: /pipeline/cost-by-model API, noProgressRate field.percent

MetricDefinitionCalculationUnit
Autonomy grant (max_auto_size)The largest track size the Strategist may apply automatically without operator approval, as set in the repository autonomy grant.Read from the max_auto_size field on GET /api/autonomy/:owner/:repo. Possible values: small, medium, large, epic. A repository with no grant configured defaults to small. The limit is enforced at approval time, against the grant as it stands when a snapshot is approved. Changes require an explicit grant update by an operator.—
Strategy snapshot statusThe lifecycle status of a Strategist-proposed strategy snapshot.Read from the status field on StrategySnapshot returned by GET /api/strategy/:owner/:repo/snapshots. A snapshot starts proposed. Approving it applies it at once and makes it active (the previous active one becomes superseded); declining sets it rejected; a refused plain approve sets it invalid; a charter change or a newer active strategy sets it superseded. With automatic approval Colony applies a valid proposal itself. There is no resting approved state.—

Cross-store audit findings that surface drift between Postgres pipeline state, dependency edges, SI linkage, and projected GitHub labels.

MetricDefinitionCalculationUnit
Data Consistency FindingsCount of unresolved data-consistency audit findings across the org — issues where Postgres pipeline state, dependency edges, SI linkage, or projected GitHub labels have drifted out of agreement.Count of unresolved findings grouped by kind: stale_blocked, stale_subtask_edge, orphan_missing_from_pg, si_linkage_gap, label_drift, orphan_dependency_edge. Source: the data-consistency audit.count
Stale BlockedIssues with is_blocked = true and state != 'done' but no active depends_on dependency edge — the blocked flag is stranded with no actual dependency blocking the issue.Queries issues for is_blocked = true AND state != 'done' with NOT EXISTS a dependency edge where dep_type = 'depends_on' AND status = 'active'. Source: the data-consistency audit, stale_blocked findings.count
Stale Subtask EdgeActive subtask dependency edges whose parent epic has reached done state — dangling edges not cleaned up when the epic completed.Queries dependency edges for dep_type = 'subtask' AND status = 'active', joined to the parent epic where state = 'done'. Grouped one finding per epic. Source: the data-consistency audit, stale_subtask_edge findings.count
Orphan (missing from Postgres)Open GitHub issues that have no corresponding pipeline issue — visible on GitHub but not yet ingested into the pipeline.Compares open GitHub issue numbers against the pipeline issues for the repo; issues absent from Postgres are flagged. Requires a live GitHub API call — this detector is deferred/optional in Cloud. Source: the data-consistency audit, orphan_missing_from_pg findings.count
SI Linkage GapSelf-improvement issues whose pipeline issue reached done but completed_at is still NULL — the SI completion event was not recorded.Queries self-improvement issues joined to their pipeline issues where pi.state = 'done' AND si.completed_at IS NULL. Source: the data-consistency audit, si_linkage_gap findings.count
Label DriftIssues where the Postgres authoritative state (state, is_blocked, is_paused) disagrees with the projected GitHub labels stored on the issue as labels — done issues excluded.Queries issues where state != 'done' and at least one label disagrees: stateToLabel(state) not in labels, is_blocked mismatches labels.includes(BLOCKED_LABEL), or is_paused mismatches labels.includes(PAUSED_LABEL). Source: the data-consistency audit, label_drift findings.count
Orphan Dependency EdgeDependency edges whose parent or child endpoint has no pipeline issue — the edge points at an issue the pipeline has no record of.Scans the repo’s dependency edges and flags any whose parent_issue or child_issue is absent from the pipeline issues, grouped by the missing endpoint. Source: the data-consistency audit, orphan_dependency_edge findings.count