Findings, Intelligence & Retrospectives
Findings, Intelligence & Retrospectives
Section titled “Findings, Intelligence & Retrospectives”Colony agents record three kinds of persistent artifacts as they work through issues:
Findings are structured observations emitted during a task. Repo Intelligence is a
curated knowledge base built from promoted findings and retrospective lessons.
Retrospectives are per-issue post-mortems written after an issue reaches done.
Together these artifacts give operators visibility into what agents learned, what risks were spotted, and why the codebase looks the way it does.
Findings
Section titled “Findings”A finding is a structured observation emitted by an agent during a task run. Findings capture risks, quality signals, and actionable recommendations anchored to a specific issue or PR.
Which agents emit findings
Section titled “Which agents emit findings”| Agent | Typical findings |
|---|---|
analyzer | Scope creep signals, missing context, dependency risks |
developer | Code smells, test gaps, security observations noticed during impl |
reviewer | Security issues, correctness problems, style violations |
Severity
Section titled “Severity”Every finding carries a severity level:
| Severity | Meaning |
|---|---|
info | Informational note; no action required |
low | Minor issue worth addressing eventually |
medium | Moderate risk; should be addressed before merge |
high | Significant risk; blocks autonomous merge in many workflows |
critical | Severe issue; requires immediate human attention |
Status lifecycle
Section titled “Status lifecycle”Findings move through the following statuses:
| Status | Meaning |
|---|---|
open | Newly recorded; not yet acted on |
accepted | Acknowledged as valid; tracked for future resolution |
resolved | Fixed or addressed — resolvedByIssueId points to the fix |
dismissed | Intentionally closed without action; dismissedReason recorded |
superseded | Replaced by a newer finding covering the same observation |
blocksProgress
Section titled “blocksProgress”When blocksProgress is true, the finding signals that the current agent believes the
issue cannot advance until this observation is addressed. The pipeline does not enforce this
flag automatically — it is advisory metadata for operator review and for future workflow
rules.
Inspecting findings
Section titled “Inspecting findings”CLI: colony inspect <issue-number> --findings prints findings from Postgres for the
specified issue.
API (per-issue): GET /api/issues/:owner/:repo/:issue/findings returns paginated
AgentFinding objects for a single issue. Supports limit and offset query parameters.
API (repo-wide): GET /api/findings?repo=owner/repo lists all findings for a
repository. Filter by status, severity, or category. See the
API Reference for the full field reference.
Repo Intelligence
Section titled “Repo Intelligence”Repo Intelligence (the v2 knowledge surface) is a curated set of durable observations about a repository — architectural decisions, invariants, failure patterns, and more. Items are promoted from findings and retrospective lessons, accumulating a persistent picture of what the repo’s agents have learned over time.
Item kinds (IntelligenceKind)
Section titled “Item kinds (IntelligenceKind)”| Kind | Meaning |
|---|---|
architecture | High-level structural decision (e.g. layering, module boundaries) |
invariant | Rule that must always hold (e.g. “no direct DB calls from handlers”) |
workflow_playbook | Step-by-step procedure for a recurring task |
test_strategy | How tests are organized, what to test, which frameworks to use |
failure_pattern | Recurrent failure shape and its root cause |
coupling | Unexpected or intentional dependency between two modules or services |
operator_preference | Human-expressed preference about how agents should behave |
design_decision | Rationale behind a past architectural or implementation choice |
risk_area | Code region or behavior that warrants extra scrutiny |
implementation_note | Low-level implementation detail worth remembering across tasks |
Status lifecycle
Section titled “Status lifecycle”Intelligence items progress through the following statuses:
| Status | Meaning |
|---|---|
candidate | Nominated from a single observation; not yet corroborated |
observed | Seen multiple times; accumulating evidence |
proposed | Promoted to a recommendation awaiting operator approval |
approved | Confirmed by an operator; treated as authoritative by agents |
dismissed | Rejected as inaccurate or irrelevant; dismissedReason recorded |
superseded | Replaced by a newer item; supersededBy points to the replacement |
Scopes
Section titled “Scopes”Each intelligence item can carry one or more scope links that anchor it to a specific part of the codebase:
| Scope kind | Example scopeRef | Meaning |
|---|---|---|
repo | owner/repo | Applies to the entire repository |
package | packages/core | Applies to a specific package |
path | src/auth/ | Applies to a directory subtree |
file | src/auth/middleware.ts | Applies to a single file |
symbol | AuthMiddleware.handle | Applies to a specific function or class |
route | POST /api/auth/login | Applies to an HTTP route |
test | src/__tests__/auth.test.ts | Applies to a test file or suite |
service | worker | Applies to a named service or process |
workflow | colony-default | Applies to a named workflow definition |
Evidence
Section titled “Evidence”Items link back to the artifacts that support them via IntelligenceEvidence records.
Evidence types include: issue, pr, finding, retrospective, file, commit, ci,
test, log, human.
Confidence and auto-promotion
Section titled “Confidence and auto-promotion”Every item carries a confidence score (0–1) representing how strongly the available
evidence supports the observation. Auto-promotion is performed exclusively by the
retrospect executor when it writes candidate intelligence items. When all four conditions
hold, the executor sets the item’s status directly to approved (with approvedBy: 'auto:retrospector'):
intelligence.auto_promote.enabledistrue- The retrospector’s LLM independently sets
should_promote: truefor the item confidence >= confidence_threshold(default:0.8)observationCount >= min_observation_count(default:2)
Configure auto-promotion under the intelligence.auto_promote block in
colony.config.yaml. See the Configuration Reference for
field details.
intelligence: auto_promote: enabled: true confidence_threshold: 0.85 # default: 0.8 min_observation_count: 3 # default: 2When enabled is false (the default), candidate items remain at candidate status —
they accumulate evidence and confidence but are not approved automatically. Operators review
and approve them manually via the API or dashboard.
Inspecting intelligence
Section titled “Inspecting intelligence”API (list): GET /api/intelligence?repo=owner/repo — lists items without scopes or
evidence arrays. Supports kind, status, scope_kind, scope_ref, tag, source,
and min_confidence filters.
API (detail): GET /api/intelligence/:id — returns a single item with full scopes
and evidence arrays. Use this after finding an item of interest in the list response.
See the API Reference for the complete field reference.
Retrospectives
Section titled “Retrospectives”A retrospective is a per-issue post-mortem written by the retrospect task executor
after an issue reaches done. It captures outcome, file changes, and lessons learned — and
is the primary seeding path for new Repo Intelligence items.
What a retrospective contains
Section titled “What a retrospective contains”| Field | Description |
|---|---|
outcomeClassification | Free-form outcome label (e.g. "success", "failure", "partial") |
summary | Human-readable narrative of what happened |
plannedFiles | Map of file path → action the agent planned to touch (from the analysis phase) |
actualFiles | Map of file path → action the agent actually modified (from git diff at close) |
plannedActualDelta | Structured diff between planned and actual file changes |
failures | Structured record of failures encountered during the issue lifecycle, if any |
lessons | Lessons extracted from the issue run; promoted into Repo Intelligence |
candidateIntelligence | Draft intelligence items nominated by the retrospect executor for promotion |
outcomeClassification is a free-form string, not a closed enum — values evolve as the
executor’s reasoning improves. Do not pattern-match on it for automation; treat it as a
human-readable label.
Promotion path
Section titled “Promotion path”The retrospective executor (retrospect task type) reads the issue’s full history, derives
lessons from it, and writes both the retrospective record and zero or more candidate
intelligence items. Those candidates enter the intelligence pipeline at candidate status
and advance toward approved as corroborating evidence accumulates across future issues.
Inspecting retrospectives
Section titled “Inspecting retrospectives”API: GET /api/issues/:owner/:repo/:issue/retrospective returns the retrospective
record for a completed issue, or null when none has been written yet (e.g. the issue is
still in progress or the retrospect task has not run).
See the API Reference for the complete field reference.
How the artifacts relate
Section titled “How the artifacts relate”Issue run │ ├─► Finding (emitted by analyzer / developer / reviewer) │ │ │ └─► promotes to ──► Repo Intelligence item (candidate status) │ └─► Retrospective (written by retrospect task after done) │ ├─ lessons ──────────► Repo Intelligence item (candidate status) └─ candidateIntelligence ──► Repo Intelligence item (candidate status) │ └─► auto-promoted to approved (retrospect executor, when all 4 conditions met) or manually approved by operatorFindings and retrospective lessons flow into the same Repo Intelligence store.
A sourceRetrospectiveId or sourceFindingId on an intelligence item traces it back to
its origin. Multiple observations of the same pattern raise observationCount and
confidence, contributing to the auto-promotion conditions when enabled.
Embedding backfill
Section titled “Embedding backfill”When you first enable intelligence.embedding in your config, existing
repo_intelligence_items rows have embedding = NULL. Run the backfill script once to
populate embeddings for all historical items.
When to run
Section titled “When to run”After adding or enabling the intelligence.embedding block in colony.config.yaml for
the first time. Re-running is always safe — the script targets only rows where
embedding IS NULL, so already-embedded rows are never re-processed.
Command
Section titled “Command”DATABASE_URL=postgres://... OPENAI_API_KEY=sk-... \ npx tsx packages/pipeline-store/src/scripts/backfill-intelligence-embeddings.tsRequired environment variables
Section titled “Required environment variables”| Variable | Purpose |
|---|---|
DATABASE_URL | Postgres connection string to Pipeline Store |
OPENAI_API_KEY | OpenAI API key (or the env var named in intelligence.embedding.api_key_env) |
Optional flags
Section titled “Optional flags”| Flag | Default | Description |
|---|---|---|
--batch-size <n> | 200 | Rows per embedding API request |
--limit <n> | none | Stop after embedding this many rows (useful for testing) |
Estimated cost
Section titled “Estimated cost”~7,700 items × 20 tokens each ≈ 154,000 tokens at $0.02 / 1M tokens) ≈ under $0.01 for a full backfill.text-embedding-3-small pricing
(
Config file (optional)
Section titled “Config file (optional)”If a colony.config.yaml is present in the current directory or at ~/.colony/config.yaml,
the script reads intelligence.embedding from it to pick up custom model, dimensions,
and api_key_env settings. If no config file is found the script uses defaults
(text-embedding-3-small, 1536 dimensions, OPENAI_API_KEY).
Consolidation
Section titled “Consolidation”Retrieval ranks over every active item, so a claim that is stored several times spends the
intelligence.retrieval.token_budget several times and displaces a distinct item. Write-time
reinforcement compares one incoming item with the corpus; it never compares two existing items
with each other. The consolidation script does: it clusters the stored embeddings against
themselves and collapses semantically-equivalent items into one canonical item.
Consolidation is operator-run only. Nothing runs it automatically — not on merge, deploy,
boot or migration, and not from any sweep, cron, retrospective or config flag. There is no
config key for it. The only way to change data is to run the script with --apply.
Before you run it
Section titled “Before you run it”- Run the embedding backfill first if a dry-run reports
nullEmbeddingSkipped > 0. Items without an embedding are skipped and counted, never merged. - Always review a dry-run before
--apply. The default run only reads and prints a JSON report. Check the clusters, theoversizedandconflictslists, and thereviewPairs. - There is no undo tool. Merged members are kept as
supersededrows, and the canonical item records their ids inmetadata.consolidated_from.
Commands
Section titled “Commands”# Dry-run (default): reads only, prints the reportDATABASE_URL=postgres://... \ npx tsx packages/pipeline-store/src/scripts/consolidate-intelligence-items.ts
# Apply to one repositoryDATABASE_URL=postgres://... \ npx tsx packages/pipeline-store/src/scripts/consolidate-intelligence-items.ts \ --apply --repo owner/name
# Apply to every repositoryDATABASE_URL=postgres://... \ npx tsx packages/pipeline-store/src/scripts/consolidate-intelligence-items.ts \ --apply --all-repos--apply requires an explicit target (--repo owner/name or --all-repos). With neither, the
script exits non-zero and changes nothing. Before writing, it prints the same report a dry-run
would print, then prints a second report with what was actually written.
| Flag | Default | Description |
|---|---|---|
--apply | off | Write changes. Requires --repo or --all-repos |
--repo <owner/name> | all | Restrict to one repository |
--all-repos | off | Explicit target for --apply across all repositories |
--kind <kind> | all | Restrict to one item kind |
--link-distance <d> | 0.13 | Cosine distance at or under which two items are linked into a cluster |
--review-distance <d> | 0.15 | Pairs above --link-distance and up to this are reported, never merged |
--max-cluster-size <n> | 8 | Larger connected components are reported as oversized and not merged |
--batch-size <n> | 200 | Anchor items compared against the rest of their group per query |
--statement-timeout <sec> | 30 | statement_timeout for each batched query |
--limit <n> | none | With --apply, merge at most this many clusters |
How it works
Section titled “How it works”- Only items in the statuses retrieval reads (
candidate,observed,proposed,approved) that have an embedding are considered, grouped by(repo, kind). Nothing is merged across kinds or repositories. - Distances are exact pairwise cosine distances, collected in batches of anchors.
- Clusters are transitive (single-linkage): A–B and B–C at 0.12 form one cluster even if A–C is 0.24.
- The canonical item is the only
approveditem in the cluster. Otherwise it is the highestquality_score, thenconfidence(nulls last),observation_count, earliestfirst_observed_at, and lowestid. - On the canonical item: tags become the sorted union, scopes and evidence are copied (identical
evidence is skipped),
observation_countis summed, observed timestamps span the cluster, and member ids are appended tometadata.consolidated_from. Itstitle,summary,detailsandembeddingare unchanged. No model is called. - Each member becomes
supersededwithsuperseded_byset to the canonical item. Anything that already pointed at a member is repointed to the canonical item.
Report fields
Section titled “Report fields”Reported per (repo, kind) group under groups, and summed under totals.
| Field | Meaning |
|---|---|
activeEmbeddedBefore | Active items with an embedding before the run |
activeEmbeddedAfter | Same, after the planned (dry-run) or actual (apply) merges |
clustersMerged | Clusters merged (planned in a dry-run) |
itemsSuperseded | Members superseded (planned in a dry-run) |
clustersSkipped | Apply only: clusters not merged because a member changed since planning |
reviewPairs | Pairs between --link-distance and --review-distance that did not land in one cluster, with both titles and the distance. Never merged; for a human to judge |
oversized | Connected components larger than --max-cluster-size (chaining guard). Not merged |
conflicts | Clusters with two or more approved items. Not merged |
nullEmbeddingSkipped | Active items with no embedding. Run the backfill, then re-run |
The report’s stage is plan for the pre-write report and result for what --apply wrote.