Skip to content

Findings, Intelligence & Retrospectives

Colony agents record three kinds of persistent artifacts as they work through issues: Findings are structured observations emitted during a task. Repo Intelligence is a curated knowledge base built from promoted findings and retrospective lessons. Retrospectives are per-issue post-mortems written after an issue reaches done.

Together these artifacts give operators visibility into what agents learned, what risks were spotted, and why the codebase looks the way it does.


A finding is a structured observation emitted by an agent during a task run. Findings capture risks, quality signals, and actionable recommendations anchored to a specific issue or PR.

AgentTypical findings
analyzerScope creep signals, missing context, dependency risks
developerCode smells, test gaps, security observations noticed during impl
reviewerSecurity issues, correctness problems, style violations

Every finding carries a severity level:

SeverityMeaning
infoInformational note; no action required
lowMinor issue worth addressing eventually
mediumModerate risk; should be addressed before merge
highSignificant risk; blocks autonomous merge in many workflows
criticalSevere issue; requires immediate human attention

Findings move through the following statuses:

StatusMeaning
openNewly recorded; not yet acted on
acceptedAcknowledged as valid; tracked for future resolution
resolvedFixed or addressed — resolvedByIssueId points to the fix
dismissedIntentionally closed without action; dismissedReason recorded
supersededReplaced by a newer finding covering the same observation

When blocksProgress is true, the finding signals that the current agent believes the issue cannot advance until this observation is addressed. The pipeline does not enforce this flag automatically — it is advisory metadata for operator review and for future workflow rules.

CLI: colony inspect <issue-number> --findings prints findings from Postgres for the specified issue.

API (per-issue): GET /api/issues/:owner/:repo/:issue/findings returns paginated AgentFinding objects for a single issue. Supports limit and offset query parameters.

API (repo-wide): GET /api/findings?repo=owner/repo lists all findings for a repository. Filter by status, severity, or category. See the API Reference for the full field reference.


Repo Intelligence (the v2 knowledge surface) is a curated set of durable observations about a repository — architectural decisions, invariants, failure patterns, and more. Items are promoted from findings and retrospective lessons, accumulating a persistent picture of what the repo’s agents have learned over time.

KindMeaning
architectureHigh-level structural decision (e.g. layering, module boundaries)
invariantRule that must always hold (e.g. “no direct DB calls from handlers”)
workflow_playbookStep-by-step procedure for a recurring task
test_strategyHow tests are organized, what to test, which frameworks to use
failure_patternRecurrent failure shape and its root cause
couplingUnexpected or intentional dependency between two modules or services
operator_preferenceHuman-expressed preference about how agents should behave
design_decisionRationale behind a past architectural or implementation choice
risk_areaCode region or behavior that warrants extra scrutiny
implementation_noteLow-level implementation detail worth remembering across tasks

Intelligence items progress through the following statuses:

StatusMeaning
candidateNominated from a single observation; not yet corroborated
observedSeen multiple times; accumulating evidence
proposedPromoted to a recommendation awaiting operator approval
approvedConfirmed by an operator; treated as authoritative by agents
dismissedRejected as inaccurate or irrelevant; dismissedReason recorded
supersededReplaced by a newer item; supersededBy points to the replacement

Each intelligence item can carry one or more scope links that anchor it to a specific part of the codebase:

Scope kindExample scopeRefMeaning
repoowner/repoApplies to the entire repository
packagepackages/coreApplies to a specific package
pathsrc/auth/Applies to a directory subtree
filesrc/auth/middleware.tsApplies to a single file
symbolAuthMiddleware.handleApplies to a specific function or class
routePOST /api/auth/loginApplies to an HTTP route
testsrc/__tests__/auth.test.tsApplies to a test file or suite
serviceworkerApplies to a named service or process
workflowcolony-defaultApplies to a named workflow definition

Items link back to the artifacts that support them via IntelligenceEvidence records. Evidence types include: issue, pr, finding, retrospective, file, commit, ci, test, log, human.

Every item carries a confidence score (0–1) representing how strongly the available evidence supports the observation. Auto-promotion is performed exclusively by the retrospect executor when it writes candidate intelligence items. When all four conditions hold, the executor sets the item’s status directly to approved (with approvedBy: 'auto:retrospector'):

  1. intelligence.auto_promote.enabled is true
  2. The retrospector’s LLM independently sets should_promote: true for the item
  3. confidence >= confidence_threshold (default: 0.8)
  4. observationCount >= min_observation_count (default: 2)

Configure auto-promotion under the intelligence.auto_promote block in colony.config.yaml. See the Configuration Reference for field details.

intelligence:
auto_promote:
enabled: true
confidence_threshold: 0.85 # default: 0.8
min_observation_count: 3 # default: 2

When enabled is false (the default), candidate items remain at candidate status — they accumulate evidence and confidence but are not approved automatically. Operators review and approve them manually via the API or dashboard.

API (list): GET /api/intelligence?repo=owner/repo — lists items without scopes or evidence arrays. Supports kind, status, scope_kind, scope_ref, tag, source, and min_confidence filters.

API (detail): GET /api/intelligence/:id — returns a single item with full scopes and evidence arrays. Use this after finding an item of interest in the list response.

See the API Reference for the complete field reference.


A retrospective is a per-issue post-mortem written by the retrospect task executor after an issue reaches done. It captures outcome, file changes, and lessons learned — and is the primary seeding path for new Repo Intelligence items.

FieldDescription
outcomeClassificationFree-form outcome label (e.g. "success", "failure", "partial")
summaryHuman-readable narrative of what happened
plannedFilesMap of file path → action the agent planned to touch (from the analysis phase)
actualFilesMap of file path → action the agent actually modified (from git diff at close)
plannedActualDeltaStructured diff between planned and actual file changes
failuresStructured record of failures encountered during the issue lifecycle, if any
lessonsLessons extracted from the issue run; promoted into Repo Intelligence
candidateIntelligenceDraft intelligence items nominated by the retrospect executor for promotion

outcomeClassification is a free-form string, not a closed enum — values evolve as the executor’s reasoning improves. Do not pattern-match on it for automation; treat it as a human-readable label.

The retrospective executor (retrospect task type) reads the issue’s full history, derives lessons from it, and writes both the retrospective record and zero or more candidate intelligence items. Those candidates enter the intelligence pipeline at candidate status and advance toward approved as corroborating evidence accumulates across future issues.

API: GET /api/issues/:owner/:repo/:issue/retrospective returns the retrospective record for a completed issue, or null when none has been written yet (e.g. the issue is still in progress or the retrospect task has not run).

See the API Reference for the complete field reference.


Issue run
│
├─► Finding (emitted by analyzer / developer / reviewer)
│ │
│ └─► promotes to ──► Repo Intelligence item (candidate status)
│
└─► Retrospective (written by retrospect task after done)
│
├─ lessons ──────────► Repo Intelligence item (candidate status)
└─ candidateIntelligence ──► Repo Intelligence item (candidate status)
│
└─► auto-promoted to approved
(retrospect executor, when all 4 conditions met)
or manually approved by operator

Findings and retrospective lessons flow into the same Repo Intelligence store. A sourceRetrospectiveId or sourceFindingId on an intelligence item traces it back to its origin. Multiple observations of the same pattern raise observationCount and confidence, contributing to the auto-promotion conditions when enabled.


When you first enable intelligence.embedding in your config, existing repo_intelligence_items rows have embedding = NULL. Run the backfill script once to populate embeddings for all historical items.

After adding or enabling the intelligence.embedding block in colony.config.yaml for the first time. Re-running is always safe — the script targets only rows where embedding IS NULL, so already-embedded rows are never re-processed.

Terminal window
DATABASE_URL=postgres://... OPENAI_API_KEY=sk-... \
npx tsx packages/pipeline-store/src/scripts/backfill-intelligence-embeddings.ts
VariablePurpose
DATABASE_URLPostgres connection string to Pipeline Store
OPENAI_API_KEYOpenAI API key (or the env var named in intelligence.embedding.api_key_env)
FlagDefaultDescription
--batch-size <n>200Rows per embedding API request
--limit <n>noneStop after embedding this many rows (useful for testing)

~7,700 items × 20 tokens each ≈ 154,000 tokens at text-embedding-3-small pricing ($0.02 / 1M tokens) ≈ under $0.01 for a full backfill.

If a colony.config.yaml is present in the current directory or at ~/.colony/config.yaml, the script reads intelligence.embedding from it to pick up custom model, dimensions, and api_key_env settings. If no config file is found the script uses defaults (text-embedding-3-small, 1536 dimensions, OPENAI_API_KEY).


Retrieval ranks over every active item, so a claim that is stored several times spends the intelligence.retrieval.token_budget several times and displaces a distinct item. Write-time reinforcement compares one incoming item with the corpus; it never compares two existing items with each other. The consolidation script does: it clusters the stored embeddings against themselves and collapses semantically-equivalent items into one canonical item.

Consolidation is operator-run only. Nothing runs it automatically — not on merge, deploy, boot or migration, and not from any sweep, cron, retrospective or config flag. There is no config key for it. The only way to change data is to run the script with --apply.

  • Run the embedding backfill first if a dry-run reports nullEmbeddingSkipped > 0. Items without an embedding are skipped and counted, never merged.
  • Always review a dry-run before --apply. The default run only reads and prints a JSON report. Check the clusters, the oversized and conflicts lists, and the reviewPairs.
  • There is no undo tool. Merged members are kept as superseded rows, and the canonical item records their ids in metadata.consolidated_from.
Terminal window
# Dry-run (default): reads only, prints the report
DATABASE_URL=postgres://... \
npx tsx packages/pipeline-store/src/scripts/consolidate-intelligence-items.ts
# Apply to one repository
DATABASE_URL=postgres://... \
npx tsx packages/pipeline-store/src/scripts/consolidate-intelligence-items.ts \
--apply --repo owner/name
# Apply to every repository
DATABASE_URL=postgres://... \
npx tsx packages/pipeline-store/src/scripts/consolidate-intelligence-items.ts \
--apply --all-repos

--apply requires an explicit target (--repo owner/name or --all-repos). With neither, the script exits non-zero and changes nothing. Before writing, it prints the same report a dry-run would print, then prints a second report with what was actually written.

FlagDefaultDescription
--applyoffWrite changes. Requires --repo or --all-repos
--repo <owner/name>allRestrict to one repository
--all-reposoffExplicit target for --apply across all repositories
--kind <kind>allRestrict to one item kind
--link-distance <d>0.13Cosine distance at or under which two items are linked into a cluster
--review-distance <d>0.15Pairs above --link-distance and up to this are reported, never merged
--max-cluster-size <n>8Larger connected components are reported as oversized and not merged
--batch-size <n>200Anchor items compared against the rest of their group per query
--statement-timeout <sec>30statement_timeout for each batched query
--limit <n>noneWith --apply, merge at most this many clusters
  • Only items in the statuses retrieval reads (candidate, observed, proposed, approved) that have an embedding are considered, grouped by (repo, kind). Nothing is merged across kinds or repositories.
  • Distances are exact pairwise cosine distances, collected in batches of anchors.
  • Clusters are transitive (single-linkage): A–B and B–C at 0.12 form one cluster even if A–C is 0.24.
  • The canonical item is the only approved item in the cluster. Otherwise it is the highest quality_score, then confidence (nulls last), observation_count, earliest first_observed_at, and lowest id.
  • On the canonical item: tags become the sorted union, scopes and evidence are copied (identical evidence is skipped), observation_count is summed, observed timestamps span the cluster, and member ids are appended to metadata.consolidated_from. Its title, summary, details and embedding are unchanged. No model is called.
  • Each member becomes superseded with superseded_by set to the canonical item. Anything that already pointed at a member is repointed to the canonical item.

Reported per (repo, kind) group under groups, and summed under totals.

FieldMeaning
activeEmbeddedBeforeActive items with an embedding before the run
activeEmbeddedAfterSame, after the planned (dry-run) or actual (apply) merges
clustersMergedClusters merged (planned in a dry-run)
itemsSupersededMembers superseded (planned in a dry-run)
clustersSkippedApply only: clusters not merged because a member changed since planning
reviewPairsPairs between --link-distance and --review-distance that did not land in one cluster, with both titles and the distance. Never merged; for a human to judge
oversizedConnected components larger than --max-cluster-size (chaining guard). Not merged
conflictsClusters with two or more approved items. Not merged
nullEmbeddingSkippedActive items with no embedding. Run the backfill, then re-run

The report’s stage is plan for the pre-write report and result for what --apply wrote.