Findings, Intelligence & Retrospectives

Findings, Intelligence & Retrospectives

Colony agents record three kinds of persistent artifacts as they work through issues: Findings are structured observations emitted during a task. Repo Intelligence is a curated knowledge base built from promoted findings and retrospective lessons. Retrospectives are per-issue post-mortems written after an issue reaches done.

Together these artifacts give operators visibility into what agents learned, what risks were spotted, and why the codebase looks the way it does.

---

Findings

A finding is a structured observation emitted by an agent during a task run. Findings capture risks, quality signals, and actionable recommendations anchored to a specific issue or PR.

Which agents emit findings

Agent	Typical findings
analyzer	Scope creep signals, missing context, dependency risks
developer	Code smells, test gaps, security observations noticed during impl
reviewer	Security issues, correctness problems, style violations

Severity

Every finding carries a severity level:

Severity	Meaning
info	Informational note; no action required
low	Minor issue worth addressing eventually
medium	Moderate risk; should be addressed before merge
high	Significant risk; blocks autonomous merge in many workflows
critical	Severe issue; requires immediate human attention

Status lifecycle

Findings move through the following statuses:

Status	Meaning
open	Newly recorded; not yet acted on
accepted	Acknowledged as valid; tracked for future resolution
resolved	Fixed or addressed — resolvedByIssueId points to the fix
dismissed	Intentionally closed without action; dismissedReason recorded
superseded	Replaced by a newer finding covering the same observation

blocksProgress

When blocksProgress is true, the finding signals that the current agent believes the issue cannot advance until this observation is addressed. The pipeline does not enforce this flag automatically — it is advisory metadata for operator review and for future workflow rules.

Inspecting findings

CLI: colony inspect <issue-number> --findings prints findings from Postgres for the specified issue.

API (per-issue): GET /api/issues/:owner/:repo/:issue/findings returns paginated AgentFinding objects for a single issue. Supports limit and offset query parameters.

API (repo-wide): GET /api/findings?repo=owner/repo lists all findings for a repository. Filter by status, severity, or category. See the API Reference for the full field reference.

---

Repo Intelligence

Repo Intelligence (the v2 knowledge surface) is a curated set of durable observations about a repository — architectural decisions, invariants, failure patterns, and more. Items are promoted from findings and retrospective lessons, accumulating a persistent picture of what the repo’s agents have learned over time.

Item kinds (IntelligenceKind)

Kind	Meaning
architecture	High-level structural decision (e.g. layering, module boundaries)
invariant	Rule that must always hold (e.g. “no direct DB calls from handlers”)
workflow_playbook	Step-by-step procedure for a recurring task
test_strategy	How tests are organized, what to test, which frameworks to use
failure_pattern	Recurrent failure shape and its root cause
coupling	Unexpected or intentional dependency between two modules or services
operator_preference	Human-expressed preference about how agents should behave
design_decision	Rationale behind a past architectural or implementation choice
risk_area	Code region or behavior that warrants extra scrutiny
implementation_note	Low-level implementation detail worth remembering across tasks

Status lifecycle

Intelligence items progress through the following statuses:

Status	Meaning
candidate	Nominated from a single observation; not yet corroborated
observed	Seen multiple times; accumulating evidence
proposed	Promoted to a recommendation awaiting operator approval
approved	Confirmed by an operator; treated as authoritative by agents
dismissed	Rejected as inaccurate or irrelevant; dismissedReason recorded
superseded	Replaced by a newer item; supersededBy points to the replacement

Scopes

Each intelligence item can carry one or more scope links that anchor it to a specific part of the codebase:

Scope kind	Example scopeRef	Meaning
repo	owner/repo	Applies to the entire repository
package	packages/core	Applies to a specific package
path	src/auth/	Applies to a directory subtree
file	src/auth/middleware.ts	Applies to a single file
symbol	AuthMiddleware.handle	Applies to a specific function or class
route	POST /api/auth/login	Applies to an HTTP route
test	src/__tests__/auth.test.ts	Applies to a test file or suite
service	worker	Applies to a named service or process
workflow	colony-default	Applies to a named workflow definition

Evidence

Items link back to the artifacts that support them via IntelligenceEvidence records. Evidence types include: issue, pr, finding, retrospective, file, commit, ci, test, log, human.

Confidence and auto-promotion

Every item carries a confidence score (0–1) representing how strongly the available evidence supports the observation. Auto-promotion is performed exclusively by the retrospect executor when it writes candidate intelligence items. When all four conditions hold, the executor sets the item’s status directly to approved (with approvedBy: 'auto:retrospector'):

1. intelligence.auto_promote.enabled is true
2. The retrospector’s LLM independently sets should_promote: true for the item
3. confidence >= confidence_threshold (default: 0.8)
4. observationCount >= min_observation_count (default: 2)

Configure auto-promotion under the intelligence.auto_promote block in colony.config.yaml. See the Configuration Reference for field details.

intelligence:
  auto_promote:
    enabled: true
    confidence_threshold: 0.85   # default: 0.8
    min_observation_count: 3     # default: 2

When enabled is false (the default), candidate items remain at candidate status — they accumulate evidence and confidence but are not approved automatically. Operators review and approve them manually via the API or dashboard.

Inspecting intelligence

API (list): GET /api/intelligence?repo=owner/repo — lists items without scopes or evidence arrays. Supports kind, status, scope_kind, scope_ref, tag, source, and min_confidence filters.

API (detail): GET /api/intelligence/:id — returns a single item with full scopes and evidence arrays. Use this after finding an item of interest in the list response.

See the API Reference for the complete field reference.

---

Retrospectives

A retrospective is a per-issue post-mortem written by the retrospect task executor after an issue reaches done. It captures outcome, file changes, and lessons learned — and is the primary seeding path for new Repo Intelligence items.

What a retrospective contains

Field	Description
outcomeClassification	Free-form outcome label (e.g. "success", "failure", "partial")
summary	Human-readable narrative of what happened
plannedFiles	Map of file path → action the agent planned to touch (from the analysis phase)
actualFiles	Map of file path → action the agent actually modified (from git diff at close)
plannedActualDelta	Structured diff between planned and actual file changes
failures	Structured record of failures encountered during the issue lifecycle, if any
lessons	Lessons extracted from the issue run; promoted into Repo Intelligence
candidateIntelligence	Draft intelligence items nominated by the retrospect executor for promotion

outcomeClassification is a free-form string, not a closed enum — values evolve as the executor’s reasoning improves. Do not pattern-match on it for automation; treat it as a human-readable label.

Promotion path

The retrospective executor (retrospect task type) reads the issue’s full history, derives lessons from it, and writes both the retrospective record and zero or more candidate intelligence items. Those candidates enter the intelligence pipeline at candidate status and advance toward approved as corroborating evidence accumulates across future issues.

Inspecting retrospectives

API: GET /api/issues/:owner/:repo/:issue/retrospective returns the retrospective record for a completed issue, or null when none has been written yet (e.g. the issue is still in progress or the retrospect task has not run).

See the API Reference for the complete field reference.

---

How the artifacts relate

Issue run
  │
  ├─► Finding (emitted by analyzer / developer / reviewer)
  │       │
  │       └─► promotes to ──► Repo Intelligence item (candidate status)
  │
  └─► Retrospective (written by retrospect task after done)
          │
          ├─ lessons ──────────► Repo Intelligence item (candidate status)
          └─ candidateIntelligence ──► Repo Intelligence item (candidate status)
                                            │
                                            └─► auto-promoted to approved
                                                (retrospect executor, when all 4 conditions met)
                                                or manually approved by operator

Findings and retrospective lessons flow into the same Repo Intelligence store. A sourceRetrospectiveId or sourceFindingId on an intelligence item traces it back to its origin. Multiple observations of the same pattern raise observationCount and confidence, contributing to the auto-promotion conditions when enabled.

---

Embedding backfill

When you first enable intelligence.embedding in your config, existing repo_intelligence_items rows have embedding = NULL. Run the backfill script once to populate embeddings for all historical items.

When to run

After adding or enabling the intelligence.embedding block in colony.config.yaml for the first time. Re-running is always safe — the script targets only rows where embedding IS NULL, so already-embedded rows are never re-processed.

Command

DATABASE_URL=postgres://... OPENAI_API_KEY=sk-... \
  npx tsx packages/pipeline-store/src/scripts/backfill-intelligence-embeddings.ts

Required environment variables

Variable	Purpose
DATABASE_URL	Postgres connection string to Pipeline Store
OPENAI_API_KEY	OpenAI API key (or the env var named in intelligence.embedding.api_key_env)

Optional flags

Flag	Default	Description
--batch-size <n>	200	Rows per embedding API request
--limit <n>	none	Stop after embedding this many rows (useful for testing)

Estimated cost

~7,700 items × 20 tokens each ≈ 154,000 tokens at text-embedding-3-small pricing ($0.02 / 1M tokens) ≈ under $0.01 for a full backfill.

Config file (optional)

If a colony.config.yaml is present in the current directory or at ~/.colony/config.yaml, the script reads intelligence.embedding from it to pick up custom model, dimensions, and api_key_env settings. If no config file is found the script uses defaults (text-embedding-3-small, 1536 dimensions, OPENAI_API_KEY).