First Working Version

The first version that actually worked was not a feature; it was the pipeline, complete and boring. Drop a PDF in inbox/ and, some time later, a source, a tree, a handful of claims, and one or two wiki pages exist—each with provenance, each audited. That was the moment the project became real.

The Five Stages

Intake hashes the bytes, stores them content-addressed (identical bytes are a no-op), and records the source. Nothing is edited, ever.

Extract turns bytes into sections. Markdown and text are trivial; PDF uses pdftotext -layout for the text layer, and a page whose text is shorter than a threshold is treated as scanned and OCR’d page by page—pdftocairo renders one page to stdout, piped straight into tesseract. No temp files, no daemon, and the process is gone when the call returns. Text that came from OCR is marked ocr: true in the provenance, so a lossy extraction is always visible.

Build tree produces the PageIndex hierarchy—headings for Markdown, page and heading heuristics for PDF, paragraph groups for TXT—and the model writes a title and summary per node. The tree is a JSON sidecar plus source_nodes rows; both are derived and rebuildable.

Atomize runs the agent over the tree: one claim per atom, a branch, {hash, node, ocr} provenance, cross-links to existing atoms.

Compile runs the agent again: group atoms into pages, set atoms: provenance (mandatory), cross-link pages, regenerate the per-topic indexes and append to wiki/log.md.

Status Is Derived, Never Stored

The rule I am proudest of from this phase: there is no per-source “ingested” flag. The status is computed from what exists—does it have a tree? does it have atoms? Two triggers follow from that:

1. On add—a dropped file or an API upload intakes, builds a tree, and if the source has no atoms yet, schedules an ingest.
2. On start—a sweep builds trees for sources that never got one, then ingests every source with a tree and no atoms. A failed ingest is simply still “pending” and gets retried by the next sweep.

The runner keeps an in-flight set so a document is never ingested twice or concurrently; a manual re-ingest of a running document gets a 409. Derived status plus an in-flight set covers everything a stored flag would, without the flag ever going stale.

Runs Are the Audit Trail

Every agent operation writes a runs row—op, model, tokens, status, timings—and every tool call writes a run_steps row. Both are append-only and are not touched by a reindex. This is what makes the “no silent writes” rule real: I can always ask which model, which tool, which arguments, and how many tokens produced a given page.

The Doctor Report

status (also doctor) renders a single diagnostic surface: version, vault and DB paths and their readability, FTS5 availability, whether poppler and tesseract are on the PATH, and the counts of sources, atoms, and pages. It is deliberately not a health endpoint—this is a local tool with no orchestrator—it is the answer to “why did this scan not work?”

The first working version is a pipeline, not a feature, and the chapter treats the pipeline as the product. Each stage has an auditable boundary, so later additions attach instead of forcing a redesign. Two rules carry the model’s honesty: status is derived from what exists rather than stored, and every agent operation leaves a run record. The result is a system whose failures are visible in the same place as its successes.