Exploring the Concept

Why two layers became three, and why retrieval here means reasoning over hierarchies instead of vector search.

The first sketch had two layers: raw sources, and notes written from them. It took about a day to see the lie in it. A note is prose—a paragraph that makes an argument, hedges, and refers to other notes. When a source is replaced, which sentence of that prose was wrong? You cannot say. You either keep the whole page or rewrite it. Two layers would have reproduced the exact rot I was trying to escape.

Why Three Layers

The middle layer is a claim: one sentence, in a file of its own, with the source node it came from attached. That changes everything about maintenance.

A source that is replaced can be traced to the exact claims it produced. Those claims get a status (superseded, disputed) and the pages that compiled them are rebuilt.
A claim that is wrong is one file. Fix or delete it; the wiki recompiles.
A page is derived, so it never needs to be edited by hand. Its atoms: list is its provenance and its truth source.

So the three layers answer three different questions, and each owns exactly one of them: sources/ is what did the source say, atoms/ is what do we now believe, wiki/ is how do we explain it. The middle layer is what makes an agent’s writing auditable.

Similarity Is Not Relevance

The obvious way to retrieve from this is a vector store. I rejected it, and that rejection shaped the rest of the design.

Similarity is a property of surface form; relevance is a property of intent. “This paper and that note both use the word cache” is not an answer to “is caching safe here”. Reasoning is. And a nearest-neighbour hit cannot tell you why it matched, which breaks the explainability rule from the start.

So retrieval became the human way: pick the substrate that fits the question.

The generated indexes (wiki/index.md, per-topic, per-branch) show the shape of the whole corpus.
FTS5 handles lexical lookups—cheap, exact, explainable.
The per-source tree handles long documents: read the outline, reason about which branch matters, open the section, cite the node.
The link graph handles associative trails.

Where “these two pages are near” is genuinely wanted, it is a symbolic quantity: shared tags, shared provenance atoms, plus a classical TF-IDF cosine computed on the fly. Nothing is stored and every match states its reason.

The Pipeline Falls Out

With three layers and reasoning-based retrieval, the design stops being a list of features and becomes a pipeline:

intake → extract → build tree → atomize → compile.

Each arrow is a stage with one job and one output. Everything later—the run audit, the re-ingest rules, the lint pass—is just this pipeline taken seriously.

The entry records the step from two layers to three and the reason for it: prose is not atomisable, so a replaced source can never say which sentence of a note became wrong. Making the claim the atomic unit turns that into bookkeeping—provenance, replacement and correction become mechanical. The rejection of similarity search follows from the same test: a vector finds neighbours, while a claim can be checked. Everything later—the pipeline, the lenses, the audit—derives from this one choice.

AI Insights: The Atomic Unit Is the Claim