An Encompassing Collection

The pipeline assumed every source could be read. Then I started adding links to videos, paywalls, and JavaScript apps—pages that yield nothing to extract. The honest options were to reject them or to fabricate claims about them. Both were unacceptable, so the collection grew a second kind of citizen.

Two Kinds of URL, One Collection

A URL is fetched once, main-content-extracted into Markdown, and snapshotted locally—the wiki is never built from a live page. Then one of two things happens:

• An indexed document, when the page yields enough text (a configurable character floor, 400 by default): snapshot, tree, atoms, pages. The full pipeline.
• A bookmark, when it does not: the same content-addressed snapshot shape, but metadata only—title, site, excerpt, HTTP status—and never atomized. The tree and ingest sweeps skip it.

Both are ordinary sources in the same store, so “everything I collected” stays one list. A page that does have text can still be forced into the second class with a flag, for the times when I want to keep a link without the machine drawing conclusions from it.

A bookmark is not a dead entry: it gets one outline node (title plus excerpt) so it is findable in full-text search over sources, and it has its own view grouped by host. It is also still a live URL: it carries its HTTP status and last-check time, and it is re-fetched on a refresh interval—an unchanged page gets its status refreshed, a changed page becomes a new snapshot that the normal pipeline distills.

Favicons, Derived and Disposable

Each URL source stores the site’s icon—<link rel=icon>, then readability’s favicon, then /favicon.ico—content-addressed and served from its own endpoint. It lives with the index, not the vault, because it is derived and disposable. That is the vault-versus-index rule doing its job on a feature that arrived a month later.

Topics Are Derived, Never Hardcoded

The Library needed a hierarchy, and I refused to bake one in—every user’s corpus is different, so a hardcoded taxonomy is a bug waiting to happen. Instead the organizer derives it: the model is asked for a small set of broad domains (Oberbegriffe) and reusable subtopics, then assigns every page to domain/subtopic (two passes, no re-ingesting). The config can override this with a fixed list, but by default the structure comes from the content.

Crucially, existing pages can be re-tallied without re-ingesting: the “Re-sort topics” action reads each page with its atoms and writes only the new location, moving the Markdown file and regenerating the indexes. No source is touched, no claims are re-derived. The compile prompt also receives the existing topics, so a new ingest reuses them instead of inventing a fresh bucket per source.

Tags Are Pure ASCII

A small rule with a long shadow: tags are transliterated on the way in. German umlauts take their conventional spelling (ü → ue, ö → oe, ä → ae, ß → ss) and other diacritics lose their marks (é → e). Tags live in the frontmatter—never as a trailing #tags line at the end of a body (a normalizer strips one if it slips through), never inside code, never in a heading. Tags are case-insensitive in the index. Boring, consistent, greppable.

A collection is only honest if it can hold what it did not understand. The chapter splits URLs into two citizens: a page with readable text becomes a source and produces claims, while a video, a paywall or a JavaScript app becomes a bookmark marked collected, not understood. The distinction is enforced in the data model, the search and the UI rather than explained in a tooltip. Topics are derived and curated rather than hardcoded, and tags stay ASCII so they survive every tool that touches them.