CRDT Wire Protocol
The big architectural shift this week: pushing operations instead of full documents. This transforms Syncraft from a state-based sync library into a proper operation-based CRDT system.
From Documents to Operations
Previously, the wire protocol transmitted full documents. A put() created field-level diffs locally, but the sync
orchestrator looked up the current document state and pushed the entire document blob. The server stored full documents,
and the pull path did LWW document-level merge. This was state-based sync—simple, but losing information.
Now the orchestrator pushes individual change events. Each Change entry carries: the operation type (put, delete,
snapshot), the document ID, the nested field diffs, and optionally full data for snapshots. The server stores these as
opaque encrypted blobs—it never inspects the content.
Field-Level Merge on Pull
The key change in the pull path: instead of mergeLWW(docA, docB) at the document level, we now replay incoming changes
through applyChange():
- Snapshot: full state replacement (used for initial sync and compaction)
- Delete: removes the document from local storage
- Put with fields: applies
applyDiff()— field-level merge into existing data - Put with data: snapshot-style full data replacement
For put operations with field diffs, two peers editing different fields of the same document converge correctly. Peer A
changes name, Peer B changes price — both fields survive in the merged result. Same-field conflicts use ULID ordering
(the lexicographically later ULID wins, which is time-ordered).
Encryption at the Change Level
Encryption now operates on individual changes, not documents. The EncryptedChange wire format carries:
- Plaintext metadata:
id,timestamp,deviceId,kid,nonce - Encrypted payload:
documentId,collection,operation,fields,data
The server only sees the metadata fields—the actual change content is an opaque encrypted blob. This aligns with the principle that the server is an untrusted relay, not a source of truth.
Server Becomes a Relay
The server side was simplified significantly. Instead of persisting documents and tracking versions, it now stores
SyncChange entries as flat JSON files. The push handler just writes blobs. The pull handler lists blobs after a given
ULID cursor. No document-level logic, no upserts, no conflict resolution—the server is a dumb relay for encrypted
operation logs.
Convergence Properties
With operation-based sync and field-level merge, the system satisfies CRDT convergence:
- Commutative for disjoint fields: different fields merge independently
- Total order on same fields: ULID ordering provides deterministic resolution
- Idempotent: replaying the same change is idempotent per field
- Snapshot compaction: when the log exceeds threshold, snapshots compact history without losing convergence
The system is now a CmRDT (operation-based CRDT) with LWW-Register per field.
The shift from state-based to operation-based sync is the inflection point where Syncraft transitions from “a sync library that pushes documents” to “a proper distributed synchronization engine.” Field-level diffs were always computed locally—the breakthrough was transmitting them. The server becoming a simple relay (storing opaque blobs, no document-level logic) dramatically reduces the server’s surface area and makes it easy to support new backend types.