# Orlop Metadata Mirror: Local Reads, Proven Freshness > **Status:** implemented as of v0.6.0 > ([issue #122](https://github.com/liu1700/orlop/issues/122)). The read-through > mirror ships enabled by default (`ORLOP_METADATA_MIRROR=0` disables it). > Pipelined unlink is implemented but **opt-in** > (`ORLOP_METADATA_PIPELINE=1`) until its performance gate is demonstrated on > production workload benchmarks (section 6, gate 4). Orlop's remaining performance gap is metadata-heavy work, not bulk throughput. Measured on a production agent disk (200 small files, five rounds, medians): | Workload | Durable disk | Local ext4 | Gap | |----------|-------------:|-----------:|----:| | create small files | 115.2 files/s | 1,161.8 files/s | 10.1x | | walk the tree | 69.1 ms | 4.6 ms | 15x | | delete the tree | 812.5 ms | 6.9 ms | 117.8x | | 1 GiB sequential write | 196.9 MiB/s | 953.8 MiB/s | 4.8x | Sequential I/O is already served by the chunk data plane and the local chunk cache. The metadata gap has a single cause: every `lookup`, `readdir`, `readlink`, and manifest fetch is a fresh network round trip, and every mutation pays extra round trips to learn versions the client recently saw. `unlink` costs two round trips (`MANIFEST_GET` for the version, then `MANIFEST_DELETE`), `rename` costs three, `create` costs two (`LEASE_GRANT` plus `MANIFEST_PUT`). This document specifies the fix: a **client-side metadata mirror**, hydrated and invalidated by a **server change feed**, with **explicit durability fences** for the one mutation path that becomes asynchronous. The server remains the single source of truth. Nothing in this design weakens CAS versioning, leases, audit, quota, or the per-agent path moat, and it introduces no last-writer-wins behavior anywhere. ## 1. Shape of the solution Three pieces: 1. **Revision stamping (server).** Every metadata mutation, from any writer, stamps the rows it touches with a per-tenant monotonic revision and records tombstones for deletions. This turns the tenant database itself into a coalesced change log: "everything that changed since revision N" is a query, not a replay. 2. **Change feed (wire).** Two new client ops and one new server push let a mount fetch that query incrementally through a `(rev, path)` cursor and subscribe to live change events on the existing data connection. 3. **Mirror (client).** The mount keeps a local SQLite mirror of its subtree's metadata and serves `lookup`, `getattr`, `readdir`, `readlink`, and manifest reads from it, but only while a live subscription plus a caught-up cursor prove the mirror is current. The moment that proof lapses, every read goes back to the server, exactly as today. Two designs were considered and rejected: - **Journal replay.** The session journal cannot drive a mirror: it records only manifest ops (`mkdir`, `rmdir`, `symlink`, `mknod`, `chown`, and `utimensat` write no journal rows), its `seq` is per mount session rather than per disk (a new session restarts at 1, so a sequence watermark silently skips rows across sessions), and some writers (agent purge, seeding) bypass it entirely. The journal keeps its job: audit and revert. The change feed is a separate, complete mechanism. - **Last-writer-wins sync.** Cloudflare's `computer` project keeps a local SQLite workspace mirror and syncs revisioned content-addressed deltas; its mirror shape and cursor design inform this one. But its preview protocol accepts silent lost updates in three places (concurrent pushes, type conflicts, per-path coalescing of intermediate states). Orlop already has per-path CAS and returns `ESTALE` to the losing writer; this design keeps that behavior everywhere. ## 2. Revision stamping The per-tenant SQLite database gains one counter, one column per metadata table, and one tombstone table (all additive, applied by `ensureTenantSchema` like prior column additions): ```sql create table change_counter ( singleton integer primary key check (singleton = 1), next_rev integer not null ); alter table manifests add column rev integer not null default 0; alter table dir_entries add column rev integer not null default 0; alter table symlinks add column rev integer not null default 0; alter table special_nodes add column rev integer not null default 0; create table change_tombstones ( path text primary key, rev integer not null ); ``` Every transaction that mutates metadata allocates one revision (read `next_rev`, increment it, same transaction) and stamps it onto every row it creates or modifies. Deleting a path upserts a tombstone at that revision; re-creating the path deletes the tombstone (the live row's newer revision supersedes it). A directory rename restamps the whole subtree at one revision, which is why the feed cursor is a `(rev, path)` pair rather than a scalar (section 3). Stamping is **unconditional**. It happens inside the same SQLite transaction as the mutation for every writer: the nine data-plane mutation ops, journal revert, agent purge, and offline seeding/migration tools. Unlike the journal it is not gated on a session id, because a mirror that misses even one writer's changes serves stale data. This is also why lease possession is never used as a freshness proof: journal revert and agent purge legitimately mutate metadata without holding the mount's leases, and the feed is the only channel that makes those mutations visible to the client. Because the counter, the row stamps, and the tombstones commit atomically with the mutation itself, the feed can never observe a revision without its rows or rows without their revision, including across a crash that rolls back the WAL tail. **Tombstone retention.** Tombstones are pruned after a retention window, and a `pruned_before_rev` watermark records the oldest revision the feed can still serve. A client resuming from a cursor older than that watermark is told to rebaseline (section 3). This bounds table growth without any server-side bookkeeping of client cursors. ## 3. The change feed Three new frame types on the existing mTLS data connection, mirrored in Go (`cmd/orlop-server/dataplane`) and Rust (`src/backend/dataplane`) like every other op: | Op | Hex | Direction | |----|-----|-----------| | `CHANGES_FETCH` | 0x1B | client → server | | `CHANGES_SUBSCRIBE` | 0x1C | client → server | | `CHANGES_EVENT` | 0x1D | server → client (push) | **Negotiation is explicit.** Every `CHANGES_FETCH` and `CHANGES_SUBSCRIBE` request carries `sync_protocol` (currently `1`); the response echoes the version the server will speak. A server that does not implement the feed answers with `EINVAL` (unknown op), and the client runs with the mirror disabled, byte-for-byte identical to today's behavior. A client never interprets feed frames until it has seen its own `sync_protocol` echoed, so an old client cannot misread a new stream and a new client cannot misread an old server. Unknown push frames were already discarded by older clients, so `CHANGES_EVENT` is safe to introduce on a shared connection. **`CHANGES_FETCH`** pages through the coalesced state: ``` request: { sync_protocol, cursor_rev, cursor_path, limit, include_chunks } response: { sync_protocol, entries, next_rev, next_path, current_rev, resync_required } ``` The response contains every live entry and tombstone in the caller's subtree with `(rev, path)` strictly greater than `(cursor_rev, cursor_path)`, ordered by `(rev, path)`, capped at `limit`. A fresh mirror starts from `(0, "")`; because the comparison is lexicographic on the pair, pre-existing rows whose stamp is still the schema default `0` are delivered too. The cursor is a resume point, not a snapshot: a path that changes mid-hydration simply reappears later in the stream at its new revision, and convergence holds because the cursor never advances past a revision that still has undelivered paths. `resync_required` is returned when `cursor_rev` predates `pruned_before_rev`; the client must discard its mirror and restart from `(0, "")`. Each entry is final-state, one per path: ``` { path, rev, kind, -- file | dir | symlink | special | tombstone size, mode, mtime, uid, gid, atime, nlink, inode_id, version, -- the manifest CAS version, files only link_target, rdev, chunks } -- packed chunk list, when include_chunks ``` Entries carry everything `lookup`, `getattr`, `readdir`, and `readlink` need, plus (with `include_chunks`) the manifest chunk list so reads need no `MANIFEST_GET` either. There is no rename opcode and no operation history: final-state entries are idempotent to apply, indifferent to receiver history, and make cold start the same code path as catch-up. **`CHANGES_SUBSCRIBE`** registers the connection for live events and returns `current_rev`. From then on the server pushes `CHANGES_EVENT` frames, `{ entries, current_rev, reset }`, from the same post-commit broadcast hook that feeds the journal pub/sub. Delivery is per-allocation, filtered to the agent's subtree, and non-blocking: if a subscriber's buffer overflows, the server pushes `{ reset: true, current_rev }` and drops the buffered backlog. A reset tells the client its event stream has a gap; it falls back to `CHANGES_FETCH` from its last applied cursor and serves reads from the server until it has caught back up. The subscription dies with the connection. **Authorization and audit.** Feed requests pass the same policy check and agent-subtree confinement as every other op: the feed for an agent-scoped certificate contains only paths under `/`, enforced server-side by the same `checkAgentPath` logic, and denials audit exactly like any other denied op. Fetch and subscribe are themselves audited ops. The deny and audit semantics of all existing ops are untouched. ## 4. The client mirror The mirror is one SQLite database per allocation under the existing cache root (`$XDG_CACHE_HOME/orlop`, else `$HOME/.cache/orlop`), next to the chunk cache index: ``` / chunks/… index.sqlite # chunk LRU index (existing) mirror/ .sqlite ``` One `entries` table in the shape of the wire entry (plus a derived `parent` column indexed for `readdir`), and one `mirror_meta` table holding the applied cursor, the negotiated `sync_protocol`, the server identity, and the mirror schema version. The database opens with WAL and `synchronous=NORMAL`; it is a cache, not a store of record. Validation is fail-closed, in the same spirit as the live-handoff snapshot: wrong schema version, wrong server identity, or any integrity error deletes the file and rehydrates from `(0, "")`. A missing or corrupt mirror never fails a mount; it only means the first reads go to the server. **The freshness invariant.** The mount serves a metadata read locally if and only if all of the following hold: 1. a `CHANGES_SUBSCRIBE` is live on the current data connection, 2. the applied cursor has reached the `current_rev` reported at subscribe (or by a later event), and 3. no unprocessed `reset` is pending. Otherwise, and during hydration, reconnection, or after any reset, every read goes to the server exactly as today. Freshness is proven by revisions, never by leases, and never by elapsed time. **Serving matrix, once fresh:** | Operation | Today | With mirror | |-----------|-------|-------------| | `lookup` / `getattr` | network on first touch, then unproven in-memory cache | mirror, proven | | `readdir` | `LIST` every call | mirror | | `readlink` | `READLINK` every call | mirror | | manifest fetch for reads | `MANIFEST_GET` every open | mirror (chunk list included) | | version lookup before `unlink` / `rename` | extra `MANIFEST_GET` round trips | mirror | | chunk data | chunk cache, unchanged | chunk cache, unchanged | Mutations remain synchronous wire ops. On a successful reply the client applies the result to the mirror immediately (it knows the path and the new version from the response) rather than waiting for its own change event; events apply as upserts guarded by `rev`, so the echo of a local mutation is idempotent. Mutation round trips also shrink, because the mirror already holds the CAS version: `unlink` drops from two round trips to one and `rename` from three to one. The CAS itself is unchanged; if the mirror's version is stale the server answers `ESTALE` with a `RecoveryHint`, the client refetches, reconciles the mirror, and retries, the same conflict path that exists today. The macOS NFS server gains the same mirror through the shared store layer. Because the wire `EntryWire` also gains `mtime` and `version` fields (append-only msgpack, the sanctioned compatibility path), the NFS `getattr`/`readdir` implementation stops issuing one `MANIFEST_GET` per file even when the mirror is cold. **Stale manifests and GC.** A mirror entry can briefly reference chunks that an external mutation replaced. Refcounted GC's retention window already keeps just-unreferenced chunks alive; if a `CHUNK_GET` still misses, the client drops the mirror entry, refetches the manifest, and repairs the mirror. A stale read is thus detectable and self-healing, never silently wrong bytes (chunks remain content-verified on every read). **Audit posture.** A read served from the mirror does not produce a server-side audit row, just as FUSE `getattr` served from the in-memory node table produces none today. The audit stream remains complete for mutations, for every denied operation, and for every op that reaches the wire. Paths the policy or the agent moat would deny never enter the mirror, because the feed is filtered server-side by the same checks; `EACCES` results are never cached. ## 5. Durability and fences **Read-through mode (the default).** Write acknowledgement semantics are unchanged. A metadata mutation is durable when the server's success frame arrives, meaning the transaction committed to the tenant SQLite database under WAL with `synchronous=NORMAL`: it survives a server process crash; an unclean server host crash can roll back the newest committed transactions in the un-checkpointed WAL tail. That window predates this design and is unchanged by it. A client crash after the reply loses nothing; a client crash before the reply loses at most the in-flight op, which the kernel never saw acknowledged. **Pipelined mode (opt-in until its gate is met).** One mutation becomes asynchronous: `unlink`. The FUSE reply returns once the unlink is appended to a bounded, ordered, per-mount queue; queued unlinks are issued in order over the existing connection, carrying CAS versions from the mirror. The mirror applies the unlink immediately, so every subsequent local operation (`lookup`, `readdir`, a recreate at the same path) observes it; ordering guarantees the server converges to the same state. An acknowledged-but-unsent unlink is lost if the client process or host crashes: the file simply still exists on the durable disk afterward. POSIX makes the same statement about local filesystems, where an unlink is durable only after `fsync` of the parent directory. The fences below are therefore the durability contract, and each one drains the queue (or the affected subset) and reports any queued failure before returning: | Fence | Scope drained | |-------|---------------| | `fsync` / `fdatasync` on a file or directory | whole queue | | `syncfs` | whole queue | | `rmdir` | queued unlinks under that directory | | `rename` | queued ops intersecting source or destination subtree | | clean unmount, live handoff | whole queue | | lease revoke, connection loss, subscription reset | whole queue | | any op whose server reply could contradict a queued op | affected paths | A queued unlink that fails on the server (for example `ESTALE` from a concurrent external mutation) surfaces as an error on the next fence, `fsync`-reports-writeback-error style, and forces a mirror reconcile. The queue is bounded (ops and bytes); when full, `unlink` becomes synchronous rather than growing without limit. `create` stays synchronous in all modes, but stops paying a separate `LEASE_GRANT` round trip where the mount's exclusivity already covers the path. If the adversarial suite (section 6) cannot demonstrate these semantics exactly, pipelined mode does not ship and the deliverable is the read-through mirror with synchronous mutations, per the issue's own fallback. ## 6. Acceptance gates From [issue #122](https://github.com/liu1700/orlop/issues/122), restated as the checklist this work merges against: 1. **Durability documented.** Section 5 is the specification: exact acknowledgement points, the WAL-tail caveat, the pipelined crash window, and the fence table. 2. **Adversarial tests.** Reconnect, cursor loss, `resync_required` rebaseline, duplicate and out-of-order event delivery, external mutation via journal revert and agent purge while mounted, lease loss, CAS conflict on a mirror-supplied version, and crash during a pipelined drain. Each must show the mirror either serving proven-fresh data or falling back to server reads, never serving stale data silently. 3. **Workload benchmarks.** create/stat/walk/delete on small-file trees, `npm install`, `git checkout` and `git status`, and large sequential I/O, reported with rounds, medians, and a discarded-measurements note, against both the previous release and a local-disk baseline. 4. **Performance.** At least 5x on two of the three worst metadata scenarios (create 10.1x, walk 15x, delete 117.8x), with no more than a 10% regression in large sequential I/O. Expected sources: walk from the mirror alone; delete from the mirror (two round trips to one) plus pipelining; create from the merged lease grant. 5. **Explicit negotiation.** `sync_protocol` in every feed request and response; unknown op or unknown version disables the mirror cleanly. 6. **Fallback scope.** If gate 2 fails for pipelined mode, ship read-through only; synchronous mutation commits remain the default acknowledgement semantics. ## 7. Compatibility | Client \ Server | pre-feed server | feed-capable server | |-----------------|-----------------|---------------------| | pre-mirror client | unchanged | unchanged (never sends feed ops, ignores unknown pushes) | | mirror client | `EINVAL` on first fetch, mirror disabled, behavior identical to today | negotiated via `sync_protocol` | All wire changes are additive: new op codes, append-only msgpack fields (`EntryWire.mtime`, `EntryWire.version`), no changes to existing frames, no reserved-byte reuse. The tenant database changes are additive columns and tables applied by the existing schema-ensure path. The mirror file is a pure cache: deleting it is always safe.