# Orlop Data Plane: Chunked Storage Over mTLS

The data plane is everything between the mount client and the chunk store:
the wire it speaks, how files are split into content-addressed chunks, how
those chunks are named and reassembled, and how concurrent writes stay
consistent. The mount client is `orlop` (Rust: FUSE on Linux, an in-process
localhost NFSv3 server on macOS); the server is `orlop-server` (Go, one
process per tenant).

This doc is about the storage and transport mechanics. The control plane
(auth, disk allocation, mount leases) lives in
[`design-auth.md`](/reference/design-auth/) and [`control-plane.md`](/reference/control-plane/).

The guiding constraint throughout: **the agent is untrusted.** Anything the
agent could forge (which tenant it belongs to, which chunk it is allowed to
read) must be decided from its verified certificate or from cryptographic
content, never from a value it puts on the wire.

## 1. Terminology

| Term | Meaning |
|------|---------|
| **chunk** | A variable-length slice of a file's bytes, named by the BLAKE3 hash of its contents. The unit of storage, transfer, and dedup. |
| **manifest** | The per-path record that maps a file to its ordered chunk list, plus size, mode, mtime, and a version counter. |
| **chunk store** | Server-side directory of raw chunk blobs, one file per unique hash. |
| **lease** | A server-granted capability over a path (`SHARED_READ` or `EXCLUSIVE_WRITE`) that makes write-back caching safe. |
| **journal** | A per-tenant log of recent manifest changes that supports point-in-time revert (undo). |
| **frame** | One request or response on the wire: a 16-byte header plus a msgpack payload. |

## 2. Data model: the hash is the name

A file is not stored as a byte stream. It is stored as:

1. A list of chunks, each named by `BLAKE3-256(chunk_bytes)` (32 bytes).
2. A manifest that records the path, the ordered chunk list, and metadata.

Because a chunk's name *is* the hash of its bytes, two chunks with identical
contents have the same name and are stored once. This is the property the
rest of the design leans on: dedup is automatic, integrity is checkable, and
a cached chunk can be trusted by re-hashing it.

This fits agent workloads well. Agents re-create the same `node_modules`,
the same model weights, the same datasets across sessions and across machines.
Content addressing collapses all of those copies to a single stored blob, and
a single-byte edit to a large file re-uploads one chunk (~4 MiB), not the file.

## 3. Chunking: why boundaries must move with the content

If files were split at fixed offsets (every 4 MiB, say), inserting one byte
near the front would shift every later byte across a boundary. Every
subsequent chunk would get a new hash, and the entire cache for that file
would miss. Fixed-size chunking turns a 1-byte edit into a whole-file
re-transfer.

Orlop uses **FastCDC** (content-defined chunking). Boundaries are chosen by a
rolling hash over the content, so a boundary lands at the same *content*
position regardless of what came before it. An insert disturbs only the one
or two chunks around the edit; everything after it keeps its old boundaries
and old hashes, and stays a cache hit.

The parameters are pinned and identical on both sides of the wire:

```
MIN = 1 MiB    AVG = 4 MiB    MAX = 16 MiB     (FastCDC v2020)
```

- Rust client: `CHUNK_MIN` / `CHUNK_AVG` / `CHUNK_MAX` in `src/write_handle.rs`.
- Go server: `ChunkMin` / `ChunkAvg` / `ChunkMax` in `cmd/orlop-server/cdc.go`.

The two implementations must produce byte-identical boundaries, or a file
chunked by the client would fail to dedup against the same file chunked by the
server. `tests/fastcdc_parity.rs` enforces this against a golden vector
(`tests/golden/fastcdc_chunks_go.txt`).

## 4. Content addressing and dedup

Dedup happens at three points, all for free:

- **Within a file:** repeated regions chunk to the same hash.
- **Across files and sessions:** uploading a chunk that already exists is a
  no-op; the server bumps a refcount instead of writing bytes.
- **In the client cache:** a chunk fetched once is keyed by hash, so any
  later file that references it is served locally.

The server tracks how many manifests reference each chunk (`chunks.refcount`),
which is what makes garbage collection safe (section 11).

## 5. On-disk layout

**Server chunk store:** raw blobs on the `orlop-server` host filesystem,
sharded by the first two hex characters of the hash to keep directories small:

```
<store-root>/objects/
  ab/
    ab1f… (raw chunk bytes, filename = full BLAKE3 hex)
  cd/
    cdee…
```

The chunk store writes blobs to the local filesystem as plain files
(`cmd/orlop-server/chunkstore.go`).

**Client chunk cache:** the same content-addressed shape, under the user's
cache dir (`$XDG_CACHE_HOME/orlop`, else `$HOME/.cache/orlop`):

```
<cache-root>/
  chunks/
    ab/
      ab1f…
  index.sqlite     # one row per cached chunk: (hash, size, last_access)
```

`index.sqlite` is metadata only; the bytes live next to it under `chunks/`.
Its single table tracks `last_access` so eviction can pick the
least-recently-used chunks. Refcounting is a server concern; the client just
caches and evicts by size.

## 6. Manifests

Manifests and the chunk index are a per-tenant SQLite database on the server
(`cmd/orlop-server/tenantdb.go`):

```sql
create table chunks (
  hash     blob primary key,                 -- BLAKE3, 32 bytes
  size     integer not null,
  refcount integer not null default 0 check (refcount >= 0),
  added_at integer not null
);

create table manifests (
  path     text primary key,
  inode_id integer not null,
  size     integer not null,
  mode     integer not null,
  mtime    integer not null,
  version  integer not null,                  -- monotonic per inode
  chunks   blob not null                      -- packed [hash(32) | offset(8) | len(4)] …
);

create table dir_entries (
  parent text not null,
  name   text not null,
  primary key (parent, name)
);
```

Multiple rows may share one `inode_id`: those paths are hard-link directory
entries for the same regular file and carry identical content and metadata.
Chunk refcounts are owned once per inode, not once per path.
(`symlinks` and `special_nodes` tables hold the corresponding node types;
`uid`/`gid`/`atime` columns were added to `manifests` later for POSIX
ownership and times.)

The `manifests.chunks` BLOB is the ordered chunk list packed inline:
`hash(32) | offset(8) | len(4)` per entry. To read a file you read its
manifest, then fetch the listed chunks (most of them from cache).

**Atomic updates use compare-and-swap on `version`.** A writer sends the
version it believes is current; the server applies the write only if the
stored version still matches, then increments it. A writer that lost the race
gets back errno **ESTALE (116)**, produced by `dataplane.ErrESTALE`, carried
in the error frame, and surfaced to FUSE as a real `ESTALE`. The wire speaks
errnos, not HTTP status codes.
The error frame can also carry a `RecoveryHint` with the caller's version and
the server's current version so the client can refetch and retry.

## 7. Integrity

Every chunk is self-verifying: its name is the hash of its bytes. The client
re-hashes a chunk on every cache hit (`ChunkCache::get`); a mismatch means
on-disk corruption, so the entry is deleted and refetched. BLAKE3 makes this
cheap (it runs at multiple GB/s on one core and parallelizes), so
hash-on-read costs little even when streaming GB-scale files. A 256-bit digest
also means an attacker cannot construct a different chunk that hashes to a name
they don't already hold, so "read a chunk you have no manifest for" is not a
reachable attack.

## 8. Read path

Reading a file never hits a server "read" op; reads are reassembled
client-side from chunks:

```
open("/proj/data.bin")
  └─ MANIFEST_GET /proj/data.bin   → version, size, [chunkA, chunkB, chunkC, …]
read(off, len)
  └─ for each chunk covering [off, off+len):
       cache hit?  → serve locally (hash-verify, done)
       cache miss? → CHUNK_HAS / CHUNK_GET → store in cache → serve
```

The first read of a cold file costs one round trip per *missing* chunk; the
second read (even after unmount and remount) is served entirely from local
disk. A read of byte 0 of a 100 MiB file fetches one ~4 MiB chunk, not 100 MiB.

## 9. Write path and crash ordering

Writes go chunk-first, manifest-last, so a crash can never leave a manifest
pointing at bytes that were never stored:

```
1. Client chunks the new/changed file region (FastCDC).
2. For each chunk: CHUNK_HAS → upload only the novel ones via CHUNK_PUT.
3. MANIFEST_PUT with expected_version = the version the client last saw.
   server: CAS on version → write manifest, bump/lower chunk refcounts,
               append a journal row, all in one SQLite transaction.
```

Ordering matters: chunks are durable before any manifest references them, and
the manifest swap is a single transaction that also adjusts refcounts and
records the change in the journal. If the CAS fails, the write returns ESTALE
and nothing is mutated. Write authority is checked at `MANIFEST_PUT`, not at
`CHUNK_PUT`: chunk uploads dedup globally and are content-addressed, but
binding a path to a chunk list is the privileged step.

## 10. The journal (revert)

Each successful manifest change appends a row to a per-tenant
`session_journal` (`cmd/orlop-server/journal.go`): the path, the operation
(`create` / `update` / `delete` / `rename`), the version before and after, and
enough of the prior manifest to undo the change. The append happens inside the
same transaction as the manifest write, so a change is never left unrecordable.

Two ops expose it on the wire:

- **`JOURNAL_QUERY` (0x15):** read journal rows (filtered by allocation),
  e.g. to show what an agent changed during a run.
- **`JOURNAL_REVERT_PATH` (0x18):** replay the inverse of the most recent
  change for each named path, restoring the prior bytes. The inverse is
  applied under CAS too, so a concurrent writer surfaces as a revert conflict
  rather than being silently clobbered.

This is what lets an operator roll back an agent's writes to a known-good
state after a bad run.

Each committed entry is also broadcast over an in-process pub/sub
(`cmd/orlop-server/journal_pubsub.go`) to per-allocation subscribers — a live
feed of what an agent is changing, without polling. Delivery is non-blocking: a
subscriber whose buffer fills is dropped and is expected to reconnect and
backfill, so a slow consumer can never stall a writer's commit.

## 11. Garbage collection and leases

**GC is reference-counted, not mark-and-sweep.** Because every manifest write
already maintains `chunks.refcount`, the sweeper does not need to walk
manifests to find unreachable chunks. It simply deletes rows where the refcount
has reached zero and the chunk is older than a retention window
(`cmd/orlop-server/gc.go`):

```sql
delete from chunks where refcount = 0 and added_at < <cutoff>
```

The same predicate is re-asserted on the per-row delete, so a refcount bump
landing between select and delete leaves the chunk alive. Each sweep emits a
`gc_swept_chunks` audit event. The retention window (`added_at < cutoff`) holds
a just-unreferenced chunk back from collection until it has aged past the
window, rather than deleting it the instant its refcount hits zero.

GC holds deterministic first-byte hash locks from the conditional metadata-row
delete through the file unlink. A manifest commit takes the same locks, verifies
that every referenced chunk is a regular file, and holds them through its
refcount transaction. Therefore either the manifest commits first and the GC
predicate protects it, or GC finishes first and the manifest is rejected as
referencing a missing chunk. This also covers manifest restore/revert paths that
reuse the same commit primitive.

Chunk existence probes and post-commit GC unlinks are grouped by the first
hash byte. Presence collapses repeated hashes and skips definitely absent
shards, then keeps any remaining exact stats serial because real JuiceFS tests
showed concurrent lookup outliers could regress p99 in a fully populated store.
GC uses at most 16 process-wide shard workers, overlapping remote unlink latency
without allowing a large batch to create unbounded I/O.
Presence still comes from an exact `lstat` of a fixed-length, hex-derived hash
path and rejects non-regular final entries—no index is allowed to return
"present"—so the optimization cannot report a missing chunk as present.
The internal `objects` tree is server-owned and is never exposed as a client
path. At startup the store records which of the 256 shard directories exist;
a definitely absent shard skips all per-hash stats, and `Put` publishes the
hint before it writes. An unsupported live external writer can therefore cause
one redundant upload, never a false-present response; that `Put` reconciles the
hint, and a restart rebuilds it from disk. GC opens each touched shard once and
uses descriptor-relative deletes. All hashes are validated before a delete
batch starts, and missing files remain successful idempotent deletes.

The **client cache** is collected independently: LRU eviction down to a byte
budget (default 2 GiB, configurable), picking victims by `last_access`. Losing
a cached chunk only costs a refetch, so the cache index is kept lightweight and
non-durable.

**Leases** make write-back caching safe. Without a consistency primitive,
caching writes locally would risk serving stale data; polling the server for
invalidations would wreck latency. Instead the server grants a per-path
capability:

| Mode | Holders | Allows |
|------|---------|--------|
| `SHARED_READ` | many | cache reads; no in-flight writes |
| `EXCLUSIVE_WRITE` | one | buffer writes locally; `fsync` flushes to the server |

The server can revoke a lease at any time (contention, admin action, expiry).
Revocation is **pushed on the same long-lived connection** (`LEASE_REVOKE`,
0x13) rather than discovered by polling; the holder must flush pending writes
before releasing. This is the same approach as NFSv4 delegations, SMB3
oplocks, and CephFS capabilities. For a single-user disk, grants are
effectively long-lived and revocation is rare, so leases buy near-free
write-back correctness. (A separate mount-level lease governs who holds the
mount; that is a control-plane concern, distinct from these per-path leases.)

## 12. Transport and wire

The data path is a **single long-lived mTLS connection** carrying binary
frames. Each frame is a fixed 16-byte header followed by a msgpack payload:

```
byte:  0        1        2 .. 9          10  11     12 .. 15
      +--------+--------+----------------+--------+----------------+
      | op (1) | flags  | request id (8) | rsv(2) | payload len(4) |  msgpack payload
      +--------+--------+----------------+--------+----------------+
```

(`cmd/orlop-server/dataplane/codec.go`; `flags` carries the response and error
bits; multi-byte fields are big-endian; reserved bytes must be zero.) The op
codes and msgpack message shapes are mirrored on both sides: Go in
`cmd/orlop-server/dataplane/`, Rust in `src/backend/dataplane/`. Large reads
never travel as a single frame; they are chunk fetches, capped by
`MaxPayloadLen` (64 MiB).

**Transport carrier.** The server always listens on both TCP and QUIC on the
same bind (`runV2TCPListener` + `runV2QUICListener`). The client's
`TransportMode` defaults to **`Tcp`**; `Quic` and `Auto` (try QUIC, fall back
to TCP, remember the choice) are opt-in via `ORLOP_TRANSPORT=quic|auto` on the
mount client. So: **TCP+TLS is the default; QUIC is
implemented but opt-in.** The carrier is just a pipe: the orlop binary frame
format above is identical over either, so QUIC is not an HTTP/3 protocol, only
a different socket. QUIC stays in the tree because it offers stream
multiplexing without head-of-line blocking and connection migration across
network changes; it is held opt-in pending throughput parity on large cold
reads.

### Op codes

All ops are client→server requests except `LEASE_REVOKE` and
`CHANGES_EVENT`, which the server pushes to the client. (Codes from
`cmd/orlop-server/dataplane/protocol.go`.)

| Op | Hex | Direction |
|----|-----|-----------|
| `LIST` | 0x01 | client → server |
| `STAT` | 0x02 | client → server |
| `PING` | 0x04 | client → server |
| `CLOSE` | 0x05 | client → server |
| `MANIFEST_GET` | 0x06 | client → server |
| `MANIFEST_PUT` | 0x07 | client → server |
| `CHUNK_GET` | 0x08 | client → server |
| `CHUNK_HAS` | 0x09 | client → server |
| `CHUNK_PUT` | 0x0A | client → server |
| `MANIFEST_DELETE` | 0x0B | client → server |
| `MANIFEST_RENAME` | 0x0C | client → server |
| `DIR_CREATE` | 0x0D | client → server |
| `DIR_REMOVE` | 0x0E | client → server |
| `SETATTR` | 0x0F | client → server |
| `LEASE_GRANT` | 0x10 | client → server |
| `LEASE_REFRESH` | 0x11 | client → server |
| `LEASE_RELEASE` | 0x12 | client → server |
| `LEASE_REVOKE` | 0x13 | server → client (push) |
| `JOURNAL_QUERY` | 0x15 | client → server |
| `SYMLINK` | 0x16 | client → server |
| `READLINK` | 0x17 | client → server |
| `JOURNAL_REVERT_PATH` | 0x18 | client → server |
| `MKNOD` | 0x19 | client → server |
| `LINK` | 0x1A | client → server |
| `CHANGES_FETCH` | 0x1B | client → server |
| `CHANGES_SUBSCRIBE` | 0x1C | client → server |
| `CHANGES_EVENT` | 0x1D | server → client (push) |

The three `CHANGES_*` ops are the metadata change feed backing the
client-side metadata mirror; their cursor, negotiation, and delivery
semantics are specified in
[`design-metadata-mirror.md`](/reference/design-metadata-mirror/).

A benchmark harness (`orlop-bench`, in `bench/`) drives synthetic filesystem
workloads under emulated WAN to compare TCP and QUIC; TCP stays the default
pending QUIC throughput parity on large cold reads.

## 13. Threat model

The mount client runs next to an untrusted agent, so the data plane assumes
the agent may send anything. Defenses:

- **Tenant comes from the cert, never the request.** mTLS identifies the
  client; the per-agent identity is bound to its certificate. The session is
  gated before any frame is served: a revoked leaf is dropped, and the
  intermediate that signed the leaf must carry a tenant OU matching the leaf's
  tenant SAN (fail-closed cross-tenant gate). See [`design-auth.md`](/reference/design-auth/).
- **Per-op policy still runs server-side** (`cmd/orlop-server/policy.go`); a
  valid connection does not imply a valid operation.
- **Chunks are content-verified, not trusted.** A chunk's name is its hash, so
  a tampered chunk fails verification on read, and a chunk cannot be addressed
  without already knowing its 256-bit hash.
- **Writes are authorized at the manifest, not the chunk.** Uploading a chunk
  is harmless (it dedups); binding a path to a chunk list is the gated step.

Chunks are not encrypted at rest: tenant TLS isolates the data plane in
transit, and at-rest encryption is left to the host filesystem. Per-tenant
process and database isolation, not in-band encryption, is the data-at-rest
boundary.