# Upgrade safety

The control-plane schema is the one piece of long-lived state you can't just
re-provision: it sits in front of a database an older orlop release already
migrated. This page is the contract for **in-place upgrades** — bumping the
pinned orlop version and running `orlop-control migrate up` against that
existing database.

## The guarantee

For every **supported upgrade source** below, HEAD's `orlop-control migrate up`
converges that database to the current schema and it passes the same schema
check the control plane runs at boot.

| Upgrade from | To | Backends verified in CI |
|---|---|---|
| v0.1.0 | HEAD | Postgres |
| v0.2.0 | HEAD | Postgres, SQLite |
| v0.2.1 | HEAD | Postgres, SQLite |
| v0.5.1 | HEAD | Postgres, SQLite |
| v0.5.2 | HEAD | Postgres, SQLite |
| v0.5.3 | HEAD | Postgres, SQLite |
| v0.5.4 | HEAD | Postgres, SQLite |
| v0.5.5 | HEAD | Postgres, SQLite |
| v0.5.6 | HEAD | Postgres, SQLite |

v0.1.0 predates the embedded SQLite backend, so only its Postgres path is a
supported source.

Every row is exercised on each PR by
[`.github/workflows/upgrade.yml`](https://github.com/liu1700/orlop/blob/main/.github/workflows/upgrade.yml): CI
provisions a database with *that tag's* binary, then runs HEAD's `migrate up`
against it. That second `migrate up` self-checks the result against the schema
the code requires — so a migration that leaves an older database incomplete
fails the build, not your production. (CI runs the migrate path; it does not
boot the server. The boot-time check is the *same* self-check, below, so a
database that passes `migrate up` also clears boot.)

## What keeps an upgrade safe

Two properties make the upgrade trivial to run, and one check makes a bad one
loud instead of silent.

| Mechanism | What it does | Where |
|---|---|---|
| Forward-only migrations | `up` is the only migrate subcommand. It applies migrations numbered above the database's current version and never rewrites one that already shipped — so an old database picks up exactly the migrations added since it was last upgraded. | goose `provider.Up`, `internal/storage/postgres/db/migrate.go` |
| Idempotent `migrate up` | Re-running against an up-to-date database is a no-op, so it's safe to run on every deploy. | `migrate.go` (Postgres); SQLite re-applies `CREATE TABLE IF NOT EXISTS` on open |
| Schema self-check | At the end of `migrate up` **and** on every control-plane start (when a database is configured), the live database is checked against `storage.RequiredSchema` — the tables and columns the code needs. A gap fails fast, naming exactly what's missing. | `internal/storage/schema_check.go`; boot in `main.go`; each backend's `VerifySchema` |

Why this is enough: migrations only move forward and never rewrite history, so
a database that has run every migration up to HEAD holds the current schema by
construction. The self-check is the backstop for the one way that can break — a
renumbered or squashed migration that the runner silently skips (see the
incident below). Instead of an opaque database error the first time a query
hits a missing column, you get this at `migrate up` and again at boot:

```
control-plane schema is out of date: missing columns [access_tokens.consumed_at].
Run `orlop-control migrate up` against this database. If it was already
migrated, the release may have renumbered an already-released migration —
see docs/upgrade-safety.md.
```

## Operator runbook

The minimal correct in-place upgrade, against an existing database:

```bash
orlop-control migrate up          # reads DATABASE_URL, or pass --database-url
# then start the new control-plane binary as usual
```

Before upgrading to v0.5.2, take a database backup. Migration
`0011_owner_capacity_reservations.sql` backfills one reservation per
owner/server pair and recalculates each server's free capacity, repairing the
legacy per-agent over-reservation. The migration is safe to retry, but the old
v0.5.1 allocator does not understand the new ledger: after migrating, restore
the backup before rolling the control plane back to v0.5.1.

Deployments that already ran v0.5.2 should upgrade to v0.5.3 and run its
`0012_repair_owner_capacity_reservations.sql`. It repairs historical allocations
whose per-agent tenant has no `server_vms` row and rebuilds `free_bytes` (#108).
The repair is automatic when only one pool server exists. In a multi-server
deployment, `migrate up` stops with an unresolved-owner count if placement
cannot be inferred; restore those `server_vms` rows (or create the matching
owner reservation explicitly) and rerun the idempotent migration.

v0.5.4 through v0.6.1 ship **no control-plane migration**, so unlike the two upgrades
above they roll back with a plain image revert and need no backup step.

v0.6.1 is a mount-client-only change: spilled-file flushes now use single-pass
streaming CDC and a process-wide bounded upload pipeline instead of
rematerializing the full file in memory (#131). The wire protocol and both
server components are unchanged, so the mount client can roll independently.

v0.6.0 changes both halves, compatibly (the metadata change feed + client
mirror, #122): the server applies additive columns and tables to each
per-tenant data-plane SQLite database at open (`rev` columns,
`change_counter`, `change_tombstones`) — automatic, idempotent, and ignored
by an older server after a rollback. The wire changes are new op codes plus
append-only msgpack fields, negotiated explicitly: an old client never sees
the feed, and a new client against an old server runs mirror-less. Either
half can roll independently.

v0.5.7 is an orlop-server-only change (tenant registration no longer holds the
server-wide lock across JuiceFS filesystem I/O, so a cold-cache registration can't
stall other registrations or data-plane tenant lookups, #119): nothing in
orlop-control or the mount client moves, so it can roll independently.

v0.5.6 is a mount-client-only change (the mount process releases its lease on
SIGTERM/SIGINT instead of dying with it held, #117): nothing server-side moves, so
it can roll independently of orlop-control/orlop-server.

v0.5.5 does change both halves together: orlop-control asks orlop-server whether a
mount is still live before reclaiming a lease from a dead holder (#114), over a new
`GET /control/tenants/{id}/allocations/{alloc}/mount-lease`. **Roll the two together.**
A new control plane against an old server gets a 404 from that call, which is treated
as "liveness unknown" and refuses the reclaim — i.e. it degrades to the pre-v0.5.5
behaviour rather than doing anything unsafe, so a staggered roll is survivable, just
pointless until the server catches up.

| Do | Don't |
|---|---|
| Run `migrate up` with the **new** binary before starting it. | Start a new binary against an un-migrated database — boot fails the schema check by design. |
| Run `migrate up` on every deploy; it's idempotent. | Hand-edit the schema to clear a `schema is out of date` error — run `migrate up` instead. |
| Treat a `schema is out of date` error after `migrate up` as a release bug and report it — a supported source must never hit it. | Upgrade from a source tag that isn't in the table; an unlisted source isn't covered. |

`migrate up` runs goose against Postgres; for the embedded SQLite backend it
opens the database — applying the schema — and runs the same self-check.

## Version policy

A release that breaks an in-place upgrade from a previously supported source is
a **breaking change**: ship it as a major/minor bump with explicit upgrade
notes, never a silent patch. A minor bump must stay in-place-safe. Maintain the
supported set by appending a tag to the CI matrix as it ships, and dropping one
only when it stops being a supported source.

## Migration rules (for contributors)

The v0.1.0→v0.2.0 incident (#39): squashing the released migrations reset goose
numbering to version 1, but a deployed v0.1.0 database was already at goose
version 9. goose only applies versions above the current max, so it skipped the
squashed baseline entirely — leaving those databases without
`access_tokens.consumed_at` and the `cert_revocations` table while goose
reported success.

| Rule | Why |
|---|---|
| Never renumber or rewrite an already-released migration. | Some deployed database is already at that version and will never re-run it. |
| If you squash, ship a forward **bridge** migration numbered above the highest released version, every statement guarded by `IF NOT EXISTS`. | An already-deployed database converges to the same baseline. `0010_post_squash_reconcile.sql` is the worked example; it's a no-op on a fresh database. |
| When the code starts depending on a new table or column, add it to `storage.RequiredSchema`. | Keeps the boot/migrate self-check honest. |
| SQLite needs the same care. Its schema is applied with `CREATE TABLE IF NOT EXISTS` on open, which adds missing *tables* but never a column to an existing table. | A column added to an existing SQLite table won't backfill; the self-check and the SQLite CI job catch it. |

## See also

- [`database-backends.md`](/reference/database-backends/): the two backends and how each applies its schema
- [`control-plane-runbook.md`](/reference/control-plane-runbook/): operator workflows