# Control-plane runbook Operator workflows for the orlop control plane. The CA design and rationale live in [`design-auth.md`](/reference/design-auth/); the full control-plane config reference (every env var) is in [`control-plane.md`](/reference/control-plane/), and the production security obligations are canonical in [`SECURITY.md`](https://github.com/liu1700/orlop/blob/main/SECURITY.md). For a single-node bring-up that exercises the whole stack end to end, see [`standalone-quickstart.md`](/reference/standalone-quickstart/). ## Trust hierarchy ``` org root CA (10y, ed25519; generated offline, then loaded into the │ control-plane secret store — never onto an orlop-server VM) └── tenant CA (1y, ed25519, control-plane secret store) └── agent (1h, ed25519, minted on every /agent/enroll) ``` Both the root and tenant keys live in the **control-plane** secret store at runtime: the control plane is the CA, and signs with the root key at boot (its own client cert) and on demand (tenant intermediates, server CSRs). What never reaches a **data-plane** (`orlop-server`) VM is any signing key — those hold only the public root cert as a trust anchor. Each agent leaf carries **two** SPIFFE URI SANs: - `spiffe:///tenant/` (the tenant), and - `spiffe:///agent/` (the agent). The agent SAN is the per-agent isolation point on the data plane: a connection is confined to that agent's path prefix. The tenant SAN is what the cross-tenant binding check matches the signing intermediate against, so a leaf cannot be replayed under a different tenant. The userID is recorded in the leaf Subject CommonName for audit. Because the agent is untrusted, the tenant always comes from the verified cert, never from the request body. ## Bootstrapping the CA ### 1. Org root Generate the org root on an operator workstation. The control plane needs the root **key** to mint its own client cert at boot and to sign tenant intermediates and server CSRs, so it must be provisioned into the control-plane secret store — the filesystem backend (`ORLOP_SECRETS_DIR`) below, or generated directly into encrypted Postgres (`ORLOP_SECRETS_BACKEND=postgres`). The key that must **never** reach a data-plane (`orlop-server`) VM is the root key; those VMs hold only the public root cert. ```sh # on the operator workstation; pick a vault directory you control. export ORLOP_SECRETS_DIR=/secure/operator-vault export ORLOP_TRUST_DOMAIN=orlop.example export ORLOP_ORG_NAME=ORL orlop-control ca init --root # → writes ca/root/cert.pem + ca/root/key.pem under $ORLOP_SECRETS_DIR ``` The command is idempotent: if a root already exists in `$ORLOP_SECRETS_DIR/ca/root/`, it is loaded as-is and the command is a no-op. Re-running never rotates the root. Distribute `ca/root/cert.pem`, and only the cert, to every orlop-server deploy bundle as its trust anchor. ### 2. Tenant intermediate (online, signed against the root) Run on the operator machine while the root is reachable, then provision the resulting cert and key into the control-plane secret store. How they live there depends on the backend: the filesystem backend (`ORLOP_SECRETS_DIR`) keeps them as `0600` files; the encrypted-Postgres backend (`ORLOP_SECRETS_BACKEND=postgres` + `ORLOP_SECRETS_ENC_KEY`) stores them encrypted and decrypts into process memory at boot. ```sh orlop-control ca init --tenant acme # → writes ca/tenant/acme/{cert.pem,key.pem} under $ORLOP_SECRETS_DIR ``` Idempotent. Repeat per tenant. Upload `ca/tenant//cert.pem` to the matching tenant's orlop-server VM (used as the server's client-CA trust), and provision both files into the control-plane secret store. `orlop-control ca list` prints the tenant intermediates currently loaded from the vault. ## Running the control plane With the CA provisioned, apply the schema and start the service. `orlop-control` with no subcommand starts the HTTP server. ```sh orlop-control migrate up # applies the embedded schema (Postgres or SQLite) orlop-control # start the service (PORT=8080 by default) ``` The boot-critical secrets and gates — full reference in [`control-plane.md`](/reference/control-plane/), obligations in [`SECURITY.md`](https://github.com/liu1700/orlop/blob/main/SECURITY.md): - **A CA backend** (`ORLOP_SECRETS_DIR`, or `ORLOP_SECRETS_BACKEND=postgres` with `ORLOP_SECRETS_ENC_KEY`). Without one, `/agent/enroll` and server-cert signing are not served; a postgres CA backend without an encryption key fails closed unless you set `ORLOP_SECRETS_ALLOW_PLAINTEXT=1`. - **`ORLOP_CONTROL_PLANE_TOKEN`** — the service token gating provisioning, enroll-token minting, and server-cert signing. Those routes reject until it is set. - **`ORLOP_CA_TENANT_ALLOWLIST` / `ORLOP_CA_ALLOW_DYNAMIC_TENANTS`** — gate which tenants may have a CA intermediate lazily bootstrapped on first enroll (dynamic on by default; a typo'd boolean fails boot rather than falling back to the permissive default). ## Provisioning a tenant server cert orlop-server presents a TLS server cert that must chain through the same tenant intermediate the agent receives via `/agent/enroll`. The agent uses that chain as its only server trust anchor (it does not consult the system trust store; see [`design-auth.md`](/reference/design-auth/)). Mint that cert with: ```sh orlop-control ca mint-server-cert \ --tenant acme \ --fqdn tenant-acme.orlop.example \ --out-dir /etc/orlop/tls/acme ``` Outputs (mode `0600`): | File | Contents | | ----------- | --------------------------------------------------------------- | | `cert.pem` | leaf signed by the tenant intermediate, CN and DNS SAN = `--fqdn` | | `key.pem` | ed25519 private key for the leaf | | `chain.pem` | `intermediate \|\| root`, written for operator convenience | orlop-server itself does not need `chain.pem`: Go's default TLS handshake sends only `cert.pem`, and the agent has already learned the intermediate from `/agent/enroll`. Wire `cert.pem` into orlop-server's `tls.cert_file` and `key.pem` into `tls.key_file`. The `--ttl` flag defaults to `2160h` (90 days). The command requires the tenant intermediate to be present in `$ORLOP_SECRETS_DIR/ca/tenant//`. Run it against a host where the intermediate has not been loaded (for example a fresh operator vault) and it errors out: run `ca init --tenant ` there first. ### Server cert rotation Server certs rotate by re-running `mint-server-cert` and reloading orlop-server; the new files overwrite the old in place at the same paths. There is no separate rotation flag, the command always issues a fresh leaf. Default lifetime is 90 days, so schedule rotation accordingly. If the tenant intermediate itself rotates (see [Rotation](#rotation)), every server cert minted under the previous intermediate must be re-minted: the agent's freshly fetched chain will not match the old leaf. ## Provisioning tenants, users, and agents These commands operate on the control-plane database. They all read `DATABASE_URL` from the environment (or take `--database-url`), and possession of `DATABASE_URL` is the operator credential, the same trust model as `ca init`. `DATABASE_URL` accepts either `postgres://...` or `sqlite:...`; see [`database-backends.md`](/reference/database-backends/). | Task | Command | | -------------------------------------- | ------------------------------ | | Apply the schema | `orlop-control migrate up` | | Seed a tenant and its admin user | `orlop-control user seed` | | Register a data-plane server | `orlop-control server register`| | Issue an agent enroll token | `orlop-control token issue` | | Suspend a user | `orlop-control user suspend` | `migrate up` applies the embedded schema and works for both Postgres and SQLite (SQLite applies its schema on open). Run it once before the commands below. ### Seed a tenant and an admin user ```sh orlop-control user seed \ --tenant acme \ --email alice@acme.example \ --base-url https://control.orlop.example # → # created tenant acme (only on first run) # created user alice@acme.example under tenant acme (only on first run) # admin session token: # expires at: 2026-07-27 12:34:56 UTC # admin session URL: https://control.orlop.example/admin/session?token= ``` The command provisions a tenant and an admin user, and is idempotent on tenant plus user: it creates the tenant if absent, creates the user (admin is the only role) if absent, and always mints a fresh admin session token. Optional `--tenant-name` sets the tenant display name (defaults to the tenant id). The admin session token is an ~30 day bearer credential stored in the `access_tokens` table with `purpose = 'admin_session'`. Revoke a session by deleting its row or stamping `revoked_at` on it. When `--base-url` is set, the command also prints a one-time URL the operator opens in a browser; it exchanges the token for an HttpOnly `orlop_admin_session` cookie for the control-plane dashboard, then the cookie is used on subsequent visits. Set `ORLOP_COOKIE_DOMAIN` when the web app and the control-plane API share a parent domain through a reverse proxy. ### Register a data-plane server in the placement pool ```sh orlop-control server register \ --data-addr tenant-acme.orlop.example:8443 \ --ops-addr 10.0.0.5:7878 \ --total-bytes $((10 * 1024 * 1024 * 1024)) ``` `/agent/enroll` places each account onto a server drawn from the placement pool; all later agents for that account reuse the placement. Pool capacity is reserved once per account/server for the shared account budget, not once per agent. With an empty pool, enroll has nowhere to put a disk and returns `503`, so register at least one server. The command upserts the pool row keyed on `--data-addr`; re-run it to update the ops address, capacity, or status. | Flag | Meaning | Default | | --------------- | -------------------------------------------------------------------------------- | ---------------- | | `--data-addr` | address agents dial for the data plane; must match the server's cert SAN (`tls.fqdn`) | `localhost:8443` | | `--ops-addr` | address the control plane dials for the server's ops API over mTLS | `localhost:7878` | | `--total-bytes` | pool capacity that account budgets are placed against | 10 GiB | | `--status` | pool status; only `available` servers are picked for placement | `available` | On a single local node both addresses use the same host, so one self-provisioned cert covers both connections. Re-registering resets free capacity to total, which is correct for a single node with no concurrent reservations to preserve; a multi-node operator manages capacity out of band. ### Issue an agent enroll token ```sh orlop-control token issue --agent demo --control-plane https://control.orlop.example ``` This is the agent enrollment entry point. It provisions the agent's disk allocation idempotently and mints a single-use, agent-scoped enroll token (`purpose = 'agent_enroll'`) that is short-lived (~10 minutes), then prints a ready-to-mount env block: ```sh export ORLOP_AGENT_ID=demo export ORLOP_MOUNT_POINT=./agent-disk export ORLOP_CONTROL_PLANE=https://control.orlop.example export ORLOP_ENROLL_TOKEN= orlop mount --from-env ``` The mount client trades `ORLOP_ENROLL_TOKEN` at `/agent/enroll` for a 1h agent leaf. Optional flags: `--owner UUID` (the owning account), `--size BYTES` (initial disk grant, default 1 GiB), and `--mount-point PATH`. For the full end-to-end flow (database, control plane, `server register`, the data-plane server, mount, and a durability check), follow [`standalone-quickstart.md`](/reference/standalone-quickstart/). ### Suspend a user ```sh orlop-control user suspend --email alice@acme.example ``` This stamps `users.suspended_at`. The bearer middleware joins through that column, so the user's access tokens stop validating on their next use, and `/agent/enroll` refuses to mint new leaves for them. An agent leaf already minted for that user (up to 1h old) keeps authenticating to orlop-server until it expires, because the data plane does not call back to the control plane on each request. To drop an outstanding leaf before its hour is up, get its serial onto the deny-list (see [Revocation](#revocation)); a leaf serial lands there when its mount lease is released. ## Rotation ### Tenant intermediate (yearly, no incident) 1. `ca init --tenant ` is not the rotation command. It is idempotent and will not overwrite. To rotate, first remove the existing intermediate from `$ORLOP_SECRETS_DIR/ca/tenant//` (and from the secret store), then re-run `ca init --tenant `. 2. Push the new `cert.pem` to the tenant's orlop-server VM and roll the process. Until both the new and old intermediates are trusted on the server, agents holding outstanding leaves issued by the old intermediate are denied. 3. Outstanding agent leaves expire within 1h, so the rollout window is self-healing and needs no client-side action. ### Org root (emergency only) A root rotation invalidates every intermediate and every agent leaf across all tenants. There is no online procedure: distribute a new root to every server VM and re-bootstrap every tenant intermediate against the new root. Plan a maintenance window and announce it. ## Revocation There is no CRL or OCSP, so cert expiry plus a per-serial **deny-list kill switch** is how a single leaf is revoked mid-life. Releasing a mount lease records the bound agent leaf's serial in the `cert_revocations` table. A reconcile loop on the control plane fans the active set out to the data-plane servers: ``` lease release ──> cert_revocations (control-plane DB) │ │ control-plane reconcile loop (~60s), outbound client ▼ PUT /control/cert-revocations ──> each data-plane server (hosts this route) │ ▼ server drops a matching leaf at session start; entries age out at cert expiry ``` The direction matters: `PUT /control/cert-revocations` is served by orlop-server (the data plane). The control plane is the outbound client that pushes the active serial set to every registered server's ops address on each ~60s tick (and once immediately at startup), repopulating any server that restarted with an empty in-memory list. Propagation is bounded by that interval, and entries age out automatically at the cert's own expiry. This kills a single leaked or released leaf without a tenant-wide rotation. For a broader cut-off, rotation is the blunt instrument. To cut a single tenant off immediately: 1. Rotate that tenant's intermediate following the rotation procedure above. 2. Push the new `cert.pem` only to the tenant's orlop-server VM and restart it. Do not include the old intermediate in the server's client-CA trust. 3. All outstanding leaves for that tenant fail the TLS handshake from this point. Agents that re-enroll get leaves signed by the new intermediate; the leaves they already hold (up to 1h old) become useless. To cut a single user off, use `orlop-control user suspend` (see above). ## Blast radius | Compromise | Blast radius | Recovery | | ---------------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | Agent leaf | One user, up to 1h. | None needed; the cert expires. Cut it short via the deny-list. | | Tenant intermediate | All agents in that tenant for as long as the intermediate is trusted by the server. | Rotate the intermediate. Up to 1h until all outstanding leaves expire. | | Org root | All tenants and all environments using that root. The attacker can mint intermediates that pass verification. | Emergency root rotation. Re-bootstrap every server VM and every tenant intermediate against the new root.| | Control-plane process memory | Holds the org root key, so worst case equals an org-root compromise: arbitrary tenant certs can be minted. | Emergency root rotation (above). |