Backup & Restore — Operator Guide

Obscura can take a real, restorable disaster-recovery backup of the whole system — the Postgres
database and every document blob in MinIO — on an admin-configured schedule, keep the last N
sets, and be recovered from a single documented command. This is the layer that gets you back after
an accident like a docker compose down -v that wipes the data volumes.

The backup engine is a compose sidecar (obscura-backup), not the app. The obscura container is
distroless and has no pg_dump; the sidecar is a postgres:17 image plus the MinIO client mc. It
runs a small poll loop: every ~60s it reads the backup_settings row from Postgres and, when a
scheduled backup is due or an on-demand run was requested, runs pg_dump -Fc + mc mirror into the
host-mounted ./backups directory. The database row is the only coupling — the app never talks
to the sidecar directly.

Backups are off by default. Nothing runs until an operator brings up the backup compose
profile, so the demo stack is byte-for-byte untouched until you opt in.

⚠️ NEVER run docker compose down -v. The -v flag deletes the pgdata + miniodata
volumes — that is the exact disaster this feature exists to recover from, and it once wiped the
demo. To stop a single service use docker compose stop <svc> / start <svc>. See §9.

1. What it backs up

Each backup set is a point-in-time copy of the two stores that hold your data:

• Postgres — the entire database via pg_dump -Fc (custom format: compressed, supports
  selective restore). This is every document record, folder, user, workflow, signature, audit-log
  row, and admin setting.
• MinIO blobs — all document bytes (every uploaded file / rendered PDF / version) mirrored
  out of the obscura bucket with mc mirror.

That is the complete application state. What it does not back up (by design — these are
infrastructure/dev conveniences, not customer data):

• Mailpit (the dev SMTP catcher), Keycloak / Casdoor (test OIDC IdPs), and the LDAP
  test directory. These carry no Obscura data; in production your real mail relay and identity
  provider are external systems with their own backups.

Out of scope for this feature (documented so you know the boundaries): WAL / point-in-time recovery,
incremental backups, and automatic off-site replication. For off-site copies, see §5.

⚠️ The encryption keys are NOT in the backup — back them up separately

Document blobs are encrypted at rest (age/X25519). A backup set therefore contains
ciphertext, and the key to read it is deliberately left out: storing the key beside the
ciphertext would mean anyone who copies the backup directory — a NAS, a tape, a stolen laptop —
gets the entire corpus in the clear. The backup would become the softest target in the system.

The consequence is blunt and worth stating to whoever owns recovery:

Restore this backup without the blob key and every document in it is unreadable. By anyone,
including us. There is no recovery path, no support escalation, and no vendor override.

So two things must be backed up separately, somewhere the backup directory is not:

Each set records a one-way fingerprint of both in keys.json (never the keys), so a restore
can confirm you still hold the right ones before you rely on it. restore.sh checks the
fingerprint and warns on a mismatch; restore-drill.sh fails outright. Every set also carries a
RECOVERY.txt explaining this to whoever finds the directory years from now, with no access to
this repository.

Two related consequences of at-rest encryption, so they don't surprise you:

• Blob file names are hashes of the plaintext, so sha256sum of a backup file will not match
  its name. That is expected, not corruption.
• The backup files themselves are not additionally encrypted. The document bytes inside are.
• Vault (encrypted-folder) objects restore as ciphertext under sf/ prefixes and need the
  SAME securefolder_kek.key as the source deployment. A restore into an environment with a
  different KEK yields intact-looking but permanently unreadable vaults — the fingerprint check
  above is what tells you BEFORE you rely on it. scripts/escrow-keys.sh escrows this key
  alongside the blob key; a restore runbook that skips the escrow step has not restored the
  vaults.

2. Enable the backup profile

Backups run under the opt-in backup compose profile. From the repository root:

docker compose -f deploy/docker-compose.yml --profile backup up -d obscura-backup

This starts the single obscura-backup sidecar. The default stack (docker compose up -d) never
starts it, so enabling this changes nothing about the running demo except that backups begin.

To confirm it is running and watch it:

docker compose -f deploy/docker-compose.yml ps obscura-backup
docker compose -f deploy/docker-compose.yml logs -f obscura-backup

You should see a line like obscura-backup: poll loop every 60s; target=/backups.

3. Admin settings (Admin → Backups)

Once the profile is up, an administrator controls the schedule from the Obscura UI under
Admin → Backups (server-side gated on the backup.admin permission):

• Automatic backups — on/off master switch.
• Interval (hours) — how often a scheduled backup runs. Minimum 1 (a safety floor: a 0-hour
  interval would be a runaway loop that fills the disk).
• Keep last N backups (retention count) — how many completed sets to retain; older sets are
  pruned automatically after each successful run. Minimum 1 (a safety floor: 0 would prune every
  backup away). The newest set is never pruned.
• Run backup now — triggers one immediate backup on the sidecar's next poll.

Values below a floor are rejected with a clear error, not silently clamped. Seeded defaults are
enabled, every 24h, keep 7.

Changes are written to the backup_settings row and the sidecar reads that row on each ~60s
poll, so a setting change (or a "Run backup now") takes effect within about a minute — there
is no restart needed.

4. Where backup sets live

The sidecar writes into the host-mounted ./backups directory (deploy/backups/ in the repo). Each
completed set is a timestamped folder named <TS> where TS = YYYYMMDDThhmmssZ (UTC):

deploy/backups/
  20260706T101112Z/          <- a completed backup set
    db.dump.age              <- Postgres pg_dump -Fc, age-encrypted (or db.dump if unencrypted)
    blobs/                   <- mirror of the MinIO bucket (all document bytes)
    keys.json                <- fingerprints of the keys this set needs (never the keys)
    status.json             <- {ts, result:"ok", db_bytes, blob_count, blob_bytes, duration_seconds, db_encrypted}
  20260706T221314Z/
    ...
  latest.json                <- copy of the newest successful set's status.json
  20260707T031500Z.partial/  <- a crashed or in-progress run — IGNORE it (see below)

• A set is built inside a <TS>.partial/ directory and atomically renamed to <TS>/ only on
  full success — so a set that appears as a bare <TS>/ is always complete.
• A <TS>.partial/ directory is a run that crashed or is still in progress. Ignore it for
  restore purposes; it is safe to delete. Check docker compose logs obscura-backup for the error.
• latest.json always reflects the most recent successful backup and feeds the freshness metric
  (§8).

The read-only Admin → Backups table lists these sets (timestamp, result, DB size, blob count,
blob size). The app mounts ./backups read-only — it can view sets but can never mutate or
delete a backup.

4b. Encrypting the dump (BACKUP_AGE_RECIPIENT)

The blobs in a set are already ciphertext, which for a long time left db.dump as the only
plaintext artifact — and the higher-value one. It holds every title, classification, user account,
password hash, signature record, audit-chain row and stored integration credential. Since the whole
point of a backup set is that it gets copied somewhere else, "the dump is fine because the box is
locked down" stops being true the moment you follow §5.

Set an age public key and the sidecar streams pg_dump straight into age:

# once, somewhere that is NOT a server — your workstation, or a throwaway container:
age-keygen
# public key: age1f4d...     <- this goes in the env file
# AGE-SECRET-KEY-1...        <- this goes in your password manager, NOWHERE ELSE

# deploy/prod.env (or cloud.env)
BACKUP_AGE_RECIPIENT=age1f4d...

Why a public key is the whole trick: the server can write backups that nobody on that server
can read. Compromising the machine that produces the backups does not yield the backups. That is
what makes it safe to copy a set to another host you trust less — including the other VM in a
cross-pull (§5).

• The cleartext dump never touches disk, not even briefly — pg_dump is piped into age, so a
  run that dies mid-way leaves nothing to shred.
• Leave it unset and you get db.dump in the clear, plus a loud WARNING in the sidecar log.
• keys.json records the recipient's fingerprint so a restore can tell you up front whether the
  identity you brought is the right one.

⚠️ If the identity is lost, every encrypted dump is lost with it. There is no recovery path and
that is deliberate. Escrow it the day you create it. The blobs need their own separate key
(scripts/escrow-keys.sh) — losing that loses the documents, losing this loses the metadata.

Restoring an encrypted set needs the identity passed in explicitly:

BACKUP_AGE_IDENTITY_FILE=~/backup_age.key ./scripts/restore.sh <TS>
BACKUP_AGE_IDENTITY_FILE=~/backup_age.key ./scripts/restore-drill.sh <TS>

Both decrypt to a temporary location that is removed on exit, including on failure. The drill
exercises decryption as an assertion — a drill that only ever tested plaintext sets would pass
right up until the day you discover the identity was wrong.

5. Copy backups off-box (recommended)

A backup on the same host does not survive that host dying. Point ./backups at durable, off-box
storage, or copy it there on a schedule.

Prefer pull over push. If the source host holds a credential that can write to the backup
destination, then whatever compromises the source can also delete its own backups — which is exactly
what ransomware does first. When the destination reaches in and pulls, the source holds no such
credential. Restrict the key on the source side to read-only rsync:

# ~/.ssh/authorized_keys on the SOURCE host
command="/usr/bin/rrsync -ro /path/to/backups",no-agent-forwarding,no-port-forwarding,no-pty,no-X11-forwarding ssh-ed25519 AAAA...

With §4b encryption on, the puller only ever holds ciphertext, so the destination does not need to
be as trusted as the source.

• Mount a NAS / external disk as the host ./backups directory (e.g. bind-mount a NAS export or
  an attached volume at deploy/backups). The sidecar writes straight to it.
• Or rsync the sets to another host on a cron:

bash rsync -a --delete deploy/backups/ backup-user@nas.internal:/obscura-backups/

--delete keeps the destination an exact mirror (respecting the retention already applied on the
source). Drop --delete if you want the off-site copy to keep more history than local retention.

Backup files are unencrypted, so treat the destination as sensitive (it contains all document
bytes + the database).

6. Restore procedure

Rehearse it first — restore-drill.sh (no risk, run it monthly)

Until a backup has been restored, "we have backups" is a belief, not a fact. The drill restores a
set into throwaway containers — its own network, randomly-named scratch Postgres and MinIO, no
host ports, destroyed on exit — then asserts the result and reports. It never touches the live
stack, so it is safe to run on a production host during business hours.

./scripts/restore-drill.sh                    # newest set
./scripts/restore-drill.sh 20260729T104123Z   # a specific set

It fails loudly, with a non-zero exit, if any of these does not hold:

Exit 0 means that set restores to a complete, whole records system. Keep the output — it is the
evidence an auditor asks for when they ask whether recovery has ever been tested.

Requires the backup profile image to exist (it ships pg_restore + mc). Build it with
docker compose -f deploy/docker-compose.yml --env-file deploy/mekari.env --profile backup build obscura-backup.

The real thing

Restore is an operator procedure, not an in-app button — it overwrites live data, so it lives on
the host as scripts/restore.sh and demands explicit confirmation. The backup profile must be up
(the restore is driven through the obscura-backup sidecar, which ships pg_restore + mc, so you
need no local Postgres/MinIO tooling).

Before anything destructive it now reads the dump's table of contents and refuses on a truncated
file (exit 3, live system untouched), and checks the blob-key fingerprint, warning loudly on a
mismatch — both failures used to be discoverable only after the live data was gone.

# interactive confirmation (type 'y'):
./scripts/restore.sh 20260706T101112Z

# non-interactive (e.g. from an automation runbook):
CONFIRM=yes ./scripts/restore.sh 20260706T101112Z

What the script does, in order:

1. Verifies the set — refuses unless deploy/backups/<TS>/db.dump and blobs/ both exist
   (a .partial set will not match).
2. Prints exactly what it will overwrite and requires confirmation (CONFIRM=yes, or an
   interactive y/N prompt). Without confirmation it aborts.
3. Stops obscura (docker compose stop obscura) so nothing writes during the restore.
4. Restores Postgres — pg_restore --clean --if-exists --no-owner into the live database. The
   --clean --if-exists drops and recreates every object; the live DB becomes the backup's state.
5. Restores blobs — mc mirror --overwrite --remove from the set's blobs/ back into the live
   bucket. --overwrite replaces changed objects and --remove deletes objects not in the backup,
   so the bucket becomes an exact copy of the set.
6. Starts obscura (docker compose start obscura).
7. Verifies — polls /readyz for a 200 and prints a documents row count as a sanity check.

The script never runs docker compose down -v; it only stops/starts the single obscura
service.

7. Verify a restore

After restore.sh finishes, confirm the system is healthy:

• Readiness: curl -s -o /dev/null -w '%{http_code}\n' http://localhost:38080/readyz → 200
  (the script already waits for this and warns if it did not come up).
• Document count: the script prints documents=<N>. Sanity-check it against what you expect for
  the set you restored (e.g. compare with the source system if this was a migration).
• Open a document: log in and open a document in the UI, and run a search — confirm the file
  bytes are present (proves the blob mirror worked, not just the DB restore) and that content
  search returns results.

8. Troubleshooting

9. The one rule: NEVER docker compose down -v

docker compose down -v deletes the pgdata and miniodata volumes — the live database and all
document bytes — with no undo. It once wiped the demo. To stop the stack use docker compose down
(without -v) or, for a single service, docker compose stop <svc> / docker compose start <svc>.
This backup system is your recovery path if it ever happens again — but don't make it happen.

Key custody

Escrowing the keys (do this once, and after every rotation)

A backup set is ciphertext. The keys are deliberately not in it — ciphertext and its key in one
directory means anyone who copies that directory gets the whole corpus in the clear. That leaves
"back them up separately", which is the instruction everyone means to follow and nobody does, so
there is a command for it:

./scripts/escrow-keys.sh ~/obscura-keys-$(date +%F).age

It writes the blob age identity and AUDIT_CHAIN_KEY into one file encrypted under a passphrase
you choose, verifies the file decrypts before it finishes, and refuses to write into a backup
directory. Store the file somewhere your backups are not, and the passphrase somewhere else again.

To confirm an escrow opens a given backup set, compare the fingerprint in its header with
blob_key_fingerprint in that set's keys.json. Re-run after any key rotation — an escrow of
the old key opens nothing.