Backup System — Design

Date: 2026-07-06 · Status: Approved (brainstorm)
Module: CORE platform (no license gate). Queue #4 (after LDAP ✅ + SCIM ✅ + Observability ✅). Incident-motivated: a docker compose down -v once wiped the demo pgdata + miniodata with no way to restore.

Goal

A real, restorable backup of the whole system — the Postgres database + all MinIO
document blobs — taken on a schedule, retained, and recoverable via a documented procedure.
The existing internal/backup context only exports a corpus manifest (a JSON inventory
of live documents + content hashes); it can tell you what you lost but not restore it.
This adds the missing disaster-recovery layer around it.

Decisions (from brainstorm)

Backup engine = a compose sidecar, not the app. The obscura container is distroless
(no pg_dump). A dedicated obscura-backup sidecar runs the real, battle-tested
pg_dump/pg_restore + mc (MinIO client). Reliability beats app-integration for the
one system that must actually work.
Restore = a documented restore.sh + runbook, never an in-app button. Restore
overwrites the live DB + blobs — a deliberate, offline, operator-run operation that must
not be fat-fingerable from the UI. The Admin view stays strictly read-only.
Opt-in via a backup compose profile (like ldap/keycloak/peruri), so it never
surprises the demo; production enables it, documented.
Count-based retention (keep last N sets); host-mounted target (./backups bind
mount the operator can point at a NAS/external disk).
Admin-configurable schedule + retention, not a static compose env. The admin sets
the auto-backup interval, retention, and on/off in the app; the sidecar reads those from
Postgres (the shared state it already connects to) each poll cycle — no app↔sidecar RPC.
The admin also sees available backups in the same tab.

Architecture

New compose service obscura-backup (profile backup):
- Image deploy/Dockerfile.backup = FROM postgres:17 (real pg_dump/pg_restore,
version-matched to the pgvector/pgvector:pg17 server) + the mc MinIO-client binary
(downloaded at build time) + backup.sh, restore.sh, entrypoint.sh.
- Entrypoint is a poll loop (no cron daemon needed): every ~60s it reads
backup_settings from Postgres (psql -tAc), derives the last successful backup from the
newest completed set on disk, and runs backup.sh when enabled and
now − last_success ≥ interval_hours — or when an on-demand run_requested_at flag is
set (which it clears after running). A short poll means an admin's schedule change takes
effect within a minute, not a day. Direct on-demand runs still work via
docker compose exec obscura-backup /backup.sh.
- Target: a host-mounted ./backups bind mount (- ./backups:/backups). The obscura
app mounts the same dir read-only (- ./backups:/backups:ro) for visibility only.
- Config (env, reuse the values already in the compose/mekari.env): DATABASE_URL or the
POSTGRES_* parts (postgres:5432, obscura/obscura), the MinIO creds
(minio:9000, obscura/obscura-dev-secret, bucket obscura), and BACKUP_DIR
(default /backups). The schedule/retention/on-off live in the DB (backup_settings),
not env — env only provides bootstrap fallbacks if the table is unreadable.

What a backup set contains

backup.sh for one run:
1. TS=$(date -u +%Y%m%dT%H%M%SZ); work in $BACKUP_DIR/$TS.partial/, rename to
$BACKUP_DIR/$TS/ only on full success (a crashed/partial run never looks complete).
2. db.dump — pg_dump -Fc (custom format: compressed, selective restore).
3. blobs/ — mc mirror <minio-alias>/<bucket> $BACKUP_DIR/$TS/blobs/ (all document bytes).
4. status.json — {ts, result:"ok"|"error", db_bytes, blob_count, blob_bytes, duration_seconds, error?}; on success also overwrite $BACKUP_DIR/latest.json.
5. Prune to the admin-set retention_count (from backup_settings): list completed
sets newest-first, delete beyond N, never delete the newest. (Count-based is more
predictable than age-based.)

Any step failing → write a status.json with result:"error" in the .partial dir (or a
top-level error marker), leave prior good sets untouched, exit non-zero (visible in
docker logs), and skip the rename so the set isn't treated as valid.

Admin-configurable settings (`backup_settings` singleton)

Migration 00087_backup_settings.sql — a singleton table (id=1 CHECK), seeded with safe
defaults:
- enabled bool (default true — but the sidecar only acts if the backup profile is
actually running, so this is a no-op until an operator brings the service up),
- interval_hours int (default 24 = daily; e.g. 1 hourly, 168 weekly),
- retention_count int (default 7),
- run_requested_at timestamptz null (the on-demand "run now" flag),
- updated_at timestamptz.

Anti-footgun floors (validation, like the rate-limit floor): interval_hours ≥ 1 (never
0 → no runaway backups that fill the disk) and retention_count ≥ 1 (never prune to nothing).

API (perm backup.admin, session-authed, in /api/v1, so it IS in OpenAPI):
- GET /api/v1/admin/backups/settings → {enabled, interval_hours, retention_count, floors:{interval_hours:1, retention_count:1}}.
- PUT /api/v1/admin/backups/settings → validates floors (400 backup.floor below min) and
live-applies by writing the row; the sidecar picks it up on its next poll.
- POST /api/v1/admin/backups/run → sets run_requested_at = now() (the "Run backup now"
trigger); the sidecar runs one backup on its next poll and clears the flag. Returns
202 {requested_at}. Idempotent-ish: a pending request isn't stacked.

The app writes these rows; the sidecar reads them (enabled/interval_hours/
retention_count/run_requested_at) via psql each cycle and clears run_requested_at
after an on-demand run. The DB is the only coupling — no direct app↔sidecar call.

Restore (script + runbook)

scripts/restore.sh <TS> (run by the operator, not the app):
1. Print exactly what it will overwrite (DB + bucket) and require explicit confirmation
(CONFIRM=yes env or an interactive y/N prompt) — no accidental runs.
2. docker compose stop obscura (nothing writes during restore).
3. pg_restore --clean --if-exists -d "$DATABASE_URL" $BACKUP_DIR/$TS/db.dump.
4. mc mirror --overwrite --remove $BACKUP_DIR/$TS/blobs/ <alias>/<bucket> (exact blob
state, removing extras).
5. docker compose start obscura; verify /readyz 200 + a document-count sanity check.

docs/BACKUP.md — the operator runbook: enabling the profile, the schedule/retention env,
where sets live, how to copy them off-box, the full restore procedure with the confirmation
gate, verifying a restore, and a troubleshooting table (disk full, MinIO creds, version
mismatch, partial set). Uploaded per CLAUDE.md.

App-side visibility (read-only) + a stale-backup metric

Extend internal/backup with a filesystem catalog that reads the mounted $BACKUP_DIR
(metadata only — it never serves db.dump/blob contents):
- GET /api/v1/admin/backups/status (perm backup.admin, reuse the existing perm) →
{enabled (dir present + ≥1 set), backup_dir, retention_count, last_run (from latest.json: ts,result,db_bytes,blob_count,blob_bytes,duration_seconds), sets:[{ts,result,db_bytes, blob_bytes,blob_count}] newest-first}. Gracefully returns {enabled:false} when the dir
is absent/empty (profile not enabled) — never errors.
- Config: a BACKUP_DIR env on the obscura app (default /backups); when the path
doesn't exist, the feature reports disabled.

Admin → Backups tab (new BackupsTab.tsx, registered in AdminPage), two areas:
- Settings (editable): an auto-backup on/off toggle, the interval (a NumberInput in
hours with the floor, or a friendly Daily/Weekly/Hourly select mapping to 24/168/1),
retention count (NumberInput with floor), a Save button (PUT settings), and a "Run
backup now" button (POST run) — the admin's configure-the-interval control.
- Available backups (read-only): the set history table — each available backup set
newest-first with timestamp, result, DB size, blob count/size — plus last-run status, and
a prominent note "Restore is an operator procedure — see docs/BACKUP.md" (no restore
control in the UI).
The existing corpus-manifest export (POST /admin/backups) is kept as a secondary "Export
content manifest" action for content verification. en/id.

Observability tie-in (the exact blind spot behind the incident): a scrape-time gauge
obscura_backup_last_success_timestamp_seconds (read from latest.json; absent/0 when no
backup) on the existing Prometheus registry, so a stale or failed backup is visible in
/metrics and alertable. Cheap; reuses the Phase-1 collector pattern.

Error handling

A failed backup never corrupts prior sets (atomic rename; prune only completed sets) and
surfaces via non-zero exit + docker logs obscura-backup + status.json result:"error" +
the stale metric. The app's status endpoint tolerates a missing/empty/malformed
latest.json (reports disabled or skips the bad set, never 500s). restore.sh refuses to
run without confirmation and stops the app first. The read-only mount means the app can never
mutate or delete a backup.

Testing (repo discipline: never `go test` on the live DB; NEVER `docker compose down -v`)

Go build/vet + tsc/vite. e2e against the deployed stack, demo-safe:
- Enable the profile; run one backup on-demand (docker compose --profile backup up -d
then docker compose exec obscura-backup /backup.sh). Assert the set is complete:
db.dump non-empty, blobs/ object count == the live bucket's object count,
status.json result:"ok", and the app GET /admin/backups/status lists it with sizes.
- Admin settings: GET /admin/backups/settings shows seeded defaults; PUT changes
interval_hours/retention_count (and a below-floor value → 400 backup.floor); with
the sidecar running, POST /admin/backups/run sets the flag and a fresh set appears
within ~1–2 poll cycles (and run_requested_at is cleared). Confirms admin-configurable
scheduling drives the sidecar via the DB.
- Restore proof into throwaway targets (never the live DB/bucket): createdb obscura_restore_test → pg_restore the db.dump into it → assert key table row counts
match the live DB (e.g. documents, folders, users); mc mirror the blobs/ into a
temp bucket → assert object count matches; then drop the scratch DB + temp bucket. Proves
restore correctness with zero risk to the demo.
- Retention: produce RETENTION_COUNT + 1 sets (vary TS), assert the oldest is pruned
and exactly N remain.
- Assert /metrics exposes obscura_backup_last_success_timestamp_seconds (> 0 after a
successful run). Assert /me 5 modules intact; the backup profile is off by default.
- Single-service teardown only (docker compose stop); clean up throwaway DB/bucket + the
test backup sets.

Out of scope (YAGNI)

In-app restore (the "Run backup now" trigger is in scope; restore stays a documented
operator script), point-in-time recovery / WAL archiving (interval logical dump only),
encryption of the backup archive at rest (operator's disk-level concern; documented),
off-site replication to external S3 (the host dir can be a mounted NAS; mc could target
external S3 later but not built now), backup of mailpit/keycloak/ldap (demo-only services),
incremental/differential backups, a bundled monitoring stack.