Production Deployment Runbook — 2 VMs, public HTTPS, Casdoor SSO

The end-to-end plan to stand Obscura up for the internal org (≈35 users, full features,
LLM chat outsourced to OpenAI). Target topology:

                        Internet (HTTPS 443)
                               │
                    ┌──────────▼───────────┐
   VM1  APP + DATA  │  host nginx + certbot │  103.30.246.236  (8 vCPU / 32 GB / 300 GB)
   (public IP)      │        └─► web (127.0.0.1:8091) ─► obscura ─► postgres / minio /
                    │            gotenberg / extract / stego / embed  (all localhost-only)
                    └──────────┬───────────┘
                               │ SSH (rsync pull, VM2→VM1)
                    ┌──────────▼───────────┐
   VM2  BACKUP/DR   │  cron rsync + restore │  103.30.246.237  (4 core / 8 GB / 300 GB)
                    │  verification         │
                    └──────────────────────┘

   SSO:  Casdoor at https://x056.sso.val.id  (external, already run by the org)
   AI:   OpenAI api.openai.com  (chat only; document embeddings stay local on VM1)

Everything below is anchored to the artifacts in this repo:
deploy/docker-compose.prod.yml, deploy/prod.env.example,
deploy/nginx/obscura.conf.example, deploy/update.sh.

0. Prerequisites (do these first — they block the cutover)

• [ ] DNS RE-POINT (blocker): dms.val.id currently resolves to 103.30.246.154
  (valbox, the old demo host). It must be re-pointed to VM1 103.30.246.236 before
  certbot on VM1 can issue a cert. The old demo (x056.obscura.val.id → .154) is being
  deprovisioned. Until the A record moves, §4 (TLS) will fail with an ACME challenge error.
• [ ] Casdoor redirect URI: the SSO admin must add
  https://dms.val.id/auth/callback to the Casdoor application
  (8532f70c5d4f7e45ef83) allowed redirect URIs. Without it, SSO login returns
  redirect_uri mismatch. You cannot do this from Obscura — it's a Casdoor-side change.
• [ ] License decision (see §5): full premium features in production mode require a
  GA-pinned build + a signed license. Budget ~15 min for the one-time key ceremony.
• [ ] OpenAI key for chat.

1. VM1 — base host setup

ssh vdi@103.30.246.236

# Docker Engine + Compose v2 plugin (skip if already present).
#
# Deliberately NOT `curl https://get.docker.com | sudo sh`. That hands whatever the
# server returns, unverified, to a root shell — one bad DNS answer, proxy or upstream
# compromise and the host is owned before this document's first deployment step. The
# signed apt repository below is Docker's own recommended path: the key is pinned and
# every package is signature-checked by apt.
sudo apt-get update && sudo apt-get install -y ca-certificates curl git

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /tmp/docker.asc
sudo install -m 0644 /tmp/docker.asc /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo usermod -aG docker "$USER"      # log out/in so the group takes effect

# Host firewall — ONLY ssh + http/https reach the box. This is what makes the
# 0.0.0.0-exposure question moot even if a compose port ever leaks.
sudo apt-get install -y ufw
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw --force enable

Get the code onto VM1 (git clone, or rsync the repo — the workspace already lives at a
known path on the current build host; mirror it):

git clone <repo-url> ~/obscura   &&   cd ~/obscura
# (or: rsync -a the repo to ~/obscura)

2. VM1 — secrets

cd ~/obscura/deploy
mkdir -p secrets

# Blob encryption-at-rest key (documents are encrypted on disk)
docker run --rm dyne/age age-keygen 2>/dev/null | tee secrets/blob_age.key
# (or install age locally: `age-keygen -o secrets/blob_age.key`)

# Casdoor OIDC client secret (from the integration guide) — file, not env
printf '%s' '80da87ba63dda5735ed8ea2d7d670e8550f1ac40' > secrets/oidc_client_secret
chmod 600 secrets/oidc_client_secret

# Forensic-watermark master key — a FRESH per-install secret (never the demo default)
openssl rand -hex 32           # copy this into STEGO_MASTER_KEY in prod.env

3. VM1 — production config

cp deploy/prod.env.example deploy/prod.env

Edit deploy/prod.env and fill:

The Casdoor claim mapping is automatic: Obscura reads sub/email/name from the ID
token. Casdoor's roles claim is not auto-consumed — Obscura uses its own RBAC, so
roles are assigned in-app (or the BOOTSTRAP_ADMIN_EMAIL account grants the first admin,
then assigns the rest). This matches the "Casdoor as IdP only, our built-in RBAC stays"
decision; Casbin is not wired into Obscura.

4. VM1 — TLS (nginx + certbot)

sudo apt-get install -y nginx certbot python3-certbot-nginx

sudo cp ~/obscura/deploy/nginx/obscura.conf.example \
        /etc/nginx/sites-available/dms.val.id
sudo sed -i 's/dms.example.internal/dms.val.id/g' \
        /etc/nginx/sites-available/dms.val.id
sudo ln -s /etc/nginx/sites-available/dms.val.id /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# Issue the cert (certbot rewrites the 80→443 + ssl_* lines automatically):
sudo certbot --nginx -d dms.val.id

5. VM1 — license for full features (production mode)

Production mode (which we run for Secure cookies + no dev-login) trusts ONLY a
vendor-pinned key, so premium modules need a GA build + signed license. Since you are both
vendor and customer for an internal deploy, this is a one-time ~15-minute step.

# On an offline/trusted machine with the repo + Go:
cd go
go run ./cmd/licensegen -gen-key        # → public_key=…  private_key=…  (keep private key SAFE)

# Mint the full-module license (adjust seats/expiry):
go run ./cmd/licensegen \
  -private-key '<PRIVATE_KEY_B64>' \
  -customer 'PT Your Org' \
  -modules correspondence,watermarking,ai,esign,semantic \
  -seats 40 -expiry 2027-12-31 \
  -out ~/obscura/deploy/secrets/obscura.license.json

Then build the image pinned to that public key so it trusts the license. update.sh
passes build args through, so set the pin once via the Docker build arg (add
LICENSE_PUBLIC_KEY=<PUBLIC_KEY_B64> to deploy/prod.env — the Dockerfile reads it).

Simpler interim option if you want to go live today and do the ceremony later: run
local features only (core: documents, workflow, RBAC, retention, e-signing) by
leaving the license out — the server boots core-only in production. Premium modules
(AI, semantic, watermarking, correspondence) light up once the pinned build + license
are in. Do NOT instead run OBSCURA_ENV=development on a public box. Since the
OBSCURA_ALLOW_DEV_LOGIN split, development alone no longer registers dev-login —
but it still relaxes cookie flags and dev channels, and one stray
OBSCURA_ALLOW_DEV_LOGIN=true alongside it is a full auth bypass (that route mints an
admin session for any email, with no credential). Leave both unset in production.

6. VM1 — deploy

update.sh drives the safe path (snapshot → build → migrate → health-gate → auto-rollback),
but on production always go through the wrapper:

cd ~/obscura
./deploy/update-prod.sh --yes

Never run bare ./deploy/update.sh on VM1. It defaults to deploy/mekari.env (the demo
env), which VM1 also has on disk, so a bare run silently builds against demo config:
LICENSE_PUBLIC_KEY is baked into the binary at build time, so an empty one ships a binary
that cannot verify the licence and every licensed module goes dark on the live site; and
APP_BASE_URL falls back to localhost:8091, breaking external-signing links and the WebAuthn
RP id. update-prod.sh pins the overlay + env file and refuses to run if the licence key or an
https:// base URL are missing.

A shell profile is not a sufficient substitute: Ubuntu's ~/.bashrc returns early when
non-interactive, so ssh vm1 './deploy/update.sh --yes' would still pick up the demo env.
The wrapper holds for both interactive and automated runs. It is equivalent to:

export OBSCURA_COMPOSE_FILES="deploy/docker-compose.prod.yml"   # base file is always included
export OBSCURA_ENV_FILE="deploy/prod.env"
./deploy/update.sh --yes

Verify:

curl -s https://dms.val.id/api/v1/version      # {"version":"…","commit":"…"}
curl -s https://dms.val.id/api/v1/auth/methods  # modes:["oidc"], oidc block present

Then open https://dms.val.id, click Sign in with SSO, complete Casdoor, and
confirm the BOOTSTRAP_ADMIN_EMAIL account lands with the admin role.

7. VM1 — turn on backups (local sets)

docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.prod.yml \
  --env-file deploy/prod.env --profile backup up -d obscura-backup

Then in Admin → Backups, enable automatic backups and set a low local retention
(e.g. 3) — VM2 keeps the long history (§8). See docs/BACKUP.md.

8. VM2 — off-box backup + restore verification

ssh vdi@103.30.246.237
sudo apt-get update && sudo apt-get install -y rsync docker.io docker-compose-v2 cron

# SSH key so VM2 can pull from VM1 without a password (add the pubkey to VM1's authorized_keys)
ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519
# → copy ~/.ssh/id_ed25519.pub into VM1:~/.ssh/authorized_keys

mkdir -p ~/obscura-backups
# Hourly pull of VM1's local backup sets (NO --delete → VM2 keeps more history than VM1):
( crontab -l 2>/dev/null; echo '17 * * * * rsync -a vdi@103.30.246.236:~/obscura/deploy/backups/ ~/obscura-backups/' ) | crontab -

# Firewall: VM2 only needs SSH inbound.
sudo ufw allow 22/tcp && sudo ufw --force enable

Monthly restore drill (a backup you've never restored is not a backup): on VM2, bring
up a throwaway Postgres + MinIO, pg_restore the newest dump + mc mirror the blobs into
them, and confirm the app boots against them. VM2's 4-core/8 GB is sized for exactly this.
See docs/BACKUP.md §6 for the restore commands.

Security posture summary (why this is safe to expose publicly)

• Only 22/80/443 reachable (ufw) — Postgres, MinIO, the backend and all sidecars are
  bound to 127.0.0.1 by the prod overlay (!override on the port lists) and blocked by
  the firewall regardless.
• OBSCURA_ENV=production → session cookies are Secure (HTTPS-only), the dev-login
  admin-bypass route is 404, and AUTH_MODE is enforced.
• SSO-first auth (AUTH_MODE=oidc) with a TOTP/passkey-gated break-glass admin path;
  passkeys/WebAuthn and the local-auth hardening (denylist, TOTP replay guard, timing
  equalizer, lockout) all apply. See docs/AUTH.md.
• HTTPS end to end with HSTS; the backend sets Secure cookies and builds https:// links
  off X-Forwarded-Proto.
• Encryption at rest for document blobs (age key), and a fresh per-install forensic
  watermark key.
• Updates go through deploy/update.sh (snapshot + auto-rollback); DR via the
  VM2 off-box copy. Never docker compose down -v.

Ongoing

• Update to a new version: OBSCURA_ENV_FILE=deploy/prod.env ./deploy/update.sh --ref vX.Y.Z
  (see docs/UPDATING.md).
• Rotate the OpenAI key / OIDC secret: edit deploy/prod.env (or the secret file) and
  re-run update.sh.
• Watch Admin → Observability for dependency health + rate-limit rejections.