name: obscura-deploy-procedure
description: "ALWAYS use deploy/update.sh to deploy/redeploy obscura — not raw docker compose build/up (that ships an UNSTAMPED image). Includes how the health gate behaves in Cloud."
metadata:
type: project

Use deploy/update.sh to deploy or redeploy this project — do not hand-roll docker compose build && up -d. Built 2026-07-13 (commit 7a58d17, docs/UPDATING.md) specifically so every agent/session redeploying this shared instance goes through one safe, health-gated path instead of an ad-hoc rebuild that can silently leave the demo half-updated or migrated-but-broken.

Why this matters for a multi-agent/multi-session repo: the working directory at /home/efran/remote-development/obscura is the SAME filesystem on valbox (the live demo host) as in any agent's sandbox — no push/pull step. That means any session's uncommitted edits are already "live" the moment someone rebuilds. update.sh is the one procedure that snapshots the DB first and auto-rolls-back the images if the new build doesn't come up healthy, so a bad concurrent edit (e.g. from another agent mid-task) can't brick the shared demo without a recovery path.

How to invoke it (ssh valbox does NOT keep a cd, always use absolute paths + explicit env file):

ssh valbox "OBSCURA_ENV_FILE=/home/efran/remote-development/obscura/deploy/mekari.env \
  bash /home/efran/remote-development/obscura/deploy/update.sh --yes"

Flags: --ref <tag> (fetch+checkout first), --no-backup (skip DB snapshot — avoid), --app-only (skip the web image), --yes (no interactive prompt, needed for non-interactive agent use).

What it does: pre-update pg_dump snapshot → docker compose build obscura web (version-stamped via git describe/git rev-parse → -ldflags) → up -d obscura web (migrations run automatically on boot, forward-only, goose) → health-gate on /readyz + confirms GET /api/v1/version reports the new build → automatic image rollback if it doesn't come up healthy within HEALTH_TIMEOUT (default 120s), printing the exact pg_restore command if the DB also needs restoring.

What it deliberately does NOT touch: postgres, minio, embed/extract sidecars, stego-sidecar, signadapter — only obscura and web are rebuilt/restarted, so another agent's work on e.g. the stego sidecar is never disturbed by a redeploy of unrelated changes (verified in practice: deploying auth/admin-UI work while a stego session had uncommitted stego-sidecar-side changes in the tree left deploy-stego-sidecar-1 completely untouched).

GOTCHA: the pre-update snapshot goes to deploy/pre-update/ (override via OBSCURA_SNAPSHOT_DIR), NOT deploy/backups/ — that dir is root-owned by the obscura-backup sidecar and the host user (efran) can't write there; the script fails loudly with a clear error if the snapshot dir isn't writable rather than silently skipping.

Version identity, so you can always confirm what's actually running before/after: boot log (version=… commit=…), GET /api/v1/version, Admin → Licensing "Software version" row. Expect a -dirty suffix on git describe whenever ANY session (yours or another agent's) has uncommitted changes in the shared tree — that's correct, not a bug; commit first if you want a clean version string.

Full reference: docs/UPDATING.md (also rendered at https://x056.think.val.id/UPDATING.md). Related: [[obscura-dev-gotchas]] (the snapshot-dir and ssh-absolute-path gotchas are duplicated there too, in the gotchas-list format).

Version stamping — plain compose builds an UNSTAMPED image (2026-07-30)

update.sh is what exports OBSCURA_VERSION/OBSCURA_COMMIT into the Docker build args that become
the binary's ldflags. A bare docker compose build produces an image reporting
{"version":"dev","commit":""}, which blinds GET /api/v1/version, obscura status and the deploy
CLI on that host while git describe says something else. This happened on VM2 (built with plain
compose because of a since-corrected runbook caveat).

🔴 The old git describe … || echo dev fallback was worse than empty: the gate compares the new
container's reported version to $VERSION, so dev == dev PASSED and update.sh certified its own
unstamped image. Now it REFUSES (escape hatch: preset OBSCURA_VERSION for a tarball deploy), and the
server WARNs at boot on an unstamped build outside development (435b36b).

⚠️ An unstamped image currently running does NOT defeat the next run's gate — the gate reads the
version AFTER the new container is up. The pre-update Updating Obscura: dev → … line is a log line
only. Restamping is one ordinary update.sh run.

🔴 --build applies to the whole DEPENDENCY CLOSURE (fixed 2026-07-30, 1a0259e)

docker compose up -d --build <svc> builds/recreates every service <svc> depends on, transitively.
The operator console declared depends_on: [postgres, obscura], so the documented "rebuild the console
separately" step did: build operator + obscura + 3 sidecars, recreate postgres + minio + obscura.

Two consequences, both silent: obscura came back UNSTAMPED (a hand-run compose has no
OBSCURA_VERSION), reverting the stamp update.sh had just applied; and the DATA SERVICES restarted.
Hit on live VM2 by following the runbook literally.

Fixed by REMOVING the operator's depends_on + adding restart: unless-stopped, guarded by
TestOperatorServiceHasNoDependsOn. Key insight worth reusing: depends_on without
condition: service_healthy orders START only — "started" ≠ "accepting connections" — so it never
provided the ordering it appeared to. A restart policy is the real mechanism.
⚠️ web KEEPS its depends_on: obscura: nginx resolves proxy_pass upstream names at startup and
fails to boot if the name does not resolve. Not the same case.
⚠️ ORDER: update.sh FIRST, console SECOND. The reverse un-stamps obscura. Recovery = same order.

The health gate in CLOUD (schema mode)

Both legs work; the caveat that said otherwise was stale from 2026-07-29 (4ccd454).
- GET /api/v1/version is TENANT-EXEMPT on purpose (versionPath in resolveTenant) — 200 on the
apex, on a real tenant host and on an UNKNOWN tenant host. Without it the apex 404s and update.sh
rolls back a healthy deploy. Do not "fix" it back to 404. Tradeoff: it discloses the build
version to an unauthenticated caller; accepted (no tenant data, already visible to every user).
- ⚠️ /readyz is a WHOLE-STACK check — 503 while ANY configured dependency is down (measured:
gotenberg down ⇒ ready:false). A down gotenberg rolls back a build that is otherwise fine. Read
/readyz before blaming a release.