Updating Obscura — Operator Guide

This is how a running Obscura deployment moves to a new version safely — for us and for
any future client. The design goal is that a routine update is one command, is reversible,
and never leaves the instance in a broken state without a clear recovery path.

TL;DR

# On the host, from the repo root:
git fetch --tags
./deploy/update.sh --ref v1.4.0        # or just ./deploy/update.sh to deploy the current checkout

update.sh takes a pre-update database snapshot, builds the new version-stamped images,
applies migrations, restarts only the app (+ web) container, waits for a healthy boot, and
automatically rolls the images back if the new build doesn't come up. It never touches
the Postgres / MinIO data volumes.

What a version is

Every image is stamped at build time with a version (git describe) and commit.
You can always see exactly what a deployment runs, three ways:

• the boot log: obscura-server … version=v1.4.0 commit=abc123…
• GET /api/v1/version → {"version":"v1.4.0","commit":"…"}
• Admin → Licensing → Software version

Releases are tagged with semantic versioning: vMAJOR.MINOR.PATCH.
Tag a release with git tag -a v1.4.0 -m "…" before building so the stamp is meaningful.

What update.sh does, step by step

1. Resolve the target — with --ref it git fetches and checks out that tag/branch/sha;
   otherwise it builds the current checkout. It computes the version stamp from git.
2. Snapshot the database — a compressed pg_dump into deploy/pre-update/pre-update-<ts>.dump
   (override with OBSCURA_SNAPSHOT_DIR; the dir must be writable by the user running the
   script). This is the recovery point for a bad migration. Skip with --no-backup (not
   recommended).
3. Record the current images — tags what's running now, by image ID, as the rollback target.
4. Build the new obscura (+ web) images with the version stamp — or, with --pull,
   fetch the images CI already built and verify their licence trust anchor (see below).
5. Migrate + restart — docker compose up -d obscura web. Migrations are forward-only
   and run automatically at boot (MIGRATE_ON_BOOT=true); bringing the app up applies them.
6. Health-gate — polls /readyz (up to HEALTH_TIMEOUT, default 120s) and confirms
   /api/v1/version now reports the new version. ⚠️ /readyz is a whole-stack check: it is 503
   while any configured dependency is down, so a down gotenberg or MinIO rolls back a build that is
   otherwise fine. Read /readyz before suspecting the release.
7. On failure, roll back — retags the previous images and restarts. If the new binary had
   already migrated the database, it prints the exact pg_restore command to recover from the
   snapshot taken in step 2.

Flags: --ref <tag>, --no-backup, --app-only (skip the web image), --pull (see below),
--yes (no prompt).
Env: OBSCURA_HEALTH_URL (default http://localhost:38080), HEALTH_TIMEOUT,
OBSCURA_ENV_FILE, BACKUP_DIR, OBSCURA_VERSION (see below).

Prebuilt images (--pull)

Step 4 above compiles a Go dependency graph and bundles a SPA on the deployment host. On a
small box that is the whole deployment: the Cloud host is 4 cores with ~14% CPU steal, and a
deploy drove it into swap for minutes rebuilding sources that had not changed.

CI (.github/workflows/images.yml) builds the same three images once per push to main and
publishes them to GHCR. --pull fetches those instead of building:

OBSCURA_ENV_FILE=deploy/cloud.env ./deploy/update.sh --pull --ref <version> --yes

Everything else is unchanged — same snapshot, same health gate, same automatic rollback. Only
step 4 differs. Building from source remains the default and the fallback; drop --pull and the
script behaves exactly as it always has.

One-time setup per host (the packages are private):

echo "$GITHUB_TOKEN" | docker login ghcr.io -u <github-user> --password-stdin

One-time setup for the repository: add the vendor's base64 Ed25519 public key as the
LICENSE_PUBLIC_KEY repository secret. Without it CI publishes only the -devkey variant, and
--pull on a host that pins the release anchor fails to find its tag. That failure is deliberate
— see below.

Release channels and the update button

A channel is a floating registry tag a deployment follows, so a host can answer "is there an
update?" itself instead of an operator remembering a version number.

Set per host with OBSCURA_UPDATE_CHANNEL in the env file. Unset, a host carrying prod.env
defaults to stable and everything else to nightly — the default is derived from the profile
rather than fixed, because defaulting a production box to nightly is exactly the quiet mistake
this is meant to prevent.

./deploy/obscura check-update   # read-only: running vs available. Costs a few KB, not 185 MB.
./deploy/obscura update         # the button: doctor → backup → pull → health-gate → verify

check-update exits 0 up to date, 10 update available, 1 could not tell — so it can
drive a timer or a monitor. It reads the version label off the remote manifest without
downloading any layers.

Why stable refuses a -dirty build: a dirty image was built from uncommitted changes, so
its source exists on exactly one machine and cannot be reviewed, reproduced, or rolled forward
from. That is acceptable on a demo box and never on production, so it is refused rather than
trusted to whoever moved the tag. deploy/release.sh cannot move stable at all — only a
pushed git tag built in CI can, which means a laptop cannot reach production even by accident.

Why bare --pull refuses a dirty checkout: a related but separate rule, and the one you
will actually hit. Without --channel, the image to pull is named from git describe --tags --always --dirty against the tree you are standing in. If that tree has uncommitted changes the
stamp ends in -dirty, and CI publishes images for commits — never for a working tree only your
host has seen — so the tag cannot exist. This is the normal state of a shared checkout like
valbox, where some agent usually has work in progress, and it is why the demo builds from source
instead of pulling. Three ways out, in the order you probably want them:

./deploy/update.sh --channel nightly   # deploy a PUBLISHED version, ignoring this tree's state
git stash                              # or commit — then --pull names a real commit
./deploy/update.sh                     # or drop --pull and build from source

Cutting a release

./deploy/release.sh --tag v1.4.0   # stable: tags the commit, pushes it, CI builds and publishes
./deploy/release.sh                # nightly: builds THIS tree (may be dirty) and publishes it

The stable path refuses a dirty working tree: a tag must name a commit that fully describes what
was built.

🔴 Why there are two server images

The server image is published twice per version:

They are not interchangeable, and using the wrong one does not produce an error. A binary
pinned to one anchor simply fails to verify a licence signed by the other, and the deployment
degrades to core-only: it boots, it serves, /readyz is green, the health gate passes, and every
paid module is quietly absent. Nothing points at the image.

So --pull does not let you choose. It reads LICENSE_PUBLIC_KEY from the env file you named,
picks the matching variant, and then verifies the choice: CI labels each image with the
SHA-256 of the anchor it was built with, and the script compares that label against the env file
before recreating anything. A mismatch, or an image with no such label, is refused with nothing
changed. Deploy from source (--pull omitted) if you ever need to bypass this.

Always build through this script

update.sh is what turns git describe into the ldflags that stamp the binary. An image built with
a bare docker compose build carries no stamp and reports "version":"dev","commit":"" — so the boot
log, GET /api/v1/version, obscura status and the deploy CLI can no longer tell you what is
running, while git describe says something else. It has happened on a live box; the only symptom is
a cosmetic-looking "dev".

Two guards:

• The script refuses to build when it cannot derive a version. It used to fall back to the literal
  string dev, which was worse than failing: step 6's gate compares the new container's reported
  version against the one just built, so dev == dev passed, and the script certified an
  unstamped image. Deploying from a tarball with no .git? Set OBSCURA_VERSION yourself.
• The server WARNs at boot when an unstamped build starts outside development — a warning, not a
  refusal, because the binary works fine and refusing over build metadata would turn a reporting gap
  into an outage.

Note on step 6: it reads the version AFTER the new container is up, so an unstamped image currently
running does not stop the next update.sh from gating correctly.

Migration policy (why this is safe)

• Migrations are append-only and forward-only. Numbered files under go/migrations/ are
  embedded in the binary and applied in order by goose at boot. A new version only ever adds
  migrations; it never rewrites or removes an existing one. This is what lets an older or a
  newer binary boot against the same database without surprises — goose only runs versions
  greater than the current one.
• There is deliberately no automated "migrate down" in production. Rolling code back is
  instant (the image swap); rolling a schema change back is not generally safe to automate,
  so the pre-update DB snapshot is the recovery mechanism instead.
• Practical rule for authors: a migration that must be reversible in the field should be
  written expand/contract (add the new column/table in release N, start writing it, stop
  reading the old shape in N+1, drop the old shape in N+2) so no single release both adds and
  destroys. For routine additive changes this is automatic.

Full disaster recovery

update.sh's snapshot covers the database. Document blobs in MinIO are append-only across
releases, so an update doesn't endanger them — but for genuine disaster recovery (a wiped
volume, a host loss) run the scheduled backup profile, which captures Postgres and the
blob store together, and restore with scripts/restore.sh. See docs/BACKUP.md.

⚠️ Never run docker compose down -v — the -v wipes the pgdata + miniodata
volumes. update.sh only ever recreates the app containers, never the data services.

Air-gapped / no-registry clients

A client with no access to your build machine or a registry receives an update as two files:

# On the build machine — produce a version-stamped image tarball:
make bundle                       # → deploy/obscura-<version>.images.tar.gz
git archive --format=tar.gz -o deploy/obscura-<version>.src.tar.gz <tag> \
    deploy/docker-compose.yml deploy/update.sh docs/UPDATING.md docs/BACKUP.md

Deliver both. On the client host:

docker load < obscura-<version>.images.tar.gz    # loads deploy-obscura + deploy-web
# then run the update WITHOUT rebuilding — the images are already present:
#   (bring the app up on the loaded images; migrations run on boot)
docker compose -f deploy/docker-compose.yml up -d obscura web
docker compose -f deploy/docker-compose.yml logs -f obscura | grep version=

The pre-update snapshot step of update.sh is still worth running by hand first
(docker exec <pg> pg_dump -U obscura -d obscura -Fc > pre-update.dump) so the client has a
rollback point even offline.

Licensing across updates

Module entitlements come from the signed offline license file, which is independent of
the software version — updating the binary does not change what's licensed. A GA build pins
the vendor trust anchor (make release-image LICENSE_PUBLIC_KEY=…); see LICENSING.md. The
license survives updates untouched (it's a mounted file, not baked into the image).

Checklist for cutting a release

1. Merge to main, ensure go test ./... + the web build are green.
2. git tag -a vX.Y.Z -m "…" and push the tag.
3. Build + verify: make bundle (or deploy straight with ./deploy/update.sh --ref vX.Y.Z).
4. Confirm GET /api/v1/version and Admin → Licensing show vX.Y.Z after the update.