obscura-blob-key-rotation.md

← all files

name: obscura-blob-key-rotation
description: "Blob/master key ROTATION tool built 2026-08-08 (--rotate-blob-key + scripts/rotate-blob-key.sh). 🔴 BLOB_ENCRYPTION_KEY_FILE is a DEPLOYMENT master key used by FIVE subsystems, not just blobs — rotating it crash-looped the demo. Rehearsed end-to-end on the demo (963 objects, both directions). NOT yet run on prod: the four DB-stored secret classes still need a rotation pass."
metadata:
type: project

Built because the security scan found blob_age.key at 0644 on prod, and
--backfill-encrypt is not a rotation — it only encrypts objects that are still
plaintext and happily leaves a whole bucket under a leaked key. See
[[obscura-securefolder-security-audit-08-08]].

🔴 The discovery that matters

BLOB_ENCRYPTION_KEY_FILE is a deployment-wide master key. FIVE subsystems read it:
blob storage, e-sign private keys, TOTP secrets, the SMTP relay password, and
AI provider API keys. Rotating "the blob key" re-encrypted the blobs and left the other
four holding ciphertext under a key the process no longer had — the app did not degrade, it
refused to boot (esign CA provisioning: … identity did not match any of the recipients, crash loop). All five now accept retired keys for DECRYPT and encrypt only
with the current key (92061c6).

⚠️ On prod this would have been an outage inside a window opened to improve security —
and by then the rollback (restore the old key) would itself have destroyed every
already-rotated blob. Rehearse key operations on the demo. Always.

Design

Reads accept current OR retired keys; writes only ever use the current one. That
asymmetry is what makes rotation online — mid-run the bucket holds both.
--rotate-blob-key re-encrypts each object in place. --gen-blob-key mints an identity
with the same age library that reads it (a host may have no age-keygen; valbox doesn't).
Skips sf/… — secure-folder objects are sealed by a per-folder key and never carry
the deployment cipher, so re-encrypting one would destroy it. Also skips any key that is
not a bare sha256 (unknown namespace ⇒ skip and report, never guess).
🔴 Free integrity proof: the object key IS the sha256 of its plaintext. After
decrypting it verifies the hash and REFUSES to write on a mismatch. Never overwrite a
blob you have misread with what you think it should be.
Resumable via a header-only trial decrypt with the current key; a half-rotated bucket is
a working state.
scripts/rotate-blob-key.sh --dir X [--resume|--finish]. Honours OBSCURA_ENV_FILE and
OBSCURA_COMPOSE_FILES exactly like update.sh (auto-detecting prod.yml would silently
apply the prod overlay on a demo). The retired key rides a generated override that
exists only for the window, so the next ordinary deploy cannot inherit a half-rotation.

Gotchas hit while building it

🔴 Under set -e a failing command substitution aborts at the ASSIGNMENT, so the
guard on the next line never runs — the operator saw a progress line and silence.
🔴 The keys are 0600 owned by uid 65532, so the deploy user cannot even read one to
copy it. Every key-file operation needs root: as_root tries root, sudo -n, then a
root busybox container writing through the mount (valbox has no passwordless sudo).
The retired-slot copy is checked non-empty BEFORE the new key goes live: an empty retired
slot plus a new current key is the one state that loses everything.

The DB half — `--rotate-db-secrets` (2026-08-08)

Six columns hold master-key-encrypted secrets: ca_certs.key_pem,
signing_certs.key_pem, totp_enrollments.secret, smtp_settings.password_cipher,
ai_provider_settings.api_key, esign_wa_settings.api_key. All six use the SAME wire
format (age ASCII armor, legacy plaintext passed through), so there is ONE rewrap, not
four. Each new ciphertext is decrypted back and compared to the plaintext before it
replaces the only copy of the secret. scripts/rotate-blob-key.sh runs both halves every
time — an operator who runs only one ends up in the exact state this was built to end.

⚠️ A drift guard (rotatesecrets_test.go) counts encrypt call sites against the column
list and found 8 where I had listed 6. Both extras were ReEncryptLegacyKeys writing
columns already listed (the pre-existing plaintext→encrypted migration), so the list was
right — but the symptom of a genuinely missing column is SILENCE: the rotation skips that
secret and the operator destroys a key that was still needed.

🔴 Two shell traps, both of which made a failure look like success:
- docker compose run drains the script's stdin even with -T (-T only stops TTY
allocation). The rotation containers ate the typed DESTROY, so --finish printed every
warning and then quietly did nothing. Fix: </dev/null on every run.
- A read that hits EOF returns non-zero and set -e exits without a word.

Status

Rehearsed on the demo both directions (963 objects rotated, 24 sf/ skipped, second
pass rotated 0), document bytes verified byte-identical through the API each way, and the
demo left on a single key with no override. Suite 68/68 after.

Rehearsed COMPLETE on the demo 2026-08-08, including --finish: 964 blobs + 25 sf/
skipped + 5 DB secrets rewrapped, retired key destroyed, and the app then booted and
served on the new key ALONE (that boot is the proof — the e-sign CA decrypt is what
crash-looped it the first time). Document fingerprints byte-identical, suite 68/68.

✅ PROD ROTATED 2026-08-08 (dms1), ~3 min window

1741 objects + 6 DB secrets (2 CA keys, 3 signing certs, 1 SMTP password). Prod stayed
on its own commit 83bd270 and its own image throughout.

🔴 The technique that made this safe — reuse it. Prod's running binary predated the
tool, and deploying prod was NOT what was asked. So: git worktree add /tmp/… origin/main
→ docker build -t obscura-rotate:tmp → a compose override that sets ONLY
image: obscura-rotate:tmp + the retired-key mount → compose run the one-shot from it.
The serving image, container and checkout are never touched; the override and image are
deleted afterwards. A tool can be newer than the deployment it operates on.

⚠️ The app was STOPPED for the window, and that is a correctness requirement, not
tidiness: the running process holds the OLD key in memory, so a write landing mid-rotation
goes under the old key, is missed by the pass, and becomes unreadable once the old key goes.

⚠️ docker compose run has no --no-build flag (only --build) — it fails the whole
command.
⚠️ The backup sidecar caches the key too — restart it, or its sets record a fingerprint
for a key that no longer exists.

Verified: clean boot with 0 decrypt errors (the e-sign CA decrypt is what crash-looped the
demo, so booting past it IS the proof), /readyz all deps ok, single 0600/65532 key left.
Every one of the 1741 objects was hash-verified against its own content address during the
rotation; an authenticated end-user download was NOT independently checked (needs prod
credentials).