Obscura Cloud — deployment runbook

Branch feat/saas-tenancy · 2026-07-29 · Read CLOUD_STATUS.md first —
it lists what is NOT built, and there are things on that list you must not discover in
production.

This deploys the Cloud edition. It does not touch the Enterprise deployments
(dms.val.id, the x056 demo), which keep TENANCY_MODE=single and behave exactly as before.

0. Before you start — the four blockers

1. Wildcard DNS: *.<TENANT_BASE_DOMAIN> → the edge host. Tenants ARE subdomains; without
   this nothing resolves.
2. Wildcard TLS: a cert covering *.<TENANT_BASE_DOMAIN>. Let's Encrypt issues wildcards
   only via DNS-01, not HTTP-01 — so this needs a DNS provider the ACME client can update.
   Budget time for it; it is the step that most often is not ready.
3. A brand-new database. Not a copy of an Enterprise one. See §2 — the boot guard will
   refuse anything that has the single-tenant migration set in public.
4. Nothing bills. Every module you grant is free. Do not onboard a paying customer.

1. Environment

Cloud-specific settings (everything else is as deploy/prod.env.example):

# ---- Edition -----------------------------------------------------------------------------
TENANCY_MODE=schema                 # `single` (default) = Enterprise. This is the switch.
TENANT_BASE_DOMAIN=getobscura.id    # acme.getobscura.id -> tenant "acme". REQUIRED; the
                                    # server refuses to start without it in schema mode.

# ---- Connection budget -------------------------------------------------------------------
# Each ACTIVE tenant holds its own small pool. The ceiling that matters is
# TENANT_POOL_CACHE_SIZE x TENANT_POOL_MAX_CONNS, against your Postgres/pooler max_connections.
# Defaults below are 64 x 4 = 256. Size these BEFORE onboarding tenants.
TENANT_POOL_MAX_CONNS=4
TENANT_POOL_CACHE_SIZE=64

# ---- Operator control plane (optional) ---------------------------------------------------
# Bearer token for /api/v1/control/* — creates, suspends and entitles TENANTS. This is the
# highest-value secret in the deployment: an Obscura admin administers ONE tenant, this
# administers all of them. Min 32 chars. LEAVE UNSET to disable the HTTP surface entirely and
# provision by CLI only — the safer posture, and the recommended one to start.
# CONTROL_PLANE_TOKEN=$(openssl rand -hex 32)

APP_BASE_URL is the apex (e.g. https://getobscura.id). Note that WebAuthn/passkeys bind the
relying-party id to a hostname; with tenants on subdomains this needs deliberate thought before
passkeys are offered in Cloud — untested.

Connection pooling: if you put Supabase/Supavisor in front, use session mode. pgx defaults
to the extended protocol with prepared-statement caching, which transaction-mode pooling breaks.

2. Database

Start from an empty database. On boot in schema mode the server:

1. applies the control-plane migrations only (control schema + vector extension);
2. runs AssertPublicIsClean and refuses to start if public holds any table.

That guard is not pedantry. Tenant connections run search_path = <tenant>, public, so any
table in public is readable from inside every tenant and would silently answer a query for
a table missing from a tenant's own schema. Applying the single-tenant migration set to a Cloud
database populates public in one step — the guard is what stops that becoming a leak.

Application tables are created per tenant, at provision time, not at boot.

3. Bring it up

# update.sh defaults ENV_FILE to deploy/mekari.env, so Cloud MUST name its own or none of
# cloud.env applies. --yes is required over ssh.
OBSCURA_ENV_FILE=$PWD/deploy/cloud.env ./deploy/update.sh --yes

# FIRST bring-up only: update.sh snapshots the database before deploying and aborts with
# "postgres container is not running — cannot snapshot" when there is nothing to snapshot.
OBSCURA_ENV_FILE=$PWD/deploy/cloud.env ./deploy/update.sh --yes --no-backup

# Check what this build says it needs, evaluated against the live environment:
docker compose --env-file deploy/cloud.env run --rm obscura -preflight

update.sh prints the edition it is about to deploy (read from TENANCY_MODE in the env
file) before asking to proceed — check that line says CLOUD. It also now refuses an
OBSCURA_ENV_FILE that does not exist, rather than silently deploying with no settings.

The health gate, and why it works in Cloud

update.sh gates a release on two legs and rolls the images back if either fails:

Both work in Cloud. Measured on a live schema-mode stack:

GET /api/v1/version                             → 200 {"version":"…","commit":"…"}
GET /api/v1/version  Host: acme.<base>          → 200   (a real tenant)
GET /api/v1/version  Host: ghost.<base>         → 200   (an UNKNOWN tenant)
GET /api/v1/documents                           → 404   (tenant-bound paths are still gated)
GET /api/v1/documents Host: ghost.<base>        → 404
GET /api/v1/documents Host: acme.<base>         → 401   (resolved; needs auth)

🔴 /api/v1/version is exempt from tenant binding ON PURPOSE. Do not "fix" it back to 404.
It sits under /api/v1, so without the exemption the apex — which has no tenant — 404s it, and
update.sh concludes a perfectly healthy Cloud deploy failed and rolls the images back. The
exemption is an exact path match in resolveTenant (go/internal/httpapi/tenancy.go,
versionPath), which carries the same warning.

The tradeoff, stated so it is a decision and not an accident: this discloses the build version and
commit to an unauthenticated caller who has not resolved a tenant. That is a minor
fingerprinting surface — it tells someone which build you run, so which known issues apply. It is
accepted because the endpoint returns no tenant data, the same information is already visible to
every signed-in user, and the alternative is either a deploy tool that cannot verify its own release
or one that has to be taught two editions.

⚠️ /readyz is a WHOLE-STACK check, not "is obscura up". It returns 503 while any configured
dependency is down — measured: {"deps":{"engine":"ok","gotenberg":"down","minio":"ok","postgres":"ok"},"ready":false}.
So a down gotenberg makes update.sh roll back a build that is otherwise fine. If a deploy rolls
back with ready=0, read /readyz before suspecting the release.

Build with update.sh, not with plain compose

update.sh is what exports OBSCURA_VERSION/OBSCURA_COMMIT into the build args that become the
binary's ldflags. An image built with a bare docker compose build carries no stamp and reports
itself as "version":"dev","commit":"" — which blinds GET /api/v1/version, obscura status and the
deploy CLI's version reporting on that host, while git describe says something else entirely.

Two guards now exist, because the only other symptom is a cosmetic-looking "dev":

• update.sh refuses to build when it cannot derive a version (previously it fell back to the
  literal string dev, and then its own gate compared dev == dev and approved the unstamped
  image). Set OBSCURA_VERSION explicitly if you are deploying from a tarball with no .git.
• obscura-server logs a WARN at boot when an unstamped build starts outside development. It is
  a warning, not a refusal: the binary is completely functional, and refusing to boot over build
  metadata would turn a reporting gap into an outage.

Recovering a host that was built with plain compose is one ordinary update:

OBSCURA_ENV_FILE=$PWD/deploy/cloud.env ./deploy/update.sh --yes

The "dev" currently running does not stop that gate from passing: the gate compares the version
of the NEW container against what it just built, not against what was running. (The pre-update
Updating Obscura: dev → … line is a log line only.)

⚠️ update.sh builds obscura + web only — NOT the operator console. That container runs under
its own compose profile and is rebuilt by hand, so exporting the stamp is on you:

export OBSCURA_VERSION="$(git describe --tags --always --dirty)" \
       OBSCURA_COMMIT="$(git rev-parse --short=12 HEAD)"
docker compose --env-file deploy/cloud.env --profile operator up -d --build --no-deps operator

Without those two variables the console stamps dev, exactly as obscura does. The two binaries are
deployed by different commands and can legitimately sit on different commits, so "which build is this
console?" is a real question — the console answers it at the foot of its side nav, and in its boot
log, and it WARNs there when it is unstamped.

🔴 Order matters, and this step used to undo the previous one. update.sh FIRST, console SECOND.
The reverse un-stamps obscura.

The console's compose service no longer declares depends_on, which is what made the two steps
conflict: --build applies to a service's whole dependency CLOSURE, so with depends_on: [postgres, obscura] that one command rebuilt obscura and three sidecars and recreated postgres and minio.
obscura came back reporting dev, because a hand-run compose has no OBSCURA_VERSION in its
environment — so following this very section after a release silently reverted the stamp the release
had just applied. It also restarted the data services, which nobody asks for by typing "rebuild the
console".

--no-deps above is belt-and-braces: unnecessary on this compose file now, and still correct on an
older checkout. A test (TestOperatorServiceHasNoDependsOn) fails if the depends_on comes back,
because re-adding it looks like an improvement in review.

If you hit the un-stamped state: run update.sh to restamp, THEN the console command above. The
reverse order un-stamps again. Confirm with GET /api/v1/version and the absence of the
NO version stamp WARN in the obscura boot log.

Expected boot log lines in a healthy Cloud start:

"msg":"starting obscura-server" ... "tenancy":"schema"
"msg":"tenancy: schema-per-tenant routing enabled" "max_conns_per_tenant":4 "pool_cache_size":64
"msg":"control-plane migrations applied"
"msg":"tenant bootstrap sweep complete" "tenants":N

A healthy Cloud boot has zero ERROR-level lines and zero no tenant bound to context.
Both are load-bearing: that message means something is doing application work outside a
tenant, which is a bug rather than a warning to live with. (The first VM2 bring-up surfaced
one ERROR and 17 such WARNs; all are fixed — if they come back, something regressed.)

4. Create the first tenant

CLI (works with the control API disabled):

docker compose --env-file deploy/cloud.env run --rm obscura \
  -provision-tenant=acme \
  -provision-tenant-name="Acme Corp" \
  -provision-tenant-modules=correspondence,esign

HTTP (requires CONTROL_PLANE_TOKEN):

curl -X POST https://getobscura.id/api/v1/control/tenants \
  -H "Authorization: Bearer $CONTROL_PLANE_TOKEN" -H 'Content-Type: application/json' \
  -d '{"id":"acme","name":"Acme Corp","modules":["correspondence","esign"]}'

id must match ^[a-z][a-z0-9_]{0,60}$ — it becomes both the subdomain label and the schema
name. Reserved labels (www, api, admin, auth, app, …) are refused.

Provisioning is synchronous (~1s): it creates the schema, runs the full migration set into it,
issues that tenant's own CA, registers its scheduled tasks, then activates. A failure leaves
the tenant in provisioning — visibly incomplete rather than servable-and-broken.

Operations:

# suspend (billing/abuse hold — resolves but is refused, effective immediately)
curl -X PATCH .../control/tenants/acme/status -d '{"status":"suspended"}'
# replace entitlements (whole set, not a delta)
curl -X PUT   .../control/tenants/acme/modules -d '{"modules":["esign"]}'

A suspended tenant is TOLD it is suspended. Every request to its subdomain — including sign-in —
answers 403 tenant.suspended, and its app renders "This account is suspended … nothing has been
deleted" in place of every screen, live, without a reload. Effective within 30s (the registry cache
TTL). It deliberately does not say WHY: billing, abuse and a lapsed contract are between you and
whoever signed the contract, and that page is reachable by anyone who knows the subdomain.

Unknown, closed and provisioning subdomains keep the uniform 404 tenant.unresolved. Only
suspension identifies itself, because the people who meet it are the customer's own staff — the ones
who otherwise watch their system apparently evaporate and cannot tell an outage from a hold.

4a. Demo mode (only for a tenant whose password is published)

A demo publishes its administrator credential, which inverts the assumption every control in
Obscura rests on: that whoever holds users.admin wants the deployment to keep working. The
next person to sign in is a stranger, and the most ordinary acts — change my password, disable
that account, delete this user — lock out every later visitor. It takes no malice.

DEMO_MODE=true
DEMO_ADMIN_EMAIL=demo@getobscura.id      # required; the server refuses to start without it
DEMO_ADMIN_PASSWORD=<the published one>  # the credential demo.restore_admin puts back

Two halves, split by "does it lock people out":

• Refused at the request. Deleting, renaming, disabling, password-resetting, expiring,
  quota-ing or un-admining the demo account; that account changing its own password; and
  PUT /admin/auth-settings for everyone (its "require TOTP" locks out a shared account
  instantly, and no password restore can enrol a second factor for a stranger). The refusal is
  403 demo.protected and explains itself. Everything else — every other account, unlock, TOTP
  reset, sign-out-everywhere — works normally.
• Healed on a timer. demo.restore_admin runs every 5 minutes and repairs what the guard
  does not cover: re-enables, unlocks, puts the password back, re-binds the admin role. It
  writes nothing when the account is healthy, and it never creates one — an absent account
  is a no-op, which is what scopes it to the tenant that actually has it rather than to every
  tenant on the deployment.

The demo admin keeps every permission and the full UI. The guard is on a handful of
operations, not on the role — a demo admin who cannot open Roles is demonstrating a different
product. It also means the guard cannot be escalated around: an account with rbac.admin can
grant itself anything, so a guard expressed as a missing permission is one API call from being
removed.

DEMO_ADMIN_PASSWORD is a plaintext password in an env file, which is normally indefensible
and is fine only here: on a demo the value is public by construction. Never set any of this
on a deployment holding real data — this is a usability guard, not a security boundary.

Preflight reports both settings, and DEMO_MODE=true with no DEMO_ADMIN_EMAIL refuses to
boot (a demo mode protecting nothing reads as protection while matching no account at all).

4b. Operator console (obscura-operator)

The web console for tenant management — provision, suspend, entitle, seed administrators,
read usage reporting — at its own subdomain, behind its own accounts. It is a separate
binary and container from obscura-server: Enterprise never deploys it, it holds
CONTROL_PLANE_TOKEN server-side so no browser ever sees it, and its accounts live in
control.operators, a different trust domain from tenant users that nothing inside a tenant
can escalate into.

Naming

One deployment = one console. The console's hostname belongs to the deployment it
administers, not to any tenant in it:

The demo's console keeps its own name permanently, so the URL in a sales deck never
breaks when prod launches. admin is in reservedSubdomains, so no tenant can ever claim it.

⚠️ admin.demo.getobscura.id is two labels deep, which the *.getobscura.id wildcard
covers for neither DNS nor TLS. It needs its own A record and its own certificate:

certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/cloudflare.ini \
  -d admin.demo.getobscura.id

⚠️ Prod must never provision a tenant named demo — the edge sends that hostname to VM2,
so the tenant would be unreachable.

Two roles

The viewer is why the console itself is demoable: a viewer with a published password is
safe because a read-only account cannot lock anyone out, so it needs none of the
guard/restore machinery the tenant demo does. Viewers also cannot rotate their own password
(an owner issues it) — otherwise the first visitor orphans the published credential.

🔴 The published viewer belongs on the DEMO deployment only. On a deployment with real
customers it would leak the customer list to anyone with the URL.

Bring it up

# The console runs under its own compose profile, so nothing else starts it.
docker compose --env-file deploy/cloud.env --profile operator up -d --build operator

# First owner (prints a one-time password; TOTP is forced at first sign-in):
docker compose --env-file deploy/cloud.env --profile operator run --rm operator \
  -bootstrap-owner=you@valid.id

# The published read-only viewer — DEMO DEPLOYMENT ONLY. Idempotent: re-run after every
# nightly rebuild; it heals a drifted account and refuses to demote an existing owner.
docker compose --env-file deploy/cloud.env --profile operator run --rm operator \
  -seed-viewer=viewer@getobscura.id -seed-viewer-password='TryOperator2026!'

CONTROL_PLANE_TOKEN must be set (same value as obscura-server) — the console refuses to
boot without it, because a console whose every button 401s reads as broken product.

Edge

server {
    server_name admin.demo.getobscura.id;
    location / { proxy_pass http://127.0.0.1:38081; proxy_set_header Host $host;
                 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }
}

🔴 Block the raw control API on every PUBLIC vhost. Once CONTROL_PLANE_TOKEN is set,
obscura-server accepts it from any host including tenant subdomains — the token is the
only gate. Add to the tenant and apex vhosts (NOT the console's):

location /api/v1/control/ { return 404; }

Then the console is the only door, and every tenant mutation is attributed in
control.operator_audit (append-only by trigger).

Reporting

control.collect_tenant_stats snapshots per-tenant usage daily (Cloud only) into
control.tenant_stats; the console reads snapshots and never fans out across tenant schemas
on page load. A tenant provisioned since the last run shows "no data yet" rather than zeroes.
That table is deliberately the future metering source — billing will read it rather than
grow a second pipeline.

Plans and commercial terms

Plans are named module bundles (control.plans, seeded with Starter / Professional /
Enterprise). Applying one COPIES its modules onto the tenant — control.tenant_modules remains the
entitlement the request path reads, so editing a plan changes nothing for existing customers until it
is applied to them. The console reports drift from the plan rather than reconciling it, because a
customer with a hand-granted extra module is a real situation. A plan may include a module the
DEPLOYMENT licence does not cover: the entitlement is stored, stays inert, and activates by itself if
a licence covering it is later installed.

Terms are one end date per tenant plus a "this is a trial" flag, and auto_suspend, which is
off by default:

• With it off, the date is recorded and surfaced (tenant list badge, detail panel, an attention list
  on Overview) and nothing happens automatically.
• With it on, control.enforce_tenant_expiry suspends the tenant within the hour of the date
  passing, logs at WARN, and writes tenant.auto_suspended into control.operator_audit as
  system.

⚠️ The sweep is a scheduled task, so it needs the worker role. A deployment running -role=api
only will show terms and never enforce them.

Reactivating a tenant the sweep suspended, while its date is still in the past, turns auto_suspend
OFF and reports that in the response — otherwise the next sweep would undo the operator within the
hour. Move the date first if you want the automation to stay armed.

Self-serve signup requests (off by default)

SIGNUP_ENABLED=true exposes POST /api/v1/signup on the APEX — an unauthenticated form
endpoint that writes one row into the operator console's inbox and provisions nothing. Off by
default, and the route is not registered at all when off (same posture as CONTROL_PLANE_TOKEN).

The FORM belongs on the marketing site; this repo ships the endpoint. Contract:

POST https://getobscura.id/api/v1/signup     // apex, no tenant, no auth
{ "requested_id": "ptmaju",                  // optional; a WISH, not a reservation
  "org_name": "PT Maju Bersama",             // required
  "contact_email": "rina@maju.co.id",        // required
  "contact_name": "Rina", "phone": "…", "message": "…" }
→ 202 { "status": "received", "detail": "…" }        // ALWAYS, new or updated
→ 400 { "code": "tenancy.signup.invalid", … }        // only for the caller's own bad input

🔴 It never tells the caller whether a subdomain is taken, and must not be made to. A form
replying "acme is taken" is a customer-list lookup service anyone can query; collisions surface to
the operator at approval time, where the id field is editable.

Anti-abuse already in place: the per-IP auth rate-limit budget, a 1 MiB body cap, per-field caps, and
ONE pending request per email address (a resubmission updates that row rather than adding a copy).
source_ip is recorded for triage.

Approving provisions the tenant from what the OPERATOR types — id, name and plan — never from the
request. Optionally emails the administrator their invitation. Rejecting needs a reason and can email
it verbatim.

Licensing: the DEPLOYMENT holds one licence, tenants hold none

A frequent question, so it is written down: a tenant never uploads a licence. There is exactly
one licence file per deployment — yours — installed once by whoever runs the Cloud.

The entitlement a customer actually gets is the intersection:

what a tenant can use  =  the DEPLOYMENT licence  ∩  control.tenant_modules

• The licence is the outer bound: what this deployment is permitted to run at all. One signed
  file, pointed at by LICENSE_FILE and verified against the bundled public key.
• control.tenant_modules is the per-tenant half: what THIS customer bought. You set it from the
  operator console (module editor, or by applying a plan). No file, no certificate, no upload.

So the flow is one step, not two: sell the customer a plan, tick the modules in the console, done.
Nothing is expected of the customer's administrator, and there is no second artefact to keep in sync.

⚠️ POST /api/v1/admin/license is not mounted in Cloud. In Enterprise a tenant admin uploads the
licence because they are the deployment. In Cloud the same route would let any tenant administrator
replace the licence for every other tenant on the box, so the route is not registered at all when
TENANCY_MODE=schema — guarded by TestLicenseUploadIsMountedOnlyInEnterprise, which walks the real
route tree rather than trusting the code to read correctly. Admin → Licence stays a read-only view of
what the deployment holds.

A plan may name a module the licence does not cover: the entitlement is stored, stays inert, and
starts working by itself once a licence covering it is installed. Nothing to re-apply per tenant.

Quotas (storage and seats)

Per-tenant ceilings, set from the operator console (Tenant → Quotas) or the control API:

PUT /api/v1/control/tenants/{id}/quotas   # {"storage_quota_bytes": 5368709120, "user_quota": 25}
GET /api/v1/control/tenants/{id}/usage    # measured now, not the daily stats snapshot

Both default to no ceiling, and null clears one — that is how "unlimited" is expressed, so
there is no second endpoint to remember.

What being over a ceiling does, and what it does not:

Deleting is how a customer gets back under the line, so it is never blocked; reads are never blocked
because withholding a customer's own documents over a storage limit is not something this product
does. Suspension stays a deliberate operator act.

• Storage is enforced at the blob store — the one place every byte passes — after dedup (content
  they already hold costs nothing) and before the upload. So a tenant can finish at most one file
  over its ceiling; a check mid-write would tear the object instead.
• Seats count ENABLED accounts, so disabling a leaver frees a seat immediately. Enforced on
  self-registration, admin account creation and SCIM. ⚠️ NOT on first-time SSO sign-in, on
  purpose: that presents as "SSO is broken" to someone who cannot fix it and has no administrator
  present. A tenant that drifts over that way shows as an overage on the console.
• Usage is measured, cached for a minute. The ceiling is therefore approximate by design. Raising
  it takes effect immediately (both caches drop), so an operator can unblock a customer mid-call.
• It fails OPEN. If usage cannot be measured, the write is allowed. A database hiccup must not
  stop every tenant on the deployment from saving work to enforce a commercial number.
• Units are BINARY (GiB), in the console and in the refusal message the customer reads. A ceiling
  of 0 is refused — that is a suspension written in the wrong column.

Business verification (KYB) and the regulated-module gate

control.tenant_kyb holds one verification record per tenant. Granting a regulated module is
refused without a current approval — today that is esign, which carries certified e-signature,
e-Meterai and e-Stamp. Reselling those is compliant only because every purchaser is a verified
business, so the gate covers every path that writes entitlements: the module editor, applying a plan,
a bulk plan apply, and provisioning.

PUT  /api/v1/control/tenants/{id}/kyb            # record the entity + what was checked
POST /api/v1/control/tenants/{id}/kyb/decision   # {"decision":"approved"|"rejected", …}
GET  /api/v1/control/kyb?status=submitted        # the review queue

• Approving grants nothing. It removes the obstacle; the grant stays a separate audited act.
• The gate never revokes. A tenant that already holds a regulated module keeps it, so tenants
  granted before the gate existed (or whose approval lapsed) are a compliance backlog the console
  lists on the Verification page. Verify them, or withdraw the module deliberately.
• A lapsed approval stops NEW regulated grants and revokes nothing. Set a re-verification date on
  approval to get that behaviour; leave it empty for an open-ended approval.
• It fails closed: if the record cannot be read, the grant is refused.
• ⚠️ Not applicable in Enterprise — no control plane; the licence is the gate.
• The signatory's national ID number is deliberately NOT stored. See migration 00010.

Emailed administrator invitations

Seeding an administrator emails them the sign-in link and one-time password by default, and the
operator never sees the credential. The link is built from the TENANT's origin
(https://<id>.<TENANT_BASE_DOMAIN>), not the apex.

If delivery fails, the response carries invitation_emailed: false, the reason, AND the password —
the account already exists at that point, so withholding it would leave an administrator nobody can
sign in as. The console shows both outcomes.

⚠️ The SMTP channel is always constructed (SMTP_HOST defaults to localhost), so "no mail
configured" is not a state the endpoint can detect from config alone. A dead relay fails loudly and
returns the password; a reachable but misconfigured one reports success.

Prove the relay works once per deployment before relying on any of this: a tenant admin can
configure and TEST it in Administration → Notifications (POST /api/v1/admin/smtp/test), which
is the authoritative check. In Cloud that is per tenant — the settings row lives in the tenant's own
schema — so a tenant whose relay is unset falls back to whatever the environment gave the deployment.

5. Verify

curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: acme.getobscura.id'  https://getobscura.id/api/v1/me   # 401 = tenant resolved, needs auth
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: ghost.getobscura.id' https://getobscura.id/api/v1/me   # 404 = unknown tenant
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: getobscura.id'       https://getobscura.id/api/v1/me   # 404 = apex is not a tenant

A suspended tenant returns 403 tenant.suspended — the one resolution failure that
identifies itself, because the people who meet it are that customer's own staff (see §4). Unknown,
closed and provisioning all return the uniform 404.

Then confirm isolation directly:

-- each tenant has its own schema, its own CA, its own tasks; public stays empty
SELECT table_schema, count(*) FROM information_schema.tables
 WHERE table_schema NOT IN ('pg_catalog','information_schema') GROUP BY 1 ORDER BY 1;

6. Rollback

Cloud is a separate stack from Enterprise, so rollback is the ordinary
deploy/update.sh snapshot/rollback path and affects no Enterprise install.

Offboarding a tenant

Use the console (Tenants → the tenant → Close, then Offboard) or the API. Do not hand-run
DROP SCHEMA — it leaves every one of that tenant's documents in the bucket, unreferenced
and unfindable, which is a poor answer to "you deleted my data".

# 1. Close it. Reversible; the subdomain stops resolving immediately.
curl -X PATCH .../control/tenants/acme/status -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"status":"closed"}'

# 2. Purge it. Irreversible. Refused unless the tenant is already `closed`, and `confirm` must
#    name it exactly — so no single request can destroy a serving customer.
curl -X DELETE .../control/tenants/acme -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"confirm":"acme"}'
# → {"purged":"acme","schema":"t_acme","blobs_removed":312}

Order is objects → schema → registry rows, so a failure anywhere leaves a retryable state
rather than orphaned bytes. blobs_removed is reported because "we deleted your data" should be
a number rather than a claim. The console writes tenant.purge_attempted BEFORE acting and
tenant.purged after, so a purge that dies mid-way is still attributable.

🔴 There is no export. Take a database dump and a bucket copy of the acme/ prefix BEFORE
purging if the data may ever be wanted.

7. Edge configuration (observed on VM2, Ubuntu 24.04 / nginx 1.24)

• proxy_set_header Host $host; is load-bearing. The entire tenancy model reads the Host
  header; rewrite or drop it and every tenant 404s.
• nginx 1.24 rejects the standalone http2 on; directive — use listen 443 ssl http2;.
• certbot certonly with --dns-cloudflare does not install options-ssl-nginx.conf or
  ssl-dhparams.pem. A vhost that includes them will fail to start; inline the TLS settings.
• ufw does not cover Docker-published ports. Proven twice (VM1 and VM2): ufw allowed only
  22 while 8080 answered from a third host. Use the obscura-firewall.service pattern.

8. Operating notes

• Connection ceiling is the first thing that will bite as tenants grow (§1).
• Suspended tenants are skipped by all background work — their audit chain and outbox pause
  and catch up on reactivation (both are append-only).
• Shared sidecars (gotenberg, extract, embed, stego, OnlyOffice) serve all tenants with no
  per-tenant quota. The stego sidecar's known OOM is an all-tenant event.
• Per-tenant secrets are not implemented: AUDIT_CHAIN_KEY, STEGO_MASTER_KEY and the blob
  DEK are deployment-wide. A tenant cannot independently verify its own audit chain. The CA
  is per tenant.
• Blob keys are <tenant>/<hash>. Cross-tenant dedup is deliberately gone; it was a
  data-loss path (see CLOUD_STATUS.md).