Obscura Cloud — what is built, and what is not

Branch feat/saas-tenancy (pushed; unmerged) · 2026-07-29 · DEPLOYED on VM2 at
https://demo.getobscura.id

Companion to SAAS_FEASIBILITY.md (why) and
SAAS_FORK_PLAN.md (plan). Those two are largely DESIGN. This file is
the honest inventory of working code, so nobody plans against something that does not exist.

The model, in one paragraph

One repo, two editions, selected by config. Obscura Enterprise is TENANCY_MODE=single
(the default) and is byte-identical to the pre-tenancy build — no host parsing, no registry,
one pool, workers unwrapped. Obscura Cloud is TENANCY_MODE=schema: one Postgres schema
per tenant, resolved from the request host, with a control plane in its own control schema.
There is no tenant_id column anywhere, which is why ~795 queries and 64 unique constraints
were never touched.

BUILT (verified against a real Postgres + object store)

Area	What works	Commit
Query routing	Per-tenant pgxpool, `search_path` pinned at `AfterConnect`, LRU-bounded. Unscoped request in Cloud gets a fail-closed executor that errors on every call rather than the default pool.	`eb1b4f4`
Tenant identity	`kernel.TenantID`, validated `^[a-z][a-z0-9_]{0,60}$` (schema names cannot be bind parameters); `t_` prefix; 61+2 = exactly 63 so Postgres can never truncate two tenants onto one schema.	`eb1b4f4`
Host resolution	`<tenant>.<TENANT_BASE_DOMAIN>` → context, at the router root. Label-boundary suffix match, nested subdomains and reserved labels refused, unresolvable hosts 404 (no enumeration oracle).	`f0f139d`
Machine callbacks are tenant-bound	The doc-server reaches us at an internal container address and the PAdES signer timestamps over loopback — neither can name a tenant. Office tokens now carry theirs (verified before it is believed, then through the same registry gate), and the timestamp request carries the tenant's Host while still dialing the configured URL.	`f8e6d30`
Quotas	Per-tenant storage and seat ceilings, `NULL` = none. Over a ceiling refuses new uploads and new accounts and nothing else — reads, deletes and the tenant itself keep working. Enforced at the blob store (the one place every byte passes) and on the deliberate provisioning paths. Fails OPEN on a measurement error, on purpose.	`83a45f1`
Suspension is visible	A suspended tenant answers `403 tenant.suspended` on every route including sign-in, and its app renders "This account is suspended — nothing has been deleted" over every screen, live, with no reload. No reason given. Unknown/closed/provisioning keep the uniform 404.	this batch
Licensing is the operator's	One licence per DEPLOYMENT; a tenant uploads nothing. `POST /admin/license` is not mounted in Cloud (it swaps a process-wide pointer, so a tenant admin could have stripped every other tenant's modules), and `/me.license.managed` tells the tenant's Licensing tab to stop offering the upload.	this batch
Registry + provisioning	`control.tenants`, lifecycle `provisioning→active↔suspended→closed`; per-schema migration fan-out (same migration set, `search_path` pinned as a connection parameter); `-provision-tenant` CLI.	`702254f`
Entitlements	`control.tenant_modules`; the gate is licence ∩ tenant — neither alone suffices, so a control-plane row cannot sell an unlicensed module. `/me` module list is per-caller.	`97670b0`
Control API	`/api/v1/control/tenants` (list/create/get/status/modules/plan/term/users), `/plans`, operator bearer token, constant-time compare, unset ⇒ routes not registered.	`ebbb7ae`
Plans	`control.plans` — named module bundles applied to one tenant or in bulk. Apply COPIES onto `tenant_modules` (still the entitlement of record), so a plan edit changes nothing until pushed; drift is reported, not reconciled.	`772be84`
Commercial terms	One end date per tenant + trial flag + opt-in `auto_suspend`, enforced hourly per tenant. Reactivating a still-expired tenant disarms the automation rather than letting it undo the operator. Needs the worker role to enforce.	`04d2af1`
Signup queue	`control.signup_requests` + a PUBLIC `POST /api/v1/signup` (off by default; per-IP limit; one pending request per email). It provisions NOTHING and never reveals whether a subdomain is taken. An operator approves, naming the tenant id/name/plan themselves.	`da9cb27`
KYB gate	`control.tenant_kyb`. Granting a regulated module (`esign` → e-Sign/e-Meterai/e-Stamp) is refused without a current approval, on all four entitlement writers; fails closed; revokes nothing (grandfathered holders are a listed backlog).	`e037f4b`
Per-tenant bootstrap	Scheduled tasks + the tenant's OWN in-house CA issued at provision time, before activation. Idempotent, and re-run for every active tenant on worker start so half-provisioned tenants heal.	`ac7d3e8`
Tenant-aware workers	Audit chainer, outbox relay and scheduler fan out across active tenants; one tenant's failure cannot starve the others; suspended tenants are skipped.	`ac7d3e8`
Background entitlements	Background work uses the same licence∩tenant gate as the request path; `kernel.Detach` keeps the tenant on fire-and-forget goroutines.	`0e59153`
Blob isolation	Object keys namespaced `<tenant>/<hash>`.	`d275f84`
Tenant-relative links	Every user-facing absolute URL (signing invitations, verification emails, shortlinks, OnlyOffice `api_js`, OAuth issuer) points at `<tenant>.<base-domain>`, derived from the bound tenant — never the request Host.	`da6a34e`
Per-tenant passkeys	Each tenant gets its own WebAuthn relying party (RP id = its host), so a credential is usable only where it was registered.	`d2f002c`
Tenant administrator	`-provision-tenant-admin=<email>` / `admin_email` creates the tenant's first admin with a one-time password + wildcard role. Sign-in verified end to end.	`28df93c`
Clean Cloud boot	RBAC catalogue, correspondence/classification seeds, ACL backfills and esign key re-encryption run per tenant; preview-cache sweeper fans out. Zero ERROR lines, zero `no tenant bound`.	`28df93c`
Per-tenant settings	AI provider (its own API key, base URL, model, ask limit) and rate-limit budgets resolve from the calling tenant's own row. Fail safe to the deployment default, never to another tenant's.	`df66939`
Per-tenant audit keys	Each tenant's hash chain is keyed with HKDF(`AUDIT_CHAIN_KEY`, purpose, tenant) at key epoch 2, so one tenant's key cannot forge another's history. Rows carry their epoch, so pre-existing epoch-1 rows keep verifying under the deployment key — and Enterprise derives nothing, staying byte-identical.	`e5adca5`
Operator console	`obscura-operator`: own binary, own accounts, own subdomain. Owners (TOTP mandatory) manage tenants/entitlements/admins; read-only viewers let the console itself be demoed publicly. Holds `CONTROL_PLANE_TOKEN` server-side and proxies the control API; every mutation attributed in append-only `control.operator_audit`.	`f1ac123`
Offboarding (purge)	`DELETE /control/tenants/{id}` destroys objects, schema and registry rows. Gated twice — only from `closed`, and the body must name the tenant. Objects go BEFORE the schema so nothing is orphaned in the bucket; the object count is reported.	`3c5354a`
Usage reporting	`control.tenant_stats`: daily per-tenant snapshots (users/documents/letters/storage/workflows/audit) via the scheduler's existing fan-out. Deliberately the future metering source.	`91da964`
Seeded demo tenants	`-provision-tenant-admin-password` supplies a fixed credential that survives a scheduled rebuild (skips must-change only in that case).	`d6cbac2`

Real bugs found, and one class that took production to find

Cross-tenant data loss (d275f84). Blobs are content-addressed, so two tenants with
byte-identical files shared one object — while the refcount query authorising deletion only
sees the deleting tenant's schema. Tenant A destroying its copy destroyed tenant B's
document. Fixed by namespacing keys, which makes the per-schema refcount correct by
construction.
Fire-and-forget work silently dead (0e59153). The embed-on-upload and enrich-on-upload
hooks spawned goroutines with context.Background(), dropping the tenant, so in Cloud they
were refused by the pool and did nothing at all. Fixed by kernel.Detach.
The same bug twice more, found only in production (515ecb4). Document text extraction
and letter text extraction had the identical shape, so in Cloud no document and no letter
in any tenant was ever full-text extracted — search, content previews and the MCP
read/search tools returned empty while every upload reported success. 0e59153 missed
extract for an instructive reason: that hook took no context at all, so unlike its
siblings there was nothing to convert and nothing looked wrong. The absent parameter was the
camouflage. Now guarded structurally by internal/kernel/detach_guard_test.go, which parses
every non-test file under internal/ and fails on context.Background() inside a go func
— three instances across two manual sweeps is a review process that does not work.

NOT BUILT — do not plan against these

Billing — FOUNDATION BUILT on feat/billing, not merged. The ledger (control.plan_prices,
subscriptions, invoices, payments, migration 00012), a provider-agnostic gateway port with a
Midtrans adapter, and the webhook are done and verified end to end. Still missing: the console UI,
tenant-facing invoices, the renewal cron, dunning, proration and PPN. See
BILLING.md. Nothing on main charges anyone.
Self-serve signup — HALF DONE. The REQUEST queue is built (control.signup_requests, public
endpoint off by default, operator approves and provisions). Nobody self-registers: no request
becomes a tenant without a human, deliberately, until billing exists to bound it. The public FORM
belongs on the marketing site and is not in this repo.
✅ KYB gate — BUILT (control.tenant_kyb, e037f4b). Business verification per tenant, gating
every path that grants a regulated module. Per-signer KYC / PSrE certificate enrollment for
certified e-signature is still design only — that is a per-INDIVIDUAL gate, separate from this
per-BUSINESS one.
Tenant self-service. A tenant's own admin screens exist (they are the product), but a
customer cannot see or change their own subscription, and cannot submit their own KYB evidence (an
operator records it; the row is shaped so a tenant-facing form can write the same one later). The
OPERATOR console IS built — above.
Support impersonation — CLOSED as a DECISION, not a gap (2026-07-30). An operator holding a
session inside a customer's tenant is a deliberate bypass of the boundary the architecture exists to
enforce, and turns a console compromise into a document-archive compromise. See
OPERATOR_ROADMAP.md §1 Tier 3 for the reasoning and for what to use instead.
Tenant EXPORT. Purge is built (above); handing a departing customer their data as a
package is not. Take a database dump and a bucket copy before purging anything wanted.
Per-tenant secrets — HALF DONE, and the remaining half needs a decision.
The audit chain is now keyed per tenant (HKDF from AUDIT_CHAIN_KEY, epoch 2; see the
table above). STEGO_MASTER_KEY and the blob DEK are still per-DEPLOYMENT. The CA always was
per tenant.

Two honest caveats about what the audit change does and does not buy:

It contains the blast radius of a DERIVED key leak, not of a master leak. Anyone with
AUDIT_CHAIN_KEY can still forge any tenant's chain, because that is what the keys are
derived from. The win is that one tenant's key no longer compromises the others.
It does NOT yet let a tenant verify its own chain, and doing so has a real cost. The
chain is an HMAC, so verification ability is forgery ability — handing tenant A its key
lets A rewrite A's own audit history undetectably. Per-tenant derivation makes that
disclosure possible per tenant (it no longer endangers others), but whether to offer it
is a policy decision rather than a missing feature. Deliberately not shipped.

Per-tenant STEGO keys are blocked on the sidecar contract, not on the Go side. The master
key travels to the stego sidecar as a single x-stego-master-key metadata value, and decode
varies only the epoch across a candidate window. A mark embedded under the shared master
therefore cannot be decoded with a derived key, so switching would silently make existing
marks unattributable — decode simply finds nothing, which is indistinguishable from "no mark".
Doing it properly needs either two decode round-trips (on the OOM-prone sidecar) or a proto
change accepting candidate KEYS as well as epochs, i.e. a coordinated sidecar deploy.
Scoped, not started.
Noisy-neighbour quotas — PARTLY BUILT. Storage and seat ceilings exist (83a45f1, migration
00011): opt-in per tenant, enforced at the blob store and the provisioning paths, refusing new
bytes/accounts only. The auth rate-limit bucket is now per tenant (263fa57). Still shared with
no ceiling: CPU, sidecar work, and database connections.
Shared sidecars. gotenberg, embed, extract, stego, OnlyOffice are one pool for all
tenants — no per-tenant quota, and the stego sidecar's known OOM is an all-tenant event.
The /metrics AI-token series is absent in Cloud. AITokensCollector.Collect reads the
per-tenant ai_usage table from a deployment-wide endpoint, so it binds no tenant, the query
is refused, and the collector drops the series silently. Correct behaviour means per-tenant
labelled series (or a control-plane rollup), which is a design decision rather than a fix. The
rest of /metrics is unaffected — only this one collector reads tenant data.
End-to-end browser test. The SPA layer is sound by inspection (see below) but no browser
has driven a tenant subdomain.

Deployment readiness

Blocking for any real customer traffic:

Wildcard DNS + wildcard TLS for *.<TENANT_BASE_DOMAIN>. Nothing works without it.
public must contain no application tables. Enforced at boot by
AssertPublicIsClean — a Cloud database that ever had the single-tenant migration set
applied will refuse to start, on purpose.
Web UI: layer verified, browser run outstanding. The SPA calls the API with relative
paths so it follows its host; web/nginx.conf is a server_name _ catch-all forwarding
Host $host; and the session cookie sets no Domain, so it is host-only and cannot cross
tenant subdomains. What this investigation did find was a backend bug — every absolute
link was apex-anchored and would have 404'd — now fixed (da6a34e). A real browser pass is
still worth doing.
No billing — every module is free until someone builds it. Plans and end dates are recorded
and enforced operationally; no invoice is produced and no payment is taken.

Strongly recommended before traffic: per-tenant secrets (above), noisy-neighbour quotas, and
an INDEPENDENT security review of the tenant boundary. An adversarial audit was run on 2026-07-30
(audits/2026-07-30-tenant-boundary-audit.md): the
isolation itself held — nothing got one tenant's data out of another — but three shipped surfaces
handed out addresses no tenant answers, so OnlyOffice editing, internal signing and the external
approval link were all broken in Cloud (fixed, f8e6d30). That audit was run by the same author
as the code; it found real defects, which is evidence the method works, not that the author is
impartial.

See CLOUD_DEPLOYMENT.md for the actual runbook.