Tenant-boundary audit — Obscura Cloud

2026-07-30 · against 927b508 · adversarial review of the isolation seam, requested because
CLOUD_STATUS.md said the boundary "has not been adversarially reviewed by anyone but its author"
and three cross-tenant defects had surfaced in the previous day's work.

Method: a real two-tenant Cloud stack (acme, globex) on a real Postgres and object store, with
every claim checked by making the request rather than by reading the code. Findings are marked
PROVEN only where a command shows the behaviour.

What held

The isolation mechanism itself is sound, and I could not get one tenant's data out of another.

• Fail-closed executor. A request with no tenant bound gets an executor that errors on every
  call. There is no fallback to a default pool — the one shape that would silently cross tenants.
• Blob keys are namespaced inside the store adapter (objectKey), not at the call sites, so no
  caller can forget. PurgeTenant matches on "<tenant>/" with the separator, so acme cannot
  match acme2.
• The control-plane escape (kernel.WithControlPlane) is used in exactly eight places, all of
  them registry, provisioning, signup or worker fan-out. No request handler grants itself one.
• Resolver exemptions are two exact-match paths (/api/v1/version, /api/v1/signup) plus the
  control-plane prefix, which is separately gated by the operator token. No prefix is over-broad.
• Background work: the only context.Background() left in a data path is the AI-token metrics
  collector, already documented as a known gap. detach_guard_test keeps the rest closed.
• Rate-limit and cache keys were checked one by one (see Looked at, not a finding).

What did not hold — one root cause, three live consequences

Cloud routes every request to a tenant by its Host. Three shipped surfaces hand out an address
that no tenant answers. The isolation is intact in the inbound direction; the failure is in the
return direction, and it always fails late — in an email nobody on the team reads, in a document
server's fetch, in the timestamp leg of a signature.

All three are fixed in f8e6d30.

1. OnlyOffice editing could not work in Cloud at all — PROVEN

The editor config hands the document server ONLYOFFICE_OBSCURA_URL, whose default is
http://obscura:8080 — the compose service name. One environment variable cannot name every
tenant, so the doc-server's fetch arrives with a Host that resolves nothing:

GET /api/v1/office/content/<jwt>   Host: obscura:8080    → 404 tenant.unresolved
GET /api/v1/office/content/<jwt>   Host: acme.ux.local   → 200, 1181 bytes

Every content fetch and every save callback 404'd. office is in the seeded Professional and
Enterprise plans, so an operator sells it in the normal course of business.

Underneath it, a boundary hole the outage was hiding: the token's claims were
{purpose, sub, doc, version, uid, exp} — no tenant — signed with one deployment-wide HMAC
secret. Tokens were therefore interchangeable across tenants, leaving UUID unguessability as the
only thing between a replayed token and another customer's document.

Fix. The tenant rides in the token (it is the only thing that travels the whole round trip) and
resolveTenant binds from it for exactly those two paths, after verifying the signature and after
running the same servable/suspended registry gate a host-resolved request gets. An ordinary path on
the internal host still 404s.

2. The external approver's emailed link 404s — PROVEN

POST /external-approval/gates built the link from APP_BASE_URL, which in Cloud is the apex:

link emailed:  http://localhost:8099/approve/<token>   → 404 tenant.unresolved
same token at: http://acme.ux.local/approve/<token>    → 200

da6a34e fixed signing invitations, verification emails, shortlinks, api_js and the OAuth
issuer; this one was missed. It fails in the worst available place — an outside party holding a
link, with nobody on the team watching. The ISO 16175 transfer package's base_url had the same
bug, where it is a records-provenance field: it named the platform rather than the organisation the
records came from.

3. Internal signing fails outright in Cloud — PROVEN (mechanism), fix unit-tested

deploy/docker-compose.yml defaults ESIGN_TSA_URL to http://127.0.0.1:8080/api/v1/tsa — our own
responder, over loopback. docs/INTERNAL_SIGNING.md documents that as the standard setup.

POST /api/v1/tsa   Host: 127.0.0.1:8080   → 404 tenant.unresolved
POST /api/v1/tsa   Host: acme.ux.local    → 403 esign module (i.e. routed to the tenant)

This is not a degraded B-B signature. GetTSA treats any non-2xx as an error and the signature
aborts, so every internal PAdES signature fails on a Cloud tenant holding esign.

/api/v1/tsa must NOT simply be exempted: the TSA certificate is issued from the tenant's own
CA, so the timestamp has to be signed by the right tenant's key.

Fix. Only the Host header moves — the signer keeps dialing the configured (loopback) URL, so it
stays out of the TLS edge and keeps the loopback exemption from the public TSA rate limiter, which
is decided on the real TCP peer. Vendored patch #3 in third_party/pdfsign.

4. Mekari's webhook cannot be routed in Cloud — not fixed, documented

MEKARI_CALLBACK_URL is one deployment-wide URL for a per-tenant callback, so it can name at most
one tenant. Sign-IT/Peruri — the provider actually deployed — has no webhook (completion is
poll-driven by the per-tenant sweeper), so nothing live is affected. A Cloud deployment must not
configure Mekari until the callback carries a tenant.

The class, and why a test replaced the habit

Three separate sweeps have now missed instances of the same rule. origin_guard_test.go parses the
package and fails on any cfg.AppBaseURL read outside a named allowlist (each entry carrying its
reason), and on any office token minted without a tenant claim. Verified by reverting fix 2 and
watching it fail. Same posture as detach_guard_test: "three instances across two manual sweeps is
a review process that does not work."

Looked at, not a finding

Recorded so the next audit does not re-derive them.

• psreStatusCache is keyed by email with no tenant. The PSrE provider is deployment-wide and
  keys certificates by email globally, so the cache reflects the underlying truth. It becomes a
  crossing the day esign credentials go per-tenant.
• letterPreviewCache / letterheadPreviewCache are keyed by content hash (plus, for letters,
  a digest of the resolved merge map). A cross-tenant hit requires byte-identical inputs, which
  produce byte-identical output, and the cache is only consulted after the caller's own letter has
  passed its ACL check.
• analysisCache holds extracted document text keyed by a 128-bit random id, with an owner
  check on read. No tenant in the key; the id is not guessable and the owner check is a second gate.
• officeLiveKeys is keyed by subject UUID and holds a session key, not data.
• trustedBaseHost (AI SSRF allowlist) is process-wide but written once at composition from the
  deployment env, never from a tenant admin surface. A tenant's own AI base URL is checked against
  it, which is the restrictive direction.
• ip|<addr> rate-limit buckets were shared across tenants. Not a data crossing, but shared
  fate: one tenant's failed logins consumed the auth budget for every tenant behind the same source
  address — and the app has already been seen recording the Docker bridge IP for everyone, which
  collapses the whole deployment into one bucket. Fixed in 263fa57; the key is now
  ip|<tenant>|<addr>, and Enterprise keeps the plain key.
• Public token surfaces (/sign/, /s/, /approve/, shortlinks) are reached by a browser at a
  tenant host and looked up in that tenant's schema, so a token from another tenant is simply not
  found.

Still open after this audit

• Noisy-neighbour quotas — partly closed. Storage and seat ceilings shipped in 83a45f1
  (migration 00011) and the auth bucket is per tenant. CPU, sidecar work and database connections
  are still one shared pool with no per-tenant ceiling.
• Per-tenant secrets — STEGO_MASTER_KEY and the blob DEK remain deployment-wide.
• An independent reviewer. This audit was run by the same author as the code. It found real
  defects, which is evidence the method works, not evidence that the author is now impartial.