Horizontal scaling audit — capability & necessity

Date: 2026-08-25 · Branch: feat/cloud-enterprise-tenants @ d311c6f9
Method: 8 parallel code/infra audits over 207k LOC Go + 22-service compose, plus read-only recon on all three live deployments (prod dms1, Cloud VM2 dms2, demo valbox). Every load-bearing claim re-verified directly against source.

Verdict

ARCHITECTURE.md:38 claims "Horizontal scale = N stateless api + M worker replicas against one Postgres." That claim is aspirational. It is not true today, and the code says so in about six places — ratelimit.go:19-22, preview_session.go:1-7, handlers_office_builder.go:78-81, handlers_office_apply.go:42-46, analysis_cache.go:12-14, handlers_mcp_office.go:57-59 all explicitly document a single-process assumption.

Part 1 — NECESSITY: the measured case against

All numbers from live read-only recon on 2026-08-25.

Across all three deployments combined: 1,051 documents, 66 users, 3 tenants, 165 MB of database.

Supporting evidence:

• Zero saturation events, ever. 40 containers across 3 hosts: RestartCount=0, OOMKilled=false everywhere. dmesg | grep -ic "out of memory" → 0.
• Peak observed burst was 283 req/min (4.7 req/s) and the app absorbed it below 4% of a single core.
• Real daily active users on prod: 1–5. From audit_events count(distinct actor) over 14 days — peaked at 5, was 1 on 23 and 25 Aug.
• ~40% of prod "traffic" is agent traffic, not humans. Top paths: POST /api/v1/mcp 405, GET /api/v1/mcp 109 out of 1,278 requests. User-agents: claude-code/2.1.x = 483 hits. Mobile app: 45 hits total.
• The valbox load average of 26.69 is not Obscura — it is obs-test-pg (552% CPU, unrelated timescaledb) and the x056 gateway. Obscura's own container there is at 0.49%. Same attribution trap as the disk-usage figure.
• The product's own pricing model tops out below the problem. docs/PRICING.md:93: "Cloud masuk akal sampai ±500 seat" — above that the stated correct answer is on-prem perpetual, i.e. a customer's own box. docs/CLIENT_DEPLOYMENT.md:16 sizes a deployment at "~35 users → 8 vCPU / 32 GB".
• No SLA or uptime target exists anywhere in the repo. Grepped every .md for SLA|uptime|99\.[59]|RTO|RPO. Nothing has been committed to.
• No load-test harness is committed. Zero hits for k6, vegeta, locust, wrk, ab, hey, artillery, autocannon. Zero Go Benchmark*. .e2e/battle.cjs is a Promise.all of two requests.

Git history: has it ever fallen over from load?

No. 2,046 commits; 32 matching perf|slow|timeout|oom|memory|scale|concurren|leak. Three real resource-exhaustion events, none of them many-users-at-once:

1. 9db62da1 (08-14) — stego sidecar OOM. os.cpu_count() returned the host's 12 cores inside a cpus: 3.0 container, so one 2.8 MB PDF forked 11 renderers into a 4 GiB cap. One document, one request.
2. 2ec6afdf (08-04) — embed sidecar wedged five days while /readyz said "ok". A batch-size misconfiguration (16 passages needing 80–127s against a 60s timeout), not load.
3. ded752d0 (07-16) — a hardcoded 60s timeout aborted a 124s OCR; the hourly backfill then retried in a storm. Self-inflicted; the site kept serving throughout.

The one genuine capacity cliff (docs/CONCURRENCY-AUDIT.md, fixed in 9c622802) was found by the author generating synthetic load, before any user hit it.

Would app replicas even help?

No — because the app is a thin orchestrator. Every operation costing more than ~10 ms of wall-clock is in a sidecar or on someone else's network:

Adding app replicas moves zero CPU-seconds of the real work. Worse, it actively harms two paths: OFFICE_CONVERT_CONCURRENCY (default 1, config.go:349) and mcpBuilderSlots (2) are per-process semaphores, so replica #2 doubles the arrival rate onto the exact queue the semaphore exists to protect.

Part 2 — CAPABILITY: 15 confirmed blockers

Ranked by blast radius. Every one verified against source.

🔴 Money / data / security

1. The outbox relay has no claim. go/internal/platform/events/events.go:80-113

`SELECT seq, ... FROM outbox_events WHERE published_at IS NULL ORDER BY seq LIMIT $1`
// ...no transaction, no FOR UPDATE, no SKIP LOCKED, no claim column
if err := r.handler(ctx, e); err != nil { ... }        // ← side effects fire HERE
`UPDATE outbox_events SET published_at = now() WHERE seq = $1`

Runs on a 2-second ticker (wire.go:2555). grep -rn "SKIP LOCKED" go/ → zero hits repo-wide. notify.Notify mints a fresh kernel.NewID() per call with no idempotency key.
N replicas ⇒ every approval email, escalation, signing invite, workflow outcome, WhatsApp message, push notification and gatekeeper triage row fires N times. Not a rare race — the overlap window is the entire handler duration, including SMTP round-trips. This is the single worst finding.

2. Peruri serial claim is a process-local mutex. go/internal/esign/adapters/sealer_peruri.go:64-66

// snClaim serializes "look for an outstanding serial, then claim or buy it" so two
// concurrent affixes cannot both adopt the same paid-but-unstamped serial.
snClaim  sync.Mutex

Two replicas ⇒ both adopt the same paid serial, or both buy a new one at Rp 20.000. Compounded by the known-dead SN reclaim.

3. Every security budget divides by N. All in-memory maps:

ratelimit.go:19-22 already says it: "in-memory suits the single-node on-prem target. A multi-replica deployment would need a shared store."

4. Workflow instances have three blind writers. workflow/adapters/pg.go:453-460 is UPDATE workflow_instances SET state=$2 WHERE id=$1 — no status predicate, RowsAffected discarded. GetInstanceForUpdate (a real FOR UPDATE) exists and is used correctly in 9 places, but FinalizeInstance (service.go:2942), CancelInstance (:3234) and ReturnFromSubWorkflow (:3588) use plain GetInstance.
Replica A approves; replica B's cancel — which read submitted, passed its Go-side guard, and blocked on A's row lock — then stamps cancelled over approved. Reverse the order and a cancelled instance becomes approved with its signing links relit, which is precisely the invariant 47c44a50 was fixed to guarantee and verified live on prod on 08-24. Fix is three characters each.

5. Migrations race with no lock. MIGRATE_ON_BOOT defaults true (config.go:114) and compose hardcodes "true" (docker-compose.yml:470). go/migrations/embed.go:34 uses the legacy global goose.UpContext — an API that cannot lock at all — and SessionLocker appears nowhere in the tree. N replicas booting after a release all read version N and all attempt N+1 → duplicate-key on goose_db_version, non-zero exit, crash-loop on every migration-carrying deploy. In Cloud this multiplies by tenant count (MigrateAllTenants, wire.go:2214).

6. MCP office edit locks are process-local. handlers_mcp_office.go:68-73. Its own comment names the failure: "the second would land a version that silently discards the first one's work — a lost update that no dedupe can detect." There is no DB row lock behind it.

7. Licence hot-swap is per-process and per-node. licensing.go:73 (s.lic.Store) + handlers_license.go:52 (atomicWriteFile to a node-local bind mount). An admin uploads a renewal → one replica swaps. The others serve 403 module.not_licensed indefinitely, and across restarts because their bind-mounted file was never rewritten. The response reports persisted: true.

🟠 Correctness / availability

8. Three capability-token stores live in process memory and are fetched back over the network.

🔴 Stickiness cannot fix #2 and #3. The OnlyOffice doc-server is a separate, cookie-less HTTP client, and hashing the token doesn't work because the token is minted server-side (its hash is uncorrelated with the minting replica). In Enterprise the doc-server doesn't even traverse nginx — it dials obscura:8080 directly and Docker DNS round-robins. These require a shared store, not affinity.

9. Sign-IT OTP tokens in memory. sealer_signit.go:64-65. GET OTP on A, submit on B ⇒ empty token posted to Peruri ⇒ the external signer enters the correct emailed code and signing fails. No retry path recovers it.

10. Admin settings bound at boot, per process. Auth settings (auth/app/service.go:79, sole reload caller is boot), SMTP relay (notify/adapters/smtp.go:64), AI provider + API key (ai/app/provider.go:154-181, binds on first sight, no TTL), Enterprise rate limits (handlers_ratelimit.go:67, no TTL).
Concrete: an admin turns self-registration off — replica B keeps letting strangers register forever. A tenant rotates a leaked AI key — replicas B..N keep sending document text to the old vendor on the revoked key.

11. Two read-write volumes need RWX shared storage. office-fonts (compose:673, written by the app, read by OnlyOffice and Gotenberg — a font present on one node makes the same document render with different line breaks) and peruri_sharefolder (compose:680, a node-local named volume the single signadapter reads; a miss costs Rp 20.000 per the dead SN reclaim).

12. ResumeFolderEncryptions runs unguarded on every boot (wire.go:2007, :2184 — outside the RunsWorker gate). N replicas burn maxEncryptPasses on each other, then failEncryption clears the pending marker and leaves the folder plain.

🟡 Mechanics — it won't even start

13. The published port pins it to one. docker-compose.yml:685 ports: ["127.0.0.1:38080:8080"]. --scale obscura=2 fails on port allocation. update.sh:72 health-gates on that same fixed port.

14. The api/worker split is dead code. config.go:1522 RunsAPI() has zero call sites — verified. The HTTP server starts unconditionally, so OBSCURA_ROLE=worker is byte-identical to all. And docker-compose.yml:355 hardcodes OBSCURA_ROLE: all as a literal, not ${...} — the split is unreachable from any composed deployment. Nobody has ever run it.

15. The 15s drain is cut to 10s. wire.go:2780 drains for 15s; no service declares stop_grace_period anywhere in deploy/, so Docker SIGKILLs at 10s. There is no readiness/liveness distinction either: /healthz is a constant 200 (wire.go:2738), and the listener does not open until after migrations, so a warming replica is connection-refused rather than "not ready" — an LB cannot tell it from a crash.

Part 3 — the sidecar ceiling: what saturates first

Even with all 15 fixed, app replicas hit a downstream wall almost immediately. Ranked:

Nothing has deploy:/replicas: anywhere. mem_limit/cpus exist on exactly 5 of 22 services — obscura, postgres, minio, onlyoffice and web are all unbounded. Sidecar quota already sums to 15 CPUs on an 8-vCPU prod box; adding app replicas onto the same host makes that strictly worse.

Part 4 — what is actually broken today, on one replica

These came out of the audit and do not depend on scaling at all.

1. 🔴 GuardStorage ignores the bytes being written. go/internal/tenancy/app/quota.go:169

func (q *Quotas) GuardStorage(ctx context.Context, incoming int64) error {
    u, err := q.Usage(ctx)
    if err != nil { return nil }
    if !u.OverStorage() { return nil }     // ← `incoming` is NEVER read

Verified: grep -n incoming quota.go returns line 169 only — the parameter appears in the signature and nowhere else. The call site passes the real size (blob/minio.go:181), and the migration bounds the intended overshoot at one file. Actual bound: one file of unlimited size. A tenant at 4.99 GiB of a 5 GiB quota can upload 500 GB. No test varies incoming. The per-user guard already does it right at resolvers.go:337. One-line fix.
(The fail-open on a measurement error above it is deliberate and correctly documented — that part is fine.)

2. 🔴 Deploys sever in-flight signatures. wire.go:2780 drains 15s; Docker kills at 10s because no stop_grace_period is declared. For a records/signature system, a routine business-hours deploy cuts an in-flight 30–60s certified Peruri signature, a 124s OCR, a 24s stego seal. That is materially worse than the 502 window itself. Ten-minute fix.

3. 🔴 Postgres is completely untuned. pgvector/pgvector:pg17 with no command:, no postgresql.conf, no tuning anywhere in the tree. Verified live: shared_buffers=128MB, work_mem=4MB, max_connections=100 — on a 32 GB host, backing relational data + pgvector HNSW + tsvector FTS + River + the outbox. Typically 2–10× on this workload for a 20-line change.

4. 🔴 The Cloud connection budget already exceeds the Postgres default. TENANT_POOL_CACHE_SIZE 64 × TENANT_POOL_MAX_CONNS 4 = 256 potential against max_connections=100. docs/CLOUD_DEPLOYMENT.md:37 says to size this before onboarding tenants; compose ships stock Postgres. This is Cloud's first hard wall — roughly 25 concurrently-active tenants — and the failure mode is a hang, not a 503 (no acquire timeout, no statement_timeout, no WriteTimeout). It is also invisible: the metrics collector reports only the base pool; DB.TenantPoolCount() has no non-test caller.

Also worth knowing: prod's running containers predate their own config. docker inspect shows all deploy-* containers at NanoCpus=0 Memory=0, and deploy-extract-sidecar-1's PID 1 is the pre-fix single-process uvicorn app:app. The compose file has been right for five weeks; these containers have not been recreated. An env file states an intention — only docker inspect states a fact.

Recommendation

Do not build horizontal scaling. Capacity is not the constraint and will not be within the product's own pricing horizon; the work is ~3–5 weeks of de-statefulling and would then hit an OnlyOffice licence wall and a single Postgres anyway.

Do this instead, in order:

1. Fix GuardStorage — u.StorageBytes+incoming >= *u.StorageQuotaBytes. One line, broken now, unbounded blast radius. (15 min)
2. stop_grace_period: 20s on obscura. Stops deploys killing in-flight signatures. (10 min)
3. Tune Postgres + raise max_connections above 264. Biggest single throughput win available, and it removes Cloud's first hard wall. (2–3 hours)
4. Give the outbox relay a claim (FOR UPDATE SKIP LOCKED, or copy the advisory-lock pattern already working at audit/adapters/pg.go:250). Required before any second process, including the role split. (half a day)
5. Wire RunsAPI(), interpolate OBSCURA_ROLE, run 1 API + 1 worker. Multi-process, not multi-replica — buys the worker isolation and blast-radius containment that people actually want from "scaling", and the scheduler and audit chainer are already replica-safe. (~2 hours after step 4)
6. Add admission control — a bounded, per-tenant semaphore in front of extract/stego/embed. docs/CONCURRENCY-AUDIT.md:171: "There is no semaphore, worker pool or errgroup.SetLimit in the Go tree." This is what fixes noisy-neighbour, not replicas. (1–2 days)
7. Commit a load harness. The one real capacity cliff was found by ad-hoc load that was then thrown away.
8. Then buy a bigger VM if needed. 16 vCPU/64 GB ≈ Rp 120jt/yr against deal sizes of Rp 350jt–2 M — 1–5% of one deal. One box reaches ~10–20× today's prod, comfortably past "±500 seats, then sell them on-prem anyway."

The one thing a bigger box does not buy is availability — but neither do app replicas, while Postgres and MinIO are singletons on one VM. If HA is the actual goal, that is a different project (Postgres replication + MinIO erasure coding), and it should be scoped as one.

What the codebase already got right

Worth not breaking: sessions and refresh tokens are opaque DB rows with atomic reuse detection; WebAuthn ceremonies, step-up grants, OIDC state and share OTPs are all DB-backed; the audit hash chain takes a per-partition pg_advisory_xact_lock and cannot fork; the scheduler's ClaimDue is a correct atomic claim and says so in a comment; letter numbering is gapless under an advisory lock held across read and insert; billing runs on River with SKIP LOCKED + UniqueOpts; the preview page cache is content-addressed in Postgres + blobs with ON CONFLICT DO NOTHING; and the two things that normally make a licensed product un-scalable — node-lock and the anti-rollback clock — are both derived from the shared Postgres cluster and are already correct.

The auth, audit and billing layers are replica-ready. It is the ephemeral capability tokens, the outbox, the in-memory security counters and the connection budget that are not.