Concurrency audit — 2026-08-19

Every number below was measured on the x056 demo (3-CPU sidecars), not inferred from code.

Summary

🔴 1. The extract sidecar serves one request at a time, globally

This is the big one, because of what runs through it. render.rasterize delegates to this
sidecar, so it is not just "text extraction" — it is every PDF page render in the product:
secure preview, the page filmstrip, egress watermark compositing, DOCX→editable, and the text
extraction on every upload.

Cause. All five work endpoints are async def and call blocking CPU-bound code inline:

@app.post("/extract")
async def extract(file: UploadFile = File(...), ...):
    data = await file.read()
    text, truncated = dispatch(data, mime, ...)   # ← blocking: pdfium, OCR, docx parsing

In FastAPI, a def handler runs in a threadpool; an async def handler runs on the event
loop. This is exactly backwards for CPU-bound work. There is no run_in_threadpool, no
asyncio.to_thread, and uvicorn is started with no --workers, so there is exactly one
process with one event loop.

embed-sidecar next door uses the correct pattern (def embed(...)), so the fix is a
known-good shape already in the tree.

Measured — same 2.7 MB PDF, /extract:

single                    2.95s
6 concurrent, wall-clock  18.08s      ← serialized prediction was 17.71s
                          (parallel would have been ~2.95s)

Latencies clustered 9.0/9.0/9.0/18.0/18.0/18.0 — the event loop interleaves at await points,
but total throughput is one CPU's worth. The container is granted cpus: 3.0 and cannot use
more than one of them.

Second-order effect, and the one that will page someone at 2am:

/healthz idle        0.003s
/healthz under load  6.434s

The compose healthcheck is timeout: 5s. Under sustained load the sidecar fails its own
healthcheck while working perfectly — because liveness is queued behind the same blocked
event loop. Anything gated on condition: service_healthy then fails, and monitoring reports
an outage that is not one.

🔴 The obvious fix is wrong, and it crashes the service

My first recommendation here was to make the five handlers sync (def), letting FastAPI run
them in its thread pool. That reads like the correct FastAPI idiom. It segfaults.

Built and load-tested it: the same image, the same six-request load, one variable changed.

EXTRACT_MAX_CONCURRENCY=1   survives, exit 0
EXTRACT_MAX_CONCURRENCY=3   SIGSEGV (exit 139), all 6 requests dropped, container dead

pdfium is not thread-safe. Every heavy route here goes through pypdfium2, so a thread pool
is unavailable at any size above one. The async def that caused the serialization was also
the only thing preventing the crash — accidentally load-bearing.

This is worth stating plainly because the change is a four-line diff that passes review, passes
a smoke test, and takes the sidecar down under concurrent load in production.

The fix that works: worker PROCESSES. Separate processes share no pdfium state, so N
workers is N genuinely concurrent requests with no locking. serve.py derives N from the
container's own cgroup CPU and memory limits at startup and hands it to uvicorn; the handlers
stay async def, which is now a documented correctness constraint rather than an accident.

Measured after the change, same machine, same load:

The page-render path — the hot one — is ~1.8× faster, and /healthz is 42× better and no
longer anywhere near the 5s healthcheck timeout.

/extract gains less. One request already uses 101% CPU, so six of them want 6 CPUs and the
container is capped at 3 (measured at 296% during the burst — it is saturating its quota).
That remaining ceiling is a deliberate setting, not an architectural block: raise cpus: and
the worker count follows automatically. The serialization is gone; the budget is what is left.

🔴 2. Office → PDF is one at a time, and Gotenberg has no knob for it

Gotenberg exposes --chromium-max-concurrency (default 6) but there is no LibreOffice
equivalent — only --libreoffice-restart-after and --libreoffice-max-queue-size. A single
LibreOffice instance converts, serially, by Gotenberg's design.

Measured — same DOCX through /forms/libreoffice/convert:

single                    0.49s
4 concurrent, wall-clock  2.94s       ← worse than 4× serial (1.97s)
per-request               0.42 / 1.99 / 2.47 / 2.92s

Concurrency here is not merely absent, it is negative: queueing and the restart-after-10
cycle cost more than running them back to back.

This affects official-copy generation, PDF/A export, and every docx/xlsx preview.

Fix options, in increasing order of effort:

1. Serialize deliberately in Go — a semaphore of 1 in front of the LibreOffice route, so
   requests queue in our process with a fair, observable wait instead of piling into Gotenberg.
   Cheapest, and makes the limit visible rather than emergent. ~1 hour.
2. Run N Gotenberg replicas behind the existing client, sized to cores. Real parallelism,
   costs ~700 MB RSS each. ~half a day.
3. Move conversion to a job queue with a worker pool, so a burst of uploads degrades to
   "queued" rather than "all slow". The honest long-term answer. Days.

⚠️ 3. Gotenberg is the one sidecar with no resource limit

gotenberg          NanoCpus=0            Mem=0          ← unbounded
extract-sidecar    NanoCpus=3000000000   Mem=4 GB
stego-sidecar      NanoCpus=3000000000   Mem=4 GB
embed-sidecar      NanoCpus=4000000000   Mem=3 GB

The compose file's own comment records why the others were bounded: "None of the sidecars had
ANY memory or CPU limit, so an OOM in one consumed host memory and could take Postgres with
it." Gotenberg was missed by that pass — and it is the one that runs both Chromium and
LibreOffice, the two hungriest processes in the stack, at a default concurrency of 6.

Fix: give it mem_limit and cpus like its neighbours. Ten minutes.

⚠️ 4. Stego is concurrent, but oversubscribed

No global lock, and the gRPC server takes max_workers=8 — so 8 marks can run at once. But
each request then forks its own raster pool of min(pages, cpus-1) = up to 2 processes, on a
container capped at cpus: 3.0.

Worst case is 8 × 2 = 16 processes competing for 3 CPUs, each holding page bitmaps against
a shared 4 GB cap. plan_workers sizes each pool against the memory limit correctly, but it
sizes them independently — it has no idea seven other requests are doing the same thing.

Not a correctness bug and not a serializer; it will show up as latency variance under load, and
as BrokenProcessPool (the OOM path the code already documents) before it shows up as slowness.

Fix: make the gRPC worker count and the per-request pool share one budget, or simply lower
max_workers to match the CPU quota. STEGO_MAX_WORKERS already exists as the override and is
currently unset.

⚠️ 5. Two ceilings worth knowing

DB, Cloud only. TENANT_POOL_MAX_CONNS defaults to 4. In schema mode each tenant gets
its own pool, so a single tenant's concurrent requests queue at 4 connections regardless of how
much Postgres could take. Fine at current load; the first thing to raise when a tenant grows.

No backpressure anywhere. There is no semaphore, worker pool or errgroup.SetLimit in the
Go tree. Nothing bounds how many requests pile onto a serialized sidecar — they all arrive,
all wait, and all time out together. Fixing #1 and #2 without adding a bound just moves the
cliff.

What was implemented

All four, in one branch.

1. Gotenberg bounded — cpus: 4.0, mem_limit: 6g, matching its neighbours.
2. Extract sidecar → 3 worker processes — serve.py + limits.py derive the count from the
   container's cgroup limits; handlers stay async def with a comment explaining that this is
   the pdfium thread-safety line and must not be "optimised" away. EXTRACT_MAX_CONCURRENCY
   overrides.
3. Stego pools share a budget — limits.raster_slot() counts live pools and plan_workers
   divides the CPU budget by that count. Verified: alone it still plans 2 workers exactly as
   before; with 3 pools in flight each plans 1, so the worst case falls from 16 processes on 3
   CPUs to a bounded share.
4. Office→PDF queued on our side — OFFICE_CONVERT_CONCURRENCY (default 1) bounds the
   LibreOffice route in render.GotenbergOffice.ToPDF, acquired after the request body is built
   and cancellable via context, so a reader who closes the tab releases their place.

The lesson worth keeping: the change that looked correct (sync handlers) was the one that
crashed, and the only thing that separated the two was building the image and putting load
through it. Neither review nor a single-request smoke test would have distinguished them.