OCR system:* flow is uploading empty files — 100% failure, chronic for 30 days

Found: 2026-08-26, while triaging Observatory security warnings.
Owner: the OCR engine session (ahu-ocr-akta-notaris) — hands-off from the platform side, so this is a diagnosis, not a patch.
Demo impact (28 Aug): none on the demo path. The interactive flow is green. Read "What is NOT broken" before acting.

The finding in one line

Every system:* actor in the OCR pipeline sends an empty or near-empty file to its downstream, so all three downstreams reject it with HTTP 400.

Evidence

1. Four actors, three different upstreams, 100% failure, all starting the same minute.

All began 03:14–03:19 UTC, ran to 10:25 UTC. Three independent upstreams failing identically for one caller means the caller is at fault, not any upstream.

2. The request bodies prove it. The audit stores a fingerprint of every upload (raw bytes are stubbed). Comparing the multipart envelope size:

• Failing: 297, 415, 520, 521, 521, 528 bytes
• Succeeding: 39,680 / 47,721 / 289,000 / 300,695 / 303,304 bytes

A multipart envelope carries ~200–300 bytes of boundary + headers. So the failing uploads contain a file of roughly 0–250 bytes — not a document.

3. Reproduced exactly. Posting an empty and a 200-byte file to the live service:

0-byte file   -> HTTP 400  {"detail":"Image decode failed: cannot identify image file ..."}  envelope 213 B
200-byte file -> HTTP 400  {"detail":"Image decode failed: cannot identify image file ..."}  envelope 415 B

4. The service itself is healthy. Real PNG, JPG and PDF all return 200 with correct OCR text ("rawText":"AKTA NOTARIS TEST 12345"), 24–117 ms.

5. Not new — chronic. Daily error rate for ahu-ocr system:* actors over 30 days:

Aug 01 100%   Aug 05  55%   Aug 12  39%   Aug 20  39%
Aug 02 100%   Aug 06  60%   Aug 13  48%   Aug 21  34%
Aug 03  10%   Aug 07  56%   Aug 14  64%   Aug 24  39%
Aug 04  48%   Aug 10  99%   Aug 18  69%   Aug 26  99.5%   <- today

It has never been healthy. Today is simply the worst day.

6. Retry storm on top. 89 distinct documents failed today, each retried 25–77 times, none ever succeeding. The gateway's negative cache absorbed 268 of those replays, so they never reached the GPU — the cache is doing its job, and it is why the real 400 count (90) is far below the attempt count.

What is NOT broken

• ahu-ocr/internal/dev:notaris — 122 calls, 122 OK, 0 errors today. The interactive upload path works perfectly.
• Zero shared traces between the failing system:* flow (570 traces) and the working dev:notaris flow (54 traces). These are genuinely separate flows, not two layers of one request.
• The demo surface x056.ahu-demo.ocr.val.id is served by this instance and its interactive path is green.

Where to look

The system:* path acquires document bytes somewhere the interactive path does not, and gets an empty buffer. Two concrete pointers:

• The uploads volume (ahu-ocr-staging_uploads) holds 4,069 files, exactly 1 of them zero-byte, and its newest file is Aug 25 14:35 — nothing was written on Aug 26. So the failing flow is not reading these files; it is producing its buffer some other way.
• The 400 is raised at app.py:216 in ahu-paddle-ocr (Image.open(...) on a non-image buffer). Magic-byte detection there is fine; it is being handed nothing to detect.

Two adjacent issues — DECIDED and DONE (2026-08-26)

1. The only OCR deployment in the fleet is the staging project (com.docker.compose.project=ahu-ocr-staging, workdir /home/efran/ahu-ai-staging/ocr/infra), and the public demo vhost points at it. Decision: keep it. The demo runs on staging.

2. It reported X-Surface: internal, not staging — so a month of staging batch failures scored against production signals. Fixed. No engine-repo change was needed: the override already exists (GATEWAY_SURFACE, backend/src/lib/gateway-headers.ts:31, added after the 2026-08-14 dev-retry-loop paging incident). Set GATEWAY_SURFACE=staging in infra/env/shared.env and recreated the container.

It had to go in env/shared.env, not compose.staging.yaml — build-and-ship.sh line 28 scps the compose file over the host's on every deploy, so a compose edit would silently revert on the next ship. env/shared.env is never shipped (it carries live SABH/registry creds).

Verified end-to-end: the engine's own header builder now returns {"tenant":"ahu-ocr","surface":"staging"}, and a real call through the gateway landed in the audit as ahu-ocr | staging | ... | ok.

staging was deliberately NOT added to the observatory's exclude_surfaces. Labelling and excluding are separate decisions. This is the instance the demo runs on, so excluding it would blind the early-warning rules for exactly the thing that matters on the 28th — and buy nothing, since those rules fire on zero staging traffic today. Add it after OCR has a real production deploy; it is one line, and the label is now in the audit trail waiting for it.

Platform-side changes already made (not a fix for this bug)

The Observatory was describing this failure as an attack. That was wrong and is fixed: cached REFUSED_* replays no longer count as guardrail-probing evidence. The failure itself remains fully visible — those actors still read 358 calls / 358 errors / 100% in the actors table. Nothing about this bug was hidden; only the security verdict changed.