The HTTP_400 "storm" — root cause

Date: 2026-08-14 · Verdict: not a storm and not an outage. 8 permanently-broken files retried ~5,800 times over five weeks, which is currently producing 98% of the entire platform's error volume.

What it actually is

What the upstream says, verbatim from the stored response bodies:

{"detail":"PDF rasterization failed: Unable to get page count.
 Syntax Error: Couldn't find trailer dictionary
 Syntax Error: Couldn't read xref table"}

{"detail":"Image decode failed: cannot identify image file <_io.BytesIO ...>"}

Corrupt PDFs and undecodable images. These inputs can never succeed. No amount of retrying changes a missing xref table.

Why it matters: it has destroyed the error signal

Last 7 days, all error codes:

98% of every error on the platform is these few files. That is what fired the early-warning alert reading "100% of last 15 min failed". The alert was not wrong — the calls really were all failing — but an alert that fires for a month without anyone acting is a broken alert. Any real incident is now invisible underneath this.

At the observed rate that is roughly 200 futile GPU/CPU OCR attempts per working hour.

Second bug found: zero-byte files are accepted

One of the looping hashes is e3b0c44298… — the SHA-256 of the empty string. It has 221 attempts.

The OCR database confirms it:

So the ingest path accepts a zero-byte upload, and something then submits it to OCR indefinitely. A 0-byte file cannot be OCR'd under any circumstances. Worse, a 0-byte .json reached VERIFICATION_READY — a state that implies a human should review extracted fields that cannot exist.

What it is not

Ruled out, so nobody re-treads it:

• Not the document pipeline. No Document row has been touched since 2026-08-06, yet the 400s continued through 2026-08-14. Nothing is stuck in OCR_PROCESSING.
• Not the test suite. I hashed every .pdf/.png/.jpg/.json fixture on origin/master — none matches any looping hash, and there are no corrupt/invalid/empty-named fixtures.
• Not a user hammering refresh. The caller is a system actor and spacing is machine-regular.

The trigger correlates with working hours (109 at 08:00, 77 at 09:00, 187 at 10:00, 41 at 13:00, then 8 at 02:00), so something in the internal layout path re-submits the same failed set whenever it runs.

Fixes (engine work)

1. Treat HTTP_400 from an OCR upstream as permanent. Per CONVENTIONS.md §3 only 429/503 + Retry-After are retryable; a 4xx means the input is invalid. Record the failure with the upstream's own reason and stop. A negative cache keyed on content hash would make this structural rather than incidental.
2. Reject zero-byte uploads at ingest, and reject types that cannot be documents (.json). Fail fast at the door with a clear message.
3. Surface the upstream's reason to the user — "PDF rusak / tidak dapat dibaca" — instead of silently retrying forever. Right now a notaris gets no signal and the platform burns GPU.
4. Backfill: mark the 8 known-bad documents terminal so the loop stops today, and clean up the 0-byte rows (especially the VERIFICATION_READY ones, which are lying about being reviewable).

One observability change worth considering

The EWS alerted on failure rate, which a poison-pill trivially saturates. Alerting on distinct documents failing — or grouping by error_code + doc_hash — would have said "8 documents are permanently broken" instead of "everything is down", which is both true and actionable. Fix the cause first; this is the follow-up that stops it recurring.