think
16px
820px
← all files

Incident report — OCR storage unhealthy + GPU Server wedged (2026-08-21)

Two independent faults, both recovered same morning. No data loss in either.

Incident 1 — OCR Akta (Notaris): storage: "healthy" is false

Cause. 6 orphan Document rows (4 test akta PDFs, 2 KTP images) unreachable
from any submission — leftovers of the pre-30-Jul deletion leak
(SubmissionDocument.document has no cascade; deleting a permohonan kept the
Document rows + files). The 30 Jul patch fixed the leak going forward but never
swept existing orphans. missingFiles: 0 — no bytes lost; stray KTP files
outside every retention path were a PDP liability, not a loss.

Recovery. Dry-run via the app's own GC (findOrphanDocuments) to identify
all 6, then purgeAllOrphanDocuments(): 6 rows + 6 files deleted, 0 refused
(sharing + containment guards). Verified: fresh scan 0 orphans;
/api/health/storage → healthy: true, problems: [].

Incident 2 — GPU Server (ahu-gpu-server): Bermasalah

Cause (proximate). Process wedge: uvicorn accepted connections and logged
"200 OK", but responses never reached ANY client — in-container, bridge, and
host-published paths all timed out; 438 sockets stuck in CLOSE_WAIT while the
app held only 27 fds. Zero job traffic ≥30 min; celery queue empty; CPU 0%.
Docker healthcheck failing streak 122. Root cause of the wedge itself is
unproven (state was destroyed by the restart) — CLOSE_WAIT accumulation points
at a connection-handling leak in the app's accept/response path.

Recovery. Confirmed nothing in flight (queue 0, no job logs), then
docker restart ahu-gpu-server. Healthy in 25s; /health 200 in 0.9 ms,
models_loaded: ["qwen-vlm"]. OCR extraction path restored.

Follow-up worth doing (not done): the healthcheck detects this wedge but
nothing acts on it — 122 consecutive failures restarted nothing. Consider
autoheal labels or a restart-on-unhealthy policy for this container, and an
upstream look at the CLOSE_WAIT leak in ahu-gpu-server's server loop.

Dashboard defect — why the row said "Sehat"

deriveState ranked evidence Docker-healthcheck > probe and NEVER read the
auxiliary checks: results — they rode along as Keterangan text only. So OCR
showed green "Sehat" (its Docker healthcheck pings basic liveness, which
passed) beside a red storage verdict for days. A check the operator chose to
configure is a verdict, not a footnote.

Fix (deployed). New state degraded: healthy base + any failing non-absent
aux check → red "Terganggu" (observatory 6ebedf6, dashboard 7bcd031).
Absent endpoints stay informational; a failing aux never softens down/missing.
Test covers all four directions. Both containers rebuilt + healthy.

Timeline (WIB)

  • 12:32 Efran reports board contradiction + GPU Server Bermasalah
  • 12:35 OCR storage endpoint: 6 orphan rows; disk 89% (not the cause)
  • 12:40 orphans identified (dry-run), purged; scan clean
  • 12:45 GPU server diagnosed wedged for all clients; restarted; healthy
  • 13:0x degraded-state fix implemented, tested, deployed to both containers