Agent prompt — unjam the PaddleOCR `/layout` worker on `ai-ahu`

Run this in a Claude Code session that has shell access to the GPU host
ai-ahu (192.168.83.20) — either directly on the box, or from a machine whose
SSH key is installed there (ssh efran@192.168.83.20). Paste everything below
the line as the prompt.

You have shell access to the GPU host ai-ahu (192.168.83.20), directly or via
ssh efran@192.168.83.20 '<cmd>'. A PaddleOCR service exposes a /layout
endpoint behind the AI gateway on port 8200. It is used for OCR of scanned
PDFs.

Symptom

The /layout endpoint itself is UP: a bodyless POST http://192.168.83.20:8200/layout
returns HTTP 422 in a few milliseconds, and cached results return instantly.
But real OCR of any non-trivial PDF now hangs: a 9-page PDF times out at the
client's 5-minute limit. A 3-page PDF OCR'd fine (~60s) earlier when the
worker was free, so throughput is ~20s/page when healthy.
This has persisted for ~6 HOURS with no self-recovery.

Most likely cause

A client sent a heavy OCR request and then aborted: a 45-page PDF whose pages
are ~18-megapixel full-color scans (≈3200x4800 px each), with document
unwarping (UVDoc) enabled (the akta default). Running dewarping + detection +
recognition on 18MP RGB images, 45 of them, is a large GPU workload — a strong
candidate for a CUDA out-of-memory or a hang that wedged the worker. An HTTP
abort does NOT cancel server-side OCR, so if PaddleOCR is a single worker it
stayed stuck on that job (or crashed into a bad state) and every new request now
queues behind it. It has NOT recovered after ~18+ hours, so it is genuinely
wedged, not merely slow. Confirmed from the client: the endpoint answers a
bodyless probe with HTTP 422 in ~5ms, but every real OCR request — even 2 pages,
even with unwarp off — times out at 300s.

Your goal

Get /layout healthy again (small PDFs OCR in a normal ~20s/page) with the
least disruptive action possible.

Phase 1 — READ-ONLY diagnosis first (do not restart/kill anything yet)

Find the PaddleOCR service: docker ps (look for a container whose name/image
mentions paddle / ocr / layout), and ps aux | grep -iE 'paddle|ocr|layout'.
Report the container/process name, image, uptime, and restart count.
Is it actively working or wedged? Check CPU/GPU/mem for that process and
docker logs --tail 200 <name> (or its log file). Look SPECIFICALLY for a
CUDA out-of-memory or a worker stuck on one job that started ~18h+ ago
and never returned. Also check the host kernel log (dmesg -T | tail -50)
for an OOM-killer event or GPU Xid errors around when it got stuck. A big
unwarp job on 18MP images is the suspected trigger, so GPU-memory exhaustion
is the leading hypothesis.
nvidia-smi — report utilization on both H100s and which processes hold GPU
memory (vLLM + classifier also live here). Is OCR starved or contending?
Note: the host is ~92% disk full. Check df -h — if a partition is 100%
full it can wedge a service. Report but do not aggressively clean.
Determine the concurrency model if discoverable: single worker vs pool, any
server-side request timeout, any queue. This tells us if one big request can
block all others (it appears it can).

Summarize what you found before doing anything that changes state.

Phase 2 — Targeted fix (authorized, because 6h of waiting has failed)

Only after Phase 1 confirms the OCR worker is the stuck component:

Restart ONLY the PaddleOCR container/service (e.g. docker restart <name>,
or its systemd unit if that's how it runs). This is authorized — the "wait for
it to drain" path has already been tried for 6 hours and failed.
Hard guardrails — do NOT:
touch, restart, or stop ahu-vllm (port 8001/8000), ahu-classifier,
ahu-gpu-server, the gateway (8200/8210), or any other container;
reboot or power-cycle the host;
delete files, prune docker images/volumes, or free disk to "make room"
(disk is 92% full but that is a pre-existing condition, not your task);
change any config or env.
If the PaddleOCR container's identity is ambiguous, or a restart looks like it
would disrupt anything else, or GPU memory is so full a restart might not
recover — STOP and report instead of acting.

Phase 3 — Verify and report

After the restart, confirm the endpoint is healthy: a bodyless
POST http://192.168.83.20:8200/layout should still return 422 fast, and if
you can, OCR a small PDF and confirm it completes in a normal time
(~20s/page) rather than hanging.
Report: what was stuck (was it a CUDA OOM?), what you did, current
GPU/mem/disk state, restart count, and whether the service has a
server-side request timeout and a concurrency model. Given the
trigger was likely a 45 x 18MP unwarp job, the durable hardening is:
(a) a server-side per-request timeout so an abandoned/oversized request
self-cancels; (b) an input-image resolution cap / downscale before
detection (18MP is far more than OCR needs — ~2000px on the long side is
plenty); and/or (c) a max-pages-per-request limit so callers must chunk.
Recommend whichever apply. NOTE the client side will also start downscaling
+ chunking so it stops sending 18MP x 45 in one request.

Keep the final report concise: root cause, action taken, current health, and
the one or two config hardening recommendations.