Container board — visual rework, post-reboot truth, and the dead-weight audit

Date: 2026-08-14 · Host: ai-ahu · URL: http://192.168.83.20:8320/containers

Follows the deployment report.

The board caught a real incident while I was working on it

Two containers exited at 02:54 UTC and stayed down, because both had restart=no — the same latent trap you found on the three Milvus containers before rebooting. My earlier policy audit missed them because docker ps only lists running containers; the full audit (docker ps -a) finds them.

ahu-cleanup-llm is a registered gateway upstream (cleanup-3b → http://192.168.83.20:8003/v1). I restarted it and set --restart unless-stopped.

It did not come back. Root cause:

GPU 0:  85,405 MiB used   9,915 MiB free      ← ahu-vllm holds 85 GB
GPU 1:  23,126 MiB used  72,194 MiB free
cleanup-llm: --gpu-memory-utilization 0.12 × 95,830 MiB ≈ 11.5 GiB needed

It needs ~11.5 GiB, GPU 0 has 9.9 GiB free, and it is not pinned to GPU 1 — which has 72 GiB free. Its log has been frozen mid-model-load for over ten minutes; it is wedged, not loading. That also explains the 02:54 SIGKILL: it was squeezed off GPU 0.

And this is the whole feature proving itself

ahu-cleanup-llm   docker_state = running        ← Docker says "Up"
                  state        = unhealthy       ← the probe says otherwise
                  probe        = dial tcp 192.168.83.20:8003: connection refused

Docker reports it Up. It is serving nothing. This is exactly the scenario from the original brief — "the gateway could be wedged returning 500s and still report Up" — happening for real, and the board caught it and lit Notaris → degraded through the OCR dependency.

I have not changed its GPU pinning: that is a capacity decision with real trade-offs (pin to GPU 1, or reduce vLLM's 85 GB allocation), and it is yours to make.

Current truth

Every "terblokir mismatch driver NVIDIA" note is gone — ahu-gpu-server, ahu-classifier, ahu-worker-akta and ahu-worker-llm all have active, passing healthchecks now. ai-ahu-rag is recorded as ours, reporting via a Docker healthcheck, not a probe.

1. Visual rework — quiet healthy, loud problems

The board had 42 identical cards each wearing a green pill. That is the same no-signal failure as a wall of green; it just fails by flooding rather than by lying.

• Summary band answers "is everything fine?" without reading 42 cards: whole-host verdict + total / healthy / problems / unverified / affected lanes. The verdict takes the worst thing present, and unknown outranks ok deliberately — while anything runs unchecked, the honest headline is "Sehat, sebagian belum terverifikasi", never "all good".
• Healthy recedes: no status pill, secondary ink, a hairline neutral rule rather than a success accent. Saturated colour, weight and glyphs are spent only on problems — per the house dataviz rule that saturated fills are for small marks and accents, never large blocks.
• Status is never colour-alone. Every state still carries its word; non-healthy states carry a glyph too. The board survives print, forced-colors and colour-vision deficiency.
• Problems float to the top of every lane, healthy last, alphabetical within a bucket so the list does not reshuffle every poll. This is the cheapest fix for "never scan for the red one", and it works even if colour fails entirely.
• Platform lane compacted: its ~30 healthy containers collapse into a dense one-line grid; its exceptions keep full cards. Each dense cell still carries its name and a glyph.
• Blast radius moved to the lane header — "Notaris ⚠ terdampak: degradasi karena Cleanup LLM (dugaan)". An unaffected lane collapses to a quiet count and takes no space.

2. Production splashes removed

Engineering rationale is now behind a Detail disclosure. At rest a card says: title, container name, state, source of truth, uptime. Nothing else. A test asserts the note is absent until the disclosure is opened, so it cannot creep back.

3. Dead-weight audit — proven, not guessed

The three Redis instances are not redundant. Different tenants, different client libraries, different data, all with live connections. That question is answered: no consolidation available without merging the tenants themselves.

One correction worth flagging: I initially suspected ahu-cleanup-llm was dead weight on the strength of "nothing in any container env references it". That was wrong — it is referenced from gateway.yaml, a mounted config file, which an env-only grep cannot see. Recommending a stop would have broken KTP cleanup. Config files, not just env, have to be in scope for this kind of audit.

4. OCR storage 404 — verified, not assumed

The running image is ahu-ai-ocr:337266e0, built 2026-08-11, and does not contain 07a3a73e. Checked directly:

:3520/api/health          → 200   {"status":"ok","ocrProvider":"azure-on-prem",...}
:3520/api/health/storage  → 404

So the path and port are right and the endpoint simply is not deployed. Not a wiring bug. The check already colours on the healthy field (not the status code, since the endpoint always answers 200) and bounds staleness at 300 s via ageMs. It goes green by itself when the OCR image ships; the red tag is a visible pending-deploy marker rather than a hidden gap.

Recommended next actions

1. Decide ahu-cleanup-llm's GPU — pin to GPU 1 (72 GiB free) or trim vLLM's 85 GB. Until then KTP cleanup's LLM path is down.
2. restart=no is a recurring trap. Two more containers had it. Worth a one-off sweep plus a rule that anything long-lived is created with unless-stopped.
3. Stop ahu-chatbot-orchestrator (already down) and ahu-azure-di-studio, then remove from compose.
4. Redeploy the OCR image to activate the storage check.