Container health view — deployed

Date: 2026-08-13 · Host: ai-ahu (192.168.83.20, VPN-only) · Status: live
URL: http://192.168.83.20:8320/containers (VPN) · API: GET /api/containers (executive+)

Spec: 2026-08-13-container-health-view-spec.md

The number

"Verified" means a check ran and could have failed. It counts healthy and unhealthy — both are verdicts. It does not count "the process has not exited".

What is actually still unverified — 5, honestly

These render grey, not green. The dashboard says why on each card.

Blocker — the GPU driver on ai-ahu is mismatched

Found while preparing the last four healthchecks. No new GPU container can start on this host right now.

NVRM kernel module : 580.159.03
NVIDIA userspace   : 580.173.02
$ nvidia-smi        → Failed to initialize NVML: Driver/library version mismatch
$ docker run --gpus device=0 nvidia/cuda:12.4.0-base nvidia-smi -L
  → OCI runtime create failed: open /run/nvidia-persistenced/socket: no such file or directory

The libraries were upgraded; the loaded kernel module was not. Running GPU containers survive only because they hold device handles obtained before the upgrade.

I stopped rather than proceeding. Recreating ahu-gpu-server, ahu-classifier, ahu-worker-akta or ahu-worker-llm to activate their healthchecks would have taken all four down with no way to bring them back — OCR and the classifier gone until the host is fixed.

This is worth acting on independently of this feature: the host is one container restart away from losing every GPU service. Fixing it means reloading the NVIDIA kernel modules, which in practice means a reboot (the modules are in use). Once that is done, docker compose up -d --force-recreate gpu-server classifier worker-akta worker-llm in ~/ahu-ai activates the last four healthchecks and the unverified count drops to 1.

The fabricated healthcheck

ahu-azure-di-id-document was running:

healthcheck:
  test: ["CMD-SHELL", "exit 0"]

A check that cannot fail. Docker reported it (healthy), and the board would have rendered it green on that basis — the exact failure mode this feature exists to prevent, sitting inside the data source.

It could not be fixed in place: that image ships with no shell, bash, curl, wget or python3, so Docker has nothing to run inside it. That is presumably why exit 0 was there. Its only dependent (azure-di-studio) uses the plain list form of depends_on, not condition: service_healthy, so removing it blocked nothing.

Removed, and replaced with an observatory-side probe on :5002/ready. The board now reports it as healthy with source probe observatory — a weaker, accurate claim instead of a stronger, false one.

Healthchecks added

Active now (container recreated, verified passing):

Written, awaiting the driver fix: ahu-gpu-server (/health), ahu-classifier (/ready), ahu-worker-akta + ahu-worker-llm (scoped celery inspect ping).

Two details worth keeping:

• redis-cli ping alone is not enough. It exits 0 even when the server answers with an error — e.g. LOADING while an AOF replays. "Still loading" is not "ready to serve", hence the grep -q PONG.
• An unscoped celery inspect ping is a fake green. It replies OK whenever any worker on the broker is alive, so worker-llm would report healthy while dead if worker-akta were up. -d celery@$(hostname) scopes it to the container's own node. Verified: it returns non-zero for a node that does not exist.

Every command was run in-container before being written into compose, including a negative control to confirm it can actually fail.

Observatory-side probes (9) for containers whose compose we do not own: ai-ahu-rag, ahu-chatbot-orchestrator, ahu-chat-interface (obert's), ahu-dcgm-exporter, ahu-cleanup-llm, tei-qwen3-embed (bare docker run), ahu-paddle-ocr, ahu-azure-di-studio, ahu-azure-di-id-document (no shell in image).

Verified against reality, not fixtures

Stopped keep-keep-websocket-server-1 — genuinely non-critical (realtime UI only) but with a real dependency edge into the Internal lane.

before   state=healthy   source=docker_healthcheck   internal lane: calm
stop     ↓
after    state=down      source=docker_state         status="Exited (0) 23 seconds ago"
         internal lane:  affected=True  by=[keep-keep-websocket-server-1]
         public/notaris/platform lanes: still calm
restart  ↓
after    state=healthy   source=docker_healthcheck   internal lane: calm again

The tile changed, the right lane lit up, and — as importantly — the other three lanes did not. A board that reddens everything is the same failure as one that greens everything.

Data source

tecnativa/docker-socket-proxy with CONTAINERS=1 and nothing else, no published ports, on the observatory's compose network. Verified after deploy:

GET  /containers/json           → 200   (the only thing allowed)
POST /containers/create         → 403
POST /containers/ahu-vllm/stop  → 403
GET  /images/json               → 403
GET  /info                      → 403
GET  /exec                      → 403

The raw Docker socket is never mounted into the observatory. It is root-equivalent on the host — anything that can reach it can start a privileged container that mounts / — and the observatory is a network-reachable HTTP service. The proxy holds the socket; the observatory gets one read-only endpoint.

The version trap

The Engine API returns the structured Health field only on the unversioned path:

GET /containers/json       → "Health": {"Status":"healthy","FailingStreak":0}   ✅
GET /v1.44/containers/json → no Health key at all                               ❌

A version prefix does not error — it silently omits health, and every container on the host would read as having no healthcheck. Startup now rejects a docker_url containing a path for exactly this reason, and the poller types Health as a pointer so "absent" and "none" stay distinguishable: absent falls back to parsing the status string, and if that is inconclusive the state is unknown, not unverified. "There is no healthcheck" and "we could not tell" are different facts.

Design notes

Seven state names, four colours. Four colours is the visual budget, but collapsing different situations into one name is how a dashboard starts lying. Tiles show the precise state; only the colour is bucketed.

Only healthy is ever green, and an unrecognised state from a future server build falls through to grey — the default fails toward humility.

Blast radius is computed from authored edges, because Docker does not know them. Each edge carries a confidence: verified (read out of a real config or env, source noted inline in the registry) or assumed (plausible, unproven). A path is verified only if every edge on it is; one verified path beats any number of assumed ones. Assumed impacts are labelled (dugaan) in the UI. Most of the OCR→Azure-DI edges are assumed and shown as such.

Roughly 40 of the edges were upgraded from guesses to verified by reading gateway.yaml, observatory.yaml, and the containers' own environment variables rather than inferring from names.

Ignorance is not degradation. unverified/starting/unknown never trigger blast radius. Had they, all 23 unchecked containers would have lit every lane on day one and the feature would have been useless immediately.

Staleness is the honesty mechanism. If the poller stops writing, samples age out of the window, the API returns stale: true, every tile reads unknown, and a banner explains why — instead of a frozen board still showing the last green it saw.

Unclassified is load-bearing. ahu-docker-proxy appeared mid-deploy and surfaced immediately as an unclassified grey tile, which is exactly the intended behaviour: a container nobody mapped is the one that will break. It has since been classified.

Corrections applied (round 2)

The ahu-ai-chatbot and ahu-ocr-tidyup sessions checked their own source and
found parts of the registry wrong. Two would have made the board lie in the
specific way it exists to prevent — just inverted: crying outage instead of
crying fine.

Blast radius now distinguishes degraded from down

The public chatbot fails soft: every tool call is wrapped in try/catch +
timeout, failures become ToolResult{ok:false}, and compose() answers from
whatever survived. The page loads, the stream completes, HTTP is 200 — the
answer degrades. Encoding a lost RAG as "public is down" is false, and
painting it the same red as a real outage trains people to ignore the red.

Edges now carry effect: hard|soft; impacts carry severity: down|degraded.
A chain takes the far end down only if every link is hard; across paths the
worst outcome wins. Verified live on the deployed board:

Demonstrated end-to-end by stopping the Keep websocket again: the Internal lane
now renders amber "Menurun karena" instead of red "Mati karena".

ai-ahu-rag was being probed by a liar

Its /health handler is literally return {"status":"ok"} — tagged "Liveness
probe" in its own source, answering in 0.65 ms without touching Milvus or
TEI. A green light over a dead RAG stack.

Replaced with POST /search {"query":"apostille","top_k":1}, requiring a
non-empty hits array — which exercises TEI-embed → Milvus for real
(measured 10.8–13.6 ms). Probes gained method, body, and response-body
assertions to make that expressible. "200 with zero hits" is exactly what a
dead index or a dead embedder looks like, and no status code can see it.

A bug my own verification caught: the router was a transitive conduit

After applying the corrections, ai-ahu-rag still reported notaris: down.
Cause: the gateway's routing targets were declared as its own depends_on,
so a RAG outage propagated hard through the gateway into the notaris lane —
which never touches RAG at all.

A router is not a monolith: one upstream dying does not kill it, other routes
keep serving. The gateway now declares only its own process dependency (Redis,
soft), and each consumer declares the upstreams it actually uses, directly.
Same reasoning softened two more false-hard paths: ahu-ai-workers (batch — a
dead dependency means a job retries later, not that staff lost a service) and
the observatory (survives losing Redis; does not survive losing its own DB).

Lane fixes

ahu-chat-interface moved public → internal: no nginx vhost on any host
proxies to it, it is LAN/VPN-only, and it is the staff SQL data-agent UI.
Marking it public means a 3am page for a dev UI no citizen can reach.
ahu-ai-workers → internal (staff-console batch; it now has a real healthcheck
as of ac28589). ahu-chatbot-orchestrator is labelled legacy, its probe
labelled liveness-not-readiness, and nothing depends on it — so its blast
radius is structurally zero, not merely small.

OCR storage: an auxiliary check, and it is currently red — correctly

Containers can now carry auxiliary checks: signals a service reports about
itself that are not its liveness. They render beside the tile and never feed
state or blast radius, because "the process is serving" and "its data is
consistent" are different questions.

GET /api/health/storage always returns 200 even when unhealthy, so the check
colours on the healthy field, not the status code, and bounds staleness
with ageMs (300 s) so a cached green cannot pass as a current one.

It currently reports unexpected status 404 — the running image is
ahu-ai-ocr:337266e0 (built 2026-08-11) and does not contain 07a3a73e. I
left it live rather than commenting it out: the 404 is true, it is visibly a
pending deploy rather than a storage fault, and it will go green by itself when
the OCR image ships. Hiding it would have made the gap invisible.

Unrelated finding, visible in the screenshot

The dashboard's existing alert bar shows "Error-rate spike — 100% of the last 15 min of AI calls failed (84 calls)". I checked whether my gateway restart caused it. It did not.

HTTP_400 failures on paddleocr / doc-classifier-svc / azure-di-onprem have been running all day, long before I touched the host:

…and back through 03:00–05:00. My first host action was ~10:12 UTC and the gateway restarted at 10:21:26, well after the pattern was established. The "100%" figure is a windowing artifact: the only traffic in that particular 15 minutes was one failing OCR batch.

HTTP_400 is a client error — the OCR app is sending requests the upstreams reject, not an infrastructure outage — which is consistent with the upstreams themselves probing healthy.

Characterisation (cheap, so here it is)

Last 12 hours, status=error AND error_code=HTTP_400:

It is one client. 88% of the errors are a single user_id on the
internal OCR surface hitting PaddleOCR's ocr operation. 409 distinct
trace_ids means 409 genuinely separate requests, not one request retrying —
so it is a caller issuing malformed requests repeatedly, all day, roughly
04:00–14:00 with a lull 06:00–07:00.

PaddleOCR itself is fine (:8108/health → 200 on every poll), so the upstream
is rejecting the payload, not failing. The next step is to look at what that
one client is sending — which needs the request bodies, i.e. real digging in
the OCR repo, not something to expand this task with. Route it separately.

Follow-ups

1. Reboot ai-ahu to resync the NVIDIA kernel module — then recreate the four GPU containers to activate their healthchecks. Independently important: GPU services currently cannot be restarted at all.
2. ahu-signature-verify — needs its owner to expose a reachable readiness endpoint.
3. Awaited from other sessions: ahu-ai-workers already has a genuinely good healthcheck (it verifies live queue consumers exist, not just that Redis answers). Readiness endpoints for the three obert-owned services would let their probes become real healthchecks. No dashboard change is needed when a healthcheck appears — state is derived from what the API reports at poll time, so a tile flips grey→green on its own.
4. Migrate the registry to Docker labels (ahu.audience=…) so the metadata travels with the container and cannot rot. Deferred: it means editing hands-off repos.
5. Row volume: 43 containers × 30s ≈ 124k rows/day in container_samples (a hypertable). Fine, but worth a retention policy eventually.

Files

ahu-ai-observatory — internal/config/config.go (Containers block, apply/validate), internal/store/containers.go + migrations/007_containers.sql, internal/ingest/containers.go (poller + prober + deriveState), internal/api/containers.go (computeImpacts, handler), deploy/containers-registry.yaml (the 42-entry registry), deploy/compose.yaml (socket proxy + healthchecks), deploy/observatory.example.yaml.

ahu-observatory-dashboard — src/lib/containers/state.ts, src/views/ContainersView.tsx, src/lib/api.ts, src/lib/nav.ts, src/App.tsx, src/styles/carbon.scss.

Host-only — ~/ahu-ai/docker-compose.yml, ~/keep-deploy/compose.keep.yml (both backed up as .bak-pre-healthchecks).

Tests: Go 9 packages green; dashboard 50 files / 513 tests green, lint clean. The production registry is itself under test — a typo in it fails the build rather than silently dropping a lane out of a blast radius on the deployed host.