Spec — Container health view (observatory)

Date: 2026-08-13 · Host: ai-ahu (192.168.83.20, VPN-only) · Repos: ahu-ai-observatory, ahu-observatory-dashboard

The ask, in Efran's words: "see visually what's wrong and who faces that container (public? internal? notaris?)".

1. The finding this design exists to survive

Measured on ai-ahu, 2026-08-13, via the Docker Engine API:

The 23 include the load-bearing ones: ahu-platform-gateway-1, ahu-gpu-server, ahu-classifier, ai-ahu-rag, ahu-platform-redis, ahu-observatory-db, ahu-observatory-observatory-1.

For those 23, Docker knows exactly one thing: the process has not exited. The gateway could be wedged returning 500s to every caller and Docker would still say Up 4 weeks.

Therefore the central design rule: "running, but nobody is checking" must never render green. A wall of green that means nothing is worse than no dashboard, because it will be trusted. This is the same class of error as the OCR confidence scores, which turned out to be decorative.

The unverified count is a first-class headline number. Watching it shrink is the point of the feature.

2. States — seven names, four tones

Four colours is the visual budget. But collapsing genuinely different situations into one bucket is how a dashboard starts lying, so the state name on the tile is precise and only the tone is bucketed.

starting earns its own name because a container crash-looping shows starting forever, and that is diagnostic information a grey blob would hide.

How "never green" is actually enforced

The dashboard's existing and only guarantee that something isn't tinted with a status colour is the absence of a tone, not the presence of a grey one. StatTile writes data-tone={state === 'ready' ? tone : undefined} (StatTile.tsx:58) and the SCSS has no [data-tone='neutral'] rule. The unknown bucket therefore passes tone={undefined} and carries its meaning in the label plus a gray Carbon Tag. There is no code path where an unverified container receives a success token.

3. Data source — read-only socket proxy, never the raw socket

tecnativa/docker-socket-proxy on the GPU host, locked to CONTAINERS=1 with every other permission and all write verbs left at their deny default. The observatory polls it over HTTP on the ahu-observatory_default network.

The raw Docker socket is never mounted into a network-reachable service. It is root-equivalent on the host: anything that can reach it can start a privileged container that mounts /. The observatory is an HTTP service on the VPN; it does not get the socket. If the proxy approach fails, this feature stops and gets reported — it does not fall back to the bare socket.

The version trap (found during recon, shapes the implementation)

The Docker Engine API only returns the structured Health field on the unversioned path:

GET /containers/json?all=1     → "Health": {"Status": "healthy", "FailingStreak": 0}   ✅
GET /v1.44/containers/json     → no Health key at all                                  ❌
GET /v1.41/containers/json     → no Health key at all                                  ❌

Pinning a version prefix does not error — it silently drops health data, and every container on the host would read as unverified. That fails safe (grey, not green) but it is still a lie, and it would make the headline number permanently 41.

Mitigation, in order:
1. Poll the unversioned path.
2. Type the field as a pointer in Go so "absent" and "none" are distinguishable.
3. If Health is absent, fall back to parsing the human Status string for (healthy) / (unhealthy) / (health: starting).
4. If that is also inconclusive, the state is unknown — not unverified. "The API didn't tell us" and "there is no healthcheck" are different facts and the UI shows them differently.

Rejected alternatives, not to be re-analysed: cAdvisor + Prometheus (extra component, weak on healthcheck state); SSH + parsing docker ps (worst security and reliability).

4. Layout — swimlanes by audience

41 tiles in a flat grid is unreadable. Group by who gets hurt. Mapping confirmed by Efran:

The Unclassified lane is load-bearing, not a fallback. A container nobody mapped is exactly the one that will break. Silently dropping unknown containers would make the board quietly incomplete as the host evolves; instead a new container shows up as an unclassified grey tile and asks to be classified.

5. Blast radius — the feature that makes this worth building

When a Platform container degrades, light up the audience lanes that depend on it. Redis dying is not "one red box", it is "public + notaris + internal are all affected". That is the question actually being asked.

Edges are authored on the consumer side

Docker does not know dependencies. They are declared in the registry, on the consumer (ahu-ai-chatbot-public depends_on: [gateway, rag]) because that is how people reason about them and it matches Compose's own idiom. Blast radius is then the transitive closure over reverse edges, filtered to audience lanes.

Every edge carries a confidence

depends_on:
  - {name: ahu-platform-gateway-1, confidence: verified}
  - {name: ai-ahu-rag,             confidence: assumed}

verified means we read the config or traced the call path. assumed means plausible and unproven. They render differently, and an impact reached only via assumed edges is itself labelled assumed. v1 does not invent precision it cannot demonstrate — most edges will ship as assumed, and that is the honest state of our knowledge, visible rather than hidden.

Multi-path confidence takes the max: one verified path makes the impact verified regardless of how many assumed paths also exist.

Computed in Go, not in the browser

impacts (static, from the graph) is computed server-side as a pure function over the registry and unit-tested. The frontend stays trivial: highlight lane N if any degraded container's impacts includes N. Graph maths in one tested place beats graph maths in a React render.

6. Probes — turning honest grey into honest green

Some containers cannot get a Docker healthcheck because we do not own their compose file. For those, the observatory can perform its own HTTP probe. Verified during recon, all reachable and answering:

A probe result carries source: "probe", distinct from source: "docker_healthcheck". The UI states which one produced the verdict, because "the observatory got a 200 from outside" and "the container's own healthcheck passes" are different strengths of evidence and conflating them would overstate what we know.

ahu-signature-verify stays unverified: it is loopback-bound, not in our edit scope, and unreachable from the observatory. That is reported, not papered over.

7. Honest failure modes

That last row is a stated requirement: other sessions are adding healthchecks for ahu-ai-workers and the obert-owned services. Because state derives from what the API reports at poll time rather than from anything baked into the frontend, a healthcheck appearing next week flips a tile from grey to green with zero code changes.

8. Interfaces

Config (observatory.yaml), dormant when absent

containers:
  enabled: true
  docker_url: "http://ahu-docker-proxy:2375"
  host: "ai-ahu"
  interval_s: 30
  staleness_s: 180
  probe_timeout_s: 5
  registry:
    - name: ahu-ai-chatbot-public
      title: "Chatbot Publik"
      audience: public
      depends_on:
        - {name: ahu-platform-gateway-1, confidence: verified}

Follows the house idiom exactly: one defaultContainers var, applyContainersDefaults, fail-loud validateContainers with dotted-path errors, wired as an apply→validate pair at the end of Load.

GET /api/containers — executive+ (rolesSummarySeries, same as /api/gpu/now)

{
  "polled_at": "2026-08-13T10:22:00Z",
  "stale": false,
  "host": "ai-ahu",
  "counts": {"healthy": 18, "unverified": 23, "unhealthy": 0, "down": 0,
             "missing": 0, "starting": 0, "unknown": 0, "total": 41},
  "lanes": [{"id": "public", "label": "Public (citizens)", "affected": false,
             "affected_by": []}],
  "containers": [{
    "name": "ahu-platform-gateway-1", "title": "AI Gateway", "audience": "platform",
    "state": "unverified", "source": "none",
    "docker_state": "running", "docker_health": "none", "failing_streak": 0,
    "status_text": "Up 4 weeks", "started_at": "2026-07-16T...", "image": "...",
    "registered": true, "probe_error": null,
    "depends_on": [],
    "impacts": [{"audience": "public", "confidence": "verified"}]
  }],
  "unclassified": ["some-new-container"]
}

Storage is a container_samples hypertable, matching the gpu_samples / cache_samples precedent — no in-memory state. This also gives the unverified count a history, so "watch it shrink" becomes a real time series rather than a vibe.

Dashboard

New /containers view, nav entry, executive-visible. Swimlane board built on the existing .fleet-cards / .fleet-card pattern from OperatorView (the closest precedent to a board in the codebase). Click a Platform card to pin its blast radius; click again to clear. Polls at DATA_POLL_MS, feeding the global refresh indicator automatically. WIB timestamps via the existing formatters.

9. Healthchecks we own — closing the gap ourselves

Adding real probes (a request that proves the service answers, never pgrep) to the compose files we control. In-container tooling was checked, because a healthcheck runs inside the container and most of these images are minimal:

ahu-azure-di-studio has no shell at all (exec: "sh": executable file not found), so it cannot take an in-container healthcheck. It gets an observatory-side probe on :5010/ instead, and the UI says so.

10. Non-goals for v1

CPU/memory graphs (Prometheus is already on the host at :9090 if that is wanted later); multi-host support (the schema carries host so it is not blocked, but only ai-ahu is polled); editing container state from the dashboard (read-only by design — restart authority does not belong behind a browser session); Docker labels as the metadata source (the documented migration target, deferred because it means editing hands-off repos).

11. Definition of done

1. Socket proxy deployed, locked down, verified to preserve the Health field.
2. Backend polls, stores, serves; tests pass.
3. Dashboard renders four tones across five lanes; unverified is grey everywhere.
4. Deployed to ai-ahu, screenshotted.
5. A non-critical container is deliberately stopped and both its tile and the blast radius are observed reacting. Verification against reality, not against a fixture.
6. The report states the honest count: verified vs merely running.