GPU/LLM Manager + AI Observatory — Design

Date: 2026-07-04
Status: Approved in brainstorming session; pending final spec review
Scope: Two new projects — ahu-gpu-manager and ahu-ai-observatory — that sit between the existing AI engines (chatbot, Document Intelligence/OCR, doc classifier, and future engines) and the GPU/model backend.

1. Context and goals

Three engines run today, all calling GPU-backed model servers directly with no coordination, no shared queueing, and no persisted record of any AI call:

• ai-ahu-chatbot — Agno agents (Python) + Next.js orchestrator, calling vLLM (Qwen3.6-35B-A3B-FP8) over OpenAI-compatible HTTP, plus a TEI embedding server (:8100) and (dev-only) Alibaba DashScope for synthesis.
• ahu-ocr-akta-notaris — Bun/Hono backend + Python gpu-server (FastAPI + Celery, queues akta/llm), calling vLLM, PaddleOCR (:8108), on-prem Azure Document Intelligence, and two direct vLLM endpoints (:8001 classifier, :8003 cleanup-3b) that bypass everything.
• ahu-doc-classifier — in-process LayoutLMv3 (torch, fp16) FastAPI service; calls Azure DI for OCR.

Multiple containers reserve all GPUs; nothing arbitrates contention. Token counts and latencies are computed and discarded on stdout. Two egress paths are fully uninstrumented, and one dev path calls api.openai.com directly.

Goals:

1. A GPU/LLM Manager: single choke point for all model traffic (LLM, VLM, embeddings, OCR), with priority queueing, job orchestration, model lifecycle management, health monitoring, and replica autoscaling — designed for today's single GPU host and tomorrow's 16× B200 multi-node cluster.
2. An AI Observatory: full-capture, legally-durable audit trail of every AI call, plus a custom BI dashboard (executives + ops + auditors) and the ops console for the manager.
3. Queue-awareness end to end: engines (and their human users) always know when work is queued vs processing vs dropped — no request-timeout errors caused by invisible queueing.

Non-goals:

• Business forecasting / anomaly detection / early warning (roadmap "Predictive Intelligence Engine") — though the observatory's telemetry store becomes a training data source for it later (e.g. Prediksi Load AHU).
• Auto-applied tuning: the parallelism advisor only recommends; a human applies.
• Kubernetes adoption now — placement sits behind a driver interface; the k8s (or llm-d/KServe) decision is deferred to cluster arrival.
• Modifying the engine repos directly — integration happens via prepared prompts executed in each repo when Efran gives the go-ahead (see docs/integration/).

Roadmap mapping (DataHive Engine Readiness sheet):

2. Decomposition: two projects

ahu-gpu-manager — data plane + control plane. On the hot path of every AI call; must be tiny, boring, dependency-light, rarely redeployed.

ahu-ai-observatory — audit store, query API, custom dashboard, ops console. Downstream-only consumer; churns fast; can be redeployed anytime with zero engine impact.

The contract between them is small and stable: a versioned audit event schema (over Redis Streams) and the manager's admin/status API (consumed by the console). One team can own both; the boundary permits splitting teams later.

Rejected alternatives: one platform monorepo (couples an availability-critical proxy to fast-churning UI code); three projects (the dashboard is the audit store's only consumer — a third boundary serves one client and is pure overhead).

Language: Go for gateway, controller, node agent, and observatory backend (static binaries, real concurrency on the streaming hot path, best-in-class k8s client if that future arrives). React + Vite + Tailwind/shadcn for the dashboard (the OCR frontend's stack). Accepted trade-offs: Go is a third org language; the parked FastAPI gateway prototype in ai-ahu-chatbot git history (spec 3f4525f, impl 5b2feed) is a design donor, not a code donor.

3. Topology and high availability

┌── SERVICES VMs ×2 (no GPU) ──────────────┐      ┌── GPU HOST(S) (1 today → 16×B200) ──┐
│ gateway (active-active, VIP/LB)          │─────▶│ node-agent (1 static Go binary)      │
│ controller (active-standby, PG lock)     │      │ vLLM-397B ▪ vLLM-35B ▪ cleanup-3B    │
│ observatory: ingester + API + dashboard  │      │ TEI embeddings ▪ PaddleOCR ▪ Azure DI │
│ Postgres+Timescale ▪ Redis (Sentinel)    │◀─────│ ▪ doc-classifier (SDK-instrumented)  │
└──────────────────────────────────────────┘      └──────────────────────────────────────┘
        engines ──▶ gateway VIP only; nothing reaches a model server except through it

• Neither new project needs a GPU. The stack is logically separate from day one (zero GPU dependency, zero co-location assumptions); physically it runs on the current single host until non-GPU VMs are provisioned (cheap, per Efran). Services VM shape: 8–16 cores, 32–64 GB RAM, fast sizable disk — with full-capture audit of extensive OCR traffic, storage is the budget line (plan hundreds of GB/year).
• Node agent: per-GPU-host static binary exposing container start/stop/restart (local Docker API), GPU telemetry (NVML/DCGM), and replica health probes. Agents never make decisions. Adding a cluster node = installing one binary.

HA design (software obligations vs devops wiring):

Failure math: engines survive a gateway restart (LB fails over in seconds), an observatory outage (events buffer in stream/spool), and a controller outage (traffic flows on last-known replicas; only scaling pauses).

4. ahu-gpu-manager

4.1 Registry (source of truth)

Every upstream is an entry: type (llm-openai | embedding | vlm | ocr-http), class (on_prem | external_dev), replica spec (image, GPUs/replica, VRAM, quantization), scaling bounds (min/max replicas), per-traffic-class concurrency caps, health probe. Day-one fleet: qwen-397b (8 GPU/replica FP8, min 1 max 2 — pinned in practice), qwen-35b (1 GPU, min 1 max 8 — the elastic tier), cleanup-3b, tei-embeddings, paddleocr (min 1 max 4), azure-di (fixed container), dev-phase cloud entries flagged external_dev.

On-prem enforcement: production sets allow_external_upstreams=false; the gateway refuses to route to external_dev upstreams in prod. Audit events carry the upstream class, so the dashboard can prove "0 external calls." During development the flag is on and the same plumbing reaches cloud endpoints — engines are gateway-shaped from day one; GPU-server arrival is a registry edit, not a migration. This also structurally retires the committed-DashScope-secret problem (keys live server-side in the gateway).

B200 sizing note: 16× B200 ≈ 3 TB HBM. Qwen3.5-397B-A17B @ FP8 ≈ 400 GB weights → one replica spans 4 GPUs (tight) or 8 (comfortable KV). Qwen 35B-A3B @ FP8 ≈ 35 GB → one GPU per replica. Autoscaling is replica-granular: pin the 397B, elastically scale 35B/OCR/embeddings on the remainder, time-shift batch into idle windows.

4.2 Traffic classes and priority pools

Three classes: interactive (human waiting: chat turns, click-to-classify, verifier actions), batch (submission pipelines, nightly evals, reingestion), system (probes, advisor). Per-upstream slot pools; interactive admits first, batch fills the remainder (semantics of the parked prototype, generalized). Caps are runtime-tunable via admin API — no restarts — and every change is itself an audit event.

4.3 Data-plane APIs

(a) OpenAI-compatible /v1/* — chat/completions/embeddings. model maps through the registry to a healthy replica (round-robin across same-model replicas). Engines authenticate per-engine (static bearer tokens v1); gateway injects upstream keys server-side.

(b) Job API /jobs — batch and OCR. Wire-compatible with the existing gpu-server contract (202 {job_id} → GET /jobs/{id} → queued|processing|completed|failed + stage/progress), extended with queue_position and eta_ms. Payloads support documents (multipart), not just JSON. Fan-out groups: submit N children with max_parallel; the manager schedules children across replicas and merges status (e.g. a 12-document submission fanned at 4; a 40-page deed fanned page-per-replica then merged). This centralizes what the OCR backend does locally today (DOC_EXTRACT_CONCURRENCY) in the one place that sees actual GPU pressure.

(c) Generic HTTP upstreams (ocr-http) — thin adapters for PaddleOCR's API and Azure DI's async-poll protocol, giving OCR services the same concurrency caps, health, scaling, and audit coverage as LLMs. OCR is first-class: a user clicking "classify" outranks a 200-document overnight batch on the same PaddleOCR pool.

4.4 Queue-awareness (the no-RTO contract)

• Interactive: the gateway never holds longer than the client's timeout. Hold budget per class is configured below known client timeouts (orchestrator 60 s → hold ≤ 45 s; OCR's 30 s callers → hold ≤ 20 s or route to batch). During holds, streaming requests get SSE keepalives. Past budget → 429 + Retry-After + X-Queue-Depth (engines already degrade gracefully on these).
• Batch: a queued job is not a slow job. Poll responses distinguish queued (position, eta) from processing (stage, progress). Engine-side rule (one patch in the consolidated poll client): the timeout clock runs only while processing; a separate generous cap (default 30 min, configurable) bounds total queue wait.
• Human-visible queueing: poll surfaces render "Dalam antrean — posisi 3, ±2 menit" from the poll payload. Chat surfaces opt in via X-Queue-Events: on → structured event: queue SSE frames ({position, eta_ms}) before the first token; the orchestrator forwards a status event to the browser. SDK-managed clients (Agno) fall back to keepalive comments and stay functional. Any surface may also poll GET /queue/status?upstream=….

4.5 Dropped-request detection and safe retry

Rule: an engine can always distinguish "still queued" from "gone."

• Idempotency keys everywhere: engines send Idempotency-Key (they already generate request IDs). POST /jobs dedupes — resubmission after a crash returns the existing job, never a duplicate OCR run. For chat, a short-TTL result cache keyed on it returns the completed answer if the original finished, else re-runs.
• Durable job state: jobs live in replicated Redis, not gateway memory. Gateway death loses nothing; the LB reconnects engines to the survivor (same URLs). Worker death mid-job → heartbeat/visibility timeout → auto re-queue (Celery acks-late semantics, kept).
• Honest unhappy paths: graceful drain on shutdown (finish in-flight, 503 + Retry-After for new — never silent drops); a genuinely lost job polls as 404 + JOB_UNKNOWN → shared client resubmits with the same idempotency key. The shared client library (TS + Python, thin) holds reconnect/resume/resubmit logic once — replacing the OCR backend's seven duplicated poll loops.

4.6 Controller (separate process, crash-tolerant)

• Placement driver interface: ComposeDriver (talks to node agents) now; K8sDriver (or llm-d/KServe-backed) later. The driver boundary is where the undecided cluster-platform choice plugs in without touching engine-facing contracts.
• Health monitor: replica probes + NVML/DCGM telemetry via agents; unhealthy replicas pulled from rotation, restarted via driver.
• Autoscaler: inputs = queue-wait percentiles per upstream/class (gateway) + GPU telemetry (agents); policy = registry min/max bounds, scale-up on sustained queue-wait breach, scale-down after idle cooldown. Model loads take minutes → cooldowns prevent flapping. Ships in dry-run mode (logs intended actions) before enforcement.
• Gateway serves from last-known replica state if the controller dies.

4.7 Parallelism advisor (phase 3)

Periodic analysis over observatory telemetry produces recommendations — e.g. "queue p95 on qwen-35b:batch is 41 s while GPU util p95 is 58% → raise batch slots 8→12." Heuristic rules generate findings; the on-prem 35B writes plain-language rationale. Recommendations surface in the console with one-click apply; never auto-applied (apply = an audited config change).

5. ahu-ai-observatory

5.1 Event schema (the inter-project contract)

One versioned envelope per AI call, emitted by the gateway (and by SDK-instrumented in-process tenants like the classifier), over Redis Streams:

event_id (ULID) · schema_ver · ts · engine · surface · user_id · trace_id
upstream · model · upstream_class (on_prem|external_dev) · traffic_class
operation (chat|embed|ocr|classify|job|job_child) · status · error_code
queue_ms · upstream_ms · total_ms · tokens_in/out · doc_hash · pages
job_id / parent_job_id · config_version · request_body · response_body (capped, zstd)

trace_id propagates from engines' existing request IDs (X-Request-Id), linking one user action to its whole fan-out tree of OCR + LLM calls. If the stream is unavailable, the gateway spools events to local disk and replays — no silent audit gaps; the dashboard flags spool windows.

5.2 Tiered storage (one Postgres, two characters)

• Metadata → TimescaleDB hypertable; continuous aggregates (minute/hour) power all dashboard panels. Raw ~13 months; aggregates for years. Retention configurable.
• Bodies → append-only monthly partitions, hash-chained (row_hash = H(prev_hash ‖ payload)), UPDATE/DELETE/TRUNCATE blocked by DB triggers (the OCR repo's proven audit-startup.ts immutability pattern, applied to AI calls). Per-engine retention = partition drop + chain checkpoint so remaining history stays verifiable. Optional pgcrypto encryption at rest for PII-heavy engines. Body size capped (e.g. 10 MB, truncation recorded with full-content hash).
• Scale-out path: if volume outgrows Postgres, only the ingester's write path changes (ClickHouse); event schema and query API hold.

5.3 Query API and RBAC

Go service; three roles: executive (aggregates only), operator (ops panels + console), auditor (body-level drill-down). Every body view is itself recorded — directly implementing the roadmap's "Access Audit Trail" line.

5.4 Dashboard (custom-only, per decision)

React + Vite + Tailwind/shadcn. Panels: fleet health (GPU util/VRAM/replica states), queue depth & wait percentiles per class, SLA p50/p95/p99 per engine/model, throughput (tokens/s, jobs/day, pages OCR'd), per-engine/surface/user usage, error rates, external-calls counter (prove 0 cloud egress in prod), audit search + gated call-detail view, ops console (registry edits, parallelism caps, advisor recommendations, config history). One UI, role-scoped views.

5.5 Self-monitoring

Platform self-status (gateway pools, ingester lag, stream depth, Postgres/Redis health) is a dashboard page plus an external dead-simple liveness ping — the platform is never its own only monitor.

6. Engine integration contract

Engine repos are not modified by this project. Prepared prompts in docs/integration/ are executed by Claude Code sessions inside each repo when Efran gives the go-ahead. All engine changes are dormant-until-env-flip (backwards compatible; unset gateway env = current behavior).

Failure modes (asserted by chaos drills, §7)

7. Testing

• Unit: admission pools, placement, fan-out scheduling, hash chain, event schema round-trip.
• Integration harness: docker-compose with a fake upstream (stub OpenAI server: configurable latency/failures/streams), driven by the real engine clients (openai-python, Agno, Bun fetch) to prove keepalive tolerance and header propagation.
• Chaos drills (scripted, repeatable): kill gateway mid-stream, kill Redis, kill node agent, kill worker mid-job — each asserting the §6 failure table empirically.
• Load: target < 5 ms p99 proxy overhead on LAN; streaming throughput under concurrency.
• Audit completeness: gateway request count ≡ ingested events; chain-verifier tool for the body store.

8. Rollout (each phase independently shippable)

• P0 — Seed (weeks 1–2): Go gateway with prototype semantics; staging chatbot flipped; events to stub ingester.
• P1 — Choke point + audit (→ M2): HA gateway pair, static registry, ingester + tiered store, minimal dashboard (SLA, queues, audit search). All OpenAI-compatible egress flipped; external-upstream flag enforced.
• P2 — Jobs + OCR (→ M3): Job API + fan-out groups, PaddleOCR/Azure DI adapters, consolidated poll client, queue-position UX, classifier SDK.
• P3 — Control plane (→ M4): node agent, health monitor, compose driver, autoscaler (dry-run → enforce), advisor, full dashboard + console.
• P4 — Cluster (B200 arrival): k8s/llm-d decision, second driver, 397B on-prem (retiring external_dev synthesis), HA hardening. Engines notice nothing.

9. Open decisions

1. Cluster platform (P4): Kubernetes + llm-d/KServe vs compose-driver scale-out — deferred behind the driver interface.
2. Body encryption at rest (pgcrypto) — decide per-engine at P1 store build.
3. Gateway auth hardening — static per-engine bearer tokens v1; SSO-integrated service identity later (aligns with roadmap "AHU Auth Integration", currently 40%).
4. Observatory dashboard branding/UX — design pass with the ministry's UI/UX phase for "Dashboard Performa AI".