AHU AI Platform — Server Topology & Recommended Specs

Status: Approved direction (2026-07-06). Best-practice deployment topology for the AHU AI platform, for a devops server request. Supersedes the "everything on the GPU box (ai-ahu)" bootstrap layout.

Decision: separate by TRUST ZONE + STATEFULNESS, not by service count

The right axis is not "2 boxes vs 3" — it is which trust boundaries must never share a host. Five zones, each boundary a real security / blast-radius line (not fragmentation for its own sake — do NOT go finer, e.g. one-server-per-service, which multiplies ops with no isolation gain):

GPU zone (current)   : 2× H100 NVL 96GB · vLLM (35B-A3B) + in-process tenants (classifier, gpu-server)
                        + gpu-manager NODE-AGENT — right-sized for THIS job, not for 397B (see below)
GPU cluster (future)  : 16× B200 — SEPARATE new hardware, NOT an upgrade to the current box (own zone,
                        own procurement track — see "Future: GPU cluster" below)
Control plane : gpu-manager GATEWAY + CONTROLLER · Redis (slot leases + ahu.ai.audit stream)
Governance    : observatory ingester + query API · TimescaleDB (AI audit store) · dashboard
Internal eng. : ahu-ai-ocr · chatbot INTERNAL surface · RAG/vector store
DMZ / public  : chatbot PUBLIC surface — internet-facing, WAF + LB, egress locked to Gateway only
DWH (future)  : Enterprise Data Warehouse — see "Future: Enterprise Data Warehouse" below

Why these boundaries (non-negotiables)

1. Public chatbot lives in a DMZ, split from the internal surface. A "public + internal" single deployment straddles the riskiest boundary. Public surface behind a WAF, with egress locked so it can only reach the Gateway — nothing else. A compromised public bot then cannot touch the audit DB or lateral into engines.
2. The audit store is the crown jewels (full-capture request/response PII bodies, government mandate) — it must never share a host with an internet-facing service. Internal-only, governance plane.
3. The Gateway is the sole model egress and holds upstream keys + enforces on-prem. Internal, latency-optimised, off the public plane. Model servers stay loopback-bound on the GPU host.
4. The node-agent must run ON the GPU host (manages local GPU processes / DCGM) — so ahu-gpu-manager is inherently split: gateway+controller on the control plane, node-agent on the GPU box.
5. OCR is CPU/RAM-heavy and batchy — keep it off the latency-sensitive Gateway box.
6. Redis (the seam between Gateway audit-producer and observatory ingester-consumer, plus slot leases) is internal-only, control plane.

Recommended specs

All non-GPU hosts: Ubuntu 22.04/24.04 LTS, hardened baseline, on a private VLAN, NTP-synced (mandatory — the audit hash-chain ordering and the queued-vs-processing timers depend on consistent clocks). Docker/Compose runtime today (k8s deferred behind the placement-driver interface). These are starting points with headroom — right-size after load testing.

Why the current GPU host (Server 10) needs a spec bump — and what it does NOT enable

397B-A17B cannot run in production on this box at any host-RAM size — this is a hard VRAM ceiling, not a host-sizing problem. 397B params in FP8 (1 byte/param) ≈ 397 GB of weights alone, before KV cache/activations (realistically 450GB+ needed). The box has 2× H100 NVL 96GB = 192 GB total VRAM — under half of what's required; even INT4 (~198GB) barely fits weights with zero room for KV cache. MoE ("A17B" active params/token) reduces compute per token, not the memory footprint — every expert must still be resident somewhere. The 397B model's only production home is the future B200 cluster below.

The recommended bump (32 vCPU / 256 GB / 2 TB NVMe, up from 12c/122GB) is justified by what's actually running on this box today and near-term, independent of 397B:
1. Multiple co-resident GPU tenants already share this box — ahu-classifier and ahu-gpu-server are loopback-bound here alongside vLLM, each with its own model in host RAM and its own CPU-bound preprocessing (image decode/resize). 12 cores is thin once vLLM's scheduler, the classifier, gpu-server, DCGM, and node-agent all compete for it.
2. Disk has repeatedly hit crisis levels on this exact box — 88–94% full multiple times this engagement; docker prune only reclaimed 15GB against ~ahu-ai (107GB) + ~ahu-ai-training (41GB). That's today's footprint, before model-version history (config_version tracking) or quantization work (needs ~2× space temporarily). 2TB+ NVMe converts a recurring fire-drill into headroom.
3. OS page-cache for fast model loads — the 35B-A3B model (~70GB FP16) fits VRAM comfortably, but 122GB total RAM shared across OS + vLLM + classifier + gpu-server leaves little to cache the checkpoint, risking slow restarts and real OOM pressure once OCR/chatbot/classifier traffic converges through the gateway.
4. Optional, explicitly non-production: functionally smoke-testing 397B's integration (prompt templates, gateway response-shape handling) via CPU-offload serving (vLLM CPU offload / DeepSpeed ZeRO-Infinity) is technically possible, holding most of the ~400GB of experts not resident in VRAM in host RAM — but that needs closer to 512 GB, not 256GB, for any real margin, and throughput is PCIe-bound (an order of magnitude slower). Treat as optional dev tooling, not a spec driver for the baseline box.

Future: GPU cluster (16× B200) — separate hardware, not this box

Confirmed: this is not an upgrade path for Server 10. B200 nodes ship 8 GPUs/node, so 16× B200 means ≥2 physical nodes, needing NVLink within a node and InfiniBand/RoCE between nodes, and drawing ~10–14kW per 8-GPU node — most orgs procure this as a vendor reference system (NVIDIA DGX/HGX B200 or an OEM equivalent) rather than spec it from parts, and it needs data-center-grade power/cooling most colo racks don't provide by default. Flag to devops as a distinct procurement track (facilities + vendor engagement), not an incremental server order — timeline and cost profile are entirely different from the rest of this table.

Future: Enterprise Data Warehouse (DWH)

From the platform readiness plan (Engine Readiness sheet), a fourth engine — Enterprise Data Engine (DWH) — is planned but not yet built: Analytical Data Warehouse (warehouse engine, operational metastore, domain schema, schema migrations, backup/DR, lakehouse storage), ETL/Ingestion (source connectors AHU→DWH, CDC/SCD transform, load scheduler, data quality), a Domain API layer, and its own Data Governance & Observability (audit logging, access audit trail). The Knowledge Intelligence Engine's RAG pipeline depends on it ("Live Data Sync from DWH", "Knowledge Refresh Scheduler").

This gets its own zone, separate from the AI governance plane — different data domain (core AHU registry/business data, not AI-call audit), a much larger growth profile (lakehouse implies org-wide historical data, likely TBs), and arguably higher sensitivity than the AI audit store (actual legal-entity/registry records, not LLM call metadata). Marked Phase 2 / not part of the current request — say the word if you want it requested alongside the others instead.

Network placement: the most isolated zone in the platform — no path from DMZ/public at all; ETL pull from core AHU registry systems should be one-directional/read-only; Domain API reachable only by the Internal-engines zone (and later Governance/PIE), never public.

Two request tiers (current scope — excludes future GPU cluster and DWH above)

• MVP (single instance per zone): ~5 hosts — 1 control, 1 governance, 1 internal-engine, 1 DMZ, + existing GPU. Fastest to stand up; no redundancy.
• Recommended (HA): ~8–10 hosts — 2 control, 1 API/dashboard + DB primary + DB standby, 2 internal-engine, 2 DMZ, + LB/backup. Matches the platform's active-active/standby HA design (no SPOF — the redundancy requirement from the original design).

Migration from today

Everything (gateway, observatory, DB, Redis, dashboard) currently runs on the GPU box ai-ahu, with the chatbot flipped through the gateway. Recommended sequence:
1. Public chatbot → DMZ first (biggest risk reduction per unit effort).
2. Observatory + TimescaleDB → governance plane (also frees the GPU box's disk, which has been ~88% full).
3. Gateway + Redis → control plane, leaving only vLLM + node-agent on the GPU box.
4. Stand up HA pairs + LB/VIP as the platform goes to production.

See also: docs/CONVENTIONS.md (contract), docs/deploy/2026-07-04-gpu-host-survey-and-deploy.md (current host survey).