Deployment Architecture & Standards — AHU AI Platform

Date: 2026-07-09
What this is: the foundational decisions that have to be settled before the first service gets spun up on the 6 production hosts, written so the MVP build and the eventual HA build fall out of the same structure — no rewrite when HA arrives. Covers the runtime substrate ("docker vs …"), per-service HA strategy, cross-host networking, image distribution, and the container/host hygiene standards that make it actually reliable.
Ground rules it respects: the platform's CONVENTIONS contract (gateway is the sole model egress; Docker/Compose today, k8s deferred behind the placement-driver interface), the migration plan, and the network map.

The one decision that gates everything: orchestration substrate

Recommendation: Docker Compose per host now, HA via external primitives later, Kubernetes explicitly deferred (not never — just not yet, and not for this scale). This isn't the low-ambition choice; it's the right-sized one, and the existing platform + topology doc already lean this way. Reasoning, honestly:

Option	What it buys	What it costs	Fit here
Compose + external HA primitives (keepalived VIP, HAProxy, Postgres streaming replication, Redis Sentinel)	HA without a new platform layer; every service already ships as compose; matches what the topology doc already specified (active-active GW behind a VIP, controller active-standby via PG advisory lock, Redis Sentinel)	HA wiring is per-service and manual — you assemble it rather than declaring it	Best fit. 6 hosts, one instance per zone, small team. The HA design already exists on paper; this just executes it.
Docker Swarm	Multi-node orchestration, overlay net, rolling updates, secrets — a fraction of k8s complexity, built into Docker	Declining mindshare; overlay networking would want to span zones (fights the segmentation model, see §3)	Viable if you want orchestrated HA with less manual wiring — but the overlay-vs-zones tension makes it a worse fit than it looks.
Kubernetes	The "SOTA" answer; declarative HA, self-healing, huge ecosystem	Enormous operational tax — control plane, etcd, CNI, ingress, cert-manager, an actual platform team. At 6 single-instance hosts it's mostly overhead protecting against problems you don't have yet	Wrong tool at this scale. CONVENTIONS already says "k8s deferred behind the placement-driver interface" — that call still holds. Revisit only if the org commits to a standing platform-engineering capability.

The key insight that makes this safe: HA here does not require an orchestrator. The topology doc already decomposed HA into per-service primitives — stateless services scale behind a VIP/LB, stateful ones use native replication. None of that needs Swarm or k8s. So "future-proof for HA" means structuring the services correctly now (see §2), not adopting a heavyweight substrate on day one to avoid a migration later. The migration later is a swap of front-door primitives, not a rewrite.

What "future-proof" concretely requires of every service we deploy now (so the HA swap is painless):
- Stateless services must be genuinely stateless (no local session/file state that a second replica wouldn't see) — the chatbot's move to a shared session store, the gateway holding no per-request state, etc.
- Every service binds its config/secrets from outside the image (env + mounted files), never baked in — so a 2nd replica is byte-identical.
- Every service already exposes /healthz + graceful drain (CONVENTIONS §7 mandates this) — the LB/VIP needs it to route around a draining instance.
- Databases are addressed by a name/VIP, not a hardcoded primary IP — so failover to a standby is a DNS/VIP change, not an app redeploy.

Per-service HA strategy (the matrix that drives everything)

Split every service by stateful vs. stateless, because the HA mechanism is completely different:

Service	State?	MVP (now)	HA path (later)
Gateway (ahu-gpu-manager)	Stateless	1 instance, ahu-ctrl-01	N behind an internal VIP (keepalived) — active-active, already designed
Controller	Stateless-ish (leader)	not built	active-standby via PG advisory lock (already specified)
Redis	Stateful	1 instance	Sentinel or managed; maxmemory bounded + stream trimming
Observatory API / dashboard	Stateless	1 each, ahu-gov-01	2× behind VIP — trivial, they're read-only over the DB
Observatory ingester	Stateful-ish (stream cursor)	1 instance	stays 1 until the per-event advisory lock ships — known backlog item, a 2nd replica is unsafe today (documented in the request doc)
TimescaleDB	Stateful (crown jewels)	1 instance	streaming primary→standby replication; primary feeds the backup target
Keep	Stateful (SQLite→Postgres)	1 instance	Postgres-backed, then standard DB-HA; app tier is stateless
Chatbot web/agents	Stateless if sessions externalized	1 each	2× behind LB/VIP — depends on externalizing session state now
RAG / pgvector	Stateful (vectors)	1 instance	pgvector replication + N stateless RAG readers
OCR (tidyup) + its Postgres	Mixed	1 instance	OCR workers → N behind a queue (already the design); Postgres replicated

The one thing to get right now to avoid rework: anything marked "stateless if…" (chatbot sessions especially) needs its state externalized before it's deployed, not retrofitted. That's the single highest-leverage future-proofing action, and it's app-side work (a chatbot-repo prompt), not infra.

Cross-host networking — the SOTA choice here is not a service mesh

Instinct says "multi-host → overlay network / service mesh (Cilium, Istio, Consul)." That instinct is wrong for this platform, and it's important to say why: the entire security model is that the gateway is the only cross-zone path and the firewall is default-deny between zones. An overlay network that lets any service resolve any other service by name across hosts would quietly dissolve the zone segmentation the whole design is built on. SOTA networking tech here would be actively harmful.

So:
- Within a host: Docker Compose networks (service-name DNS) — fine, unchanged.
- Across zones: stays explicit and firewall-gated, through the gateway seam. No overlay spanning zones. The few allowed cross-zone links (DMZ→gateway, engines→gateway, gov→Redis, Prometheus→exporters) are named firewall rules, and that's a feature.
- Encrypt those allowed cross-host links with WireGuard point-to-point. This is the SOTA move that actually fits: it directly closes the "internal traffic is plaintext HTTP" finding (PII on the wire between hosts) and is lighter than mTLS-everywhere or a mesh. WireGuard tunnels only between the specific host-pairs the firewall already allows — encryption follows the same narrow paths, no broad overlay. (If per-service identity is wanted later, mTLS via an internal CA is the follow-up, as the migration audit's §7-B prompt already scoped — WireGuard first because it's less app-invasive.)

Image distribution — where the current approach breaks, and what replaces it

Current: build locally → docker save tarball → scp → docker load. Keep it for MVP — it's push-based over the admin SSH path, which fits the locked-down egress (a registry pull would need every host, including DMZ, to reach out — fighting the egress lock).

Where it breaks: the moment you go multi-instance (HA), tarball-per-host stops scaling — you'd be scp-ing the same image to N hosts by hand. The fix at that point is an internal container registry (Harbor is the on-prem SOTA choice — image scanning + signing + RBAC built in), placed on the control plane or a small dedicated host, reachable only internally. DMZ still can't pull from the internet, but it can pull from the internal registry over one new named firewall path. Don't build this yet — it's the thing that changes when HA lands, flagged now so the compose files are written registry-ready (image names already fully-qualified, not :latest local tags).

Container & host hygiene standards — the "make it actually work well" checklist

These are the unglamorous SOTA basics that separate "it runs" from "it runs reliably in production." Apply to every service from the first deploy:

1. Log rotation at the Docker daemon level — do this on all 6 hosts before deploying anything. Docker's default json-file driver is unbounded — it will fill the disk, which is exactly the failure that's hit the GPU host repeatedly. Set /etc/docker/daemon.json with log-driver: json-file + max-size: 10m + max-file: 3 (or ship logs to journald). This is a concrete, do-it-now action with real history behind it.
2. Resource limits per container (mem_limit, cpus, or compose deploy.resources) — prevents one container OOM-ing or CPU-starving its neighbors on a shared host (the internal-engines and governance hosts co-locate several services). Keep's runbook already flagged this for itself; make it a standard.
3. Health + readiness on everything, wired into the deploy script's gate — CONVENTIONS already mandates /healthz; the deploy must actually wait on it and roll back if it doesn't come up.
4. Restart policy unless-stopped (already used) — survives host reboots without auto-starting things you deliberately stopped.
5. Pin images by digest, not :latest — reproducibility; a rebuild can't silently drift. (Keep's backlog already caught its own :latest usage.)
6. Image scanning in the build step (Trivy/Grype) — catches known CVEs before an image ships, cheap to add to build-and-ship.
7. Graceful drain honored on stop — compose stop_grace_period set to cover the app's drain window (the gateway already sets 40s, observatory 25s — make it deliberate per service, not defaulted to Docker's 10s which SIGKILLs mid-drain).

Config & secrets — reaffirming the already-decided path

SOPS+age now (encrypted secrets in git, decrypted at deploy time — no plaintext .env, no new running service), self-hosted Infisical later if rotation/RBAC/audit-of-secret-access is wanted. Every service's compose pulls secrets from the decrypted file, never inline. This is settled from the migration audit — restated here because "don't deploy with hardcoded creds" is a hygiene standard, not just a per-repo cleanup.

Concrete bring-up order (what "spinning up" means, first)

Tied to the migration order, with the dependency reality made explicit:

0. Host prep on all 6 (do once, before any service): the daemon.json log-rotation from §5.1, confirm resource-limit conventions, lay down the SOPS+age tooling + the WireGuard links for the paths we'll use. This is the "foundation" pass — no app containers yet.
1. Control plane first (ahu-ctrl-01: gateway + Redis). Dependency-wise this should arguably go before governance even though the migration audit listed governance first "for lowest risk" — because Observatory consumes Redis, and standing Observatory up first means temporarily pointing it at the old GPU-host Redis and re-pointing later. Cleaner to bring the control plane up first. (This ordering nuance is worth your call — see the question below.)
2. Governance (ahu-gov-01: Observatory + TimescaleDB + dashboard + Keep + the Prometheus/Grafana stack).
3. Internal engines (ahu-int-01: internal chatbot + RAG + OCR-tidyup).
4. Public (ahu-dmz-01) — last, and still gated on the LB/WAF and synthesis-tier decisions from the audits.

Each service comes up behind a health gate, with a tagged-image rollback ready, one at a time, verified before the next — same discipline as the firewall pass earlier (fresh check after each step, no batch-and-pray).

What I'd want confirmed before touching a server

The substrate recommendation (Compose + external HA primitives, k8s deferred) is well-grounded in what already exists — but it's the one call that changes how every compose file gets written, so it's worth an explicit yes before I start. The bring-up-order nuance (control-plane-first vs the audit's governance-first) is the secondary one.