think
16px
820px
← all files

Production Migration Readiness Audit — AHU AI Platform

Date: 2026-07-08
Scope: what it takes to move the platform off the single GPU box (ai-ahu, 192.168.83.20) onto the 6 hardened production hosts (192.168.82.120–125), grounded in the actual repos as they exist today.
Method: read-only survey of ahu-gpu-manager, ahu-ai-chatbot, ahu-ocr-akta-notaris, ahu-doc-classifier, ahu-ai-observatory, ahu-observatory-dashboard (deploy scripts, compose files, CI, CLAUDE.md conventions) — no engine repo was modified, per the hands-off rule.

TL;DR

The good news: there's already a working deploy discipline (build → tarball → ship-over-SSH → compose-up, with health checks and a documented cutover/rollback template) — you're not starting from zero. The bad news: that discipline was built for "everything on one box." Splitting into 5 zones surfaces several things that actively break, plus a few production stances that need enforcing, not just tidying:

  1. ~~Public chatbot calls RAG directly, bypassing the Gateway~~ — RESOLVED end-to-end (2026-07-08): the Gateway now proxies RAG search, and the chatbot is already flipped to use it. Both cross-zone RAG paths are closed — /search proxied + live, /sessions deleted (was dead code). This was the #1 migration blocker; it's fully cleared (§2.1).
  2. Several services address each other by Docker network name (ai-ahu-rag:8110, ahu-platform_default), which only resolves when both containers share one Docker host. Post-split, every cross-zone call needs a real IP:port.
  3. Legacy OCR will not go to production — only the gateway-wired, security-hardened tidyup branch ships (§2.4).
  4. External LLM calls are banned in production, full stop — and the chatbot's current "big model" (Qwen-397B via Alibaba Cloud DashScope) has no on-prem home yet. That's not a secret-rotation task, it's a product-capability gap that needs a decision before go-live (§2.6).
  5. Hardcoded dev-grade secrets are checked into several compose files — fixing this plus standing up real secrets management is now in scope, not deferred (§2.5).
  6. Correction from the last pass: the nginx box currently fronting staging (103.30.246.154) is Efran's own private server, borrowed for this project — not part of the production stack. The LB/WAF question is genuinely open again (§4), not cheaply solved.

Below: what maps where, what breaks and needs a decision, a hardening checklist, migration order, and a pipeline recommendation.

1. What actually moves where (repo → host)

Not everything migrates. GPU-resident tenants stay on the GPU box; only the CPU-side platform/engine services move.

Service (repo) Today Target (new topology) Notes
Gateway + Redis (ahu-gpu-manager) ai-ahu :8200 ahu-ctrl-01 (192.168.82.120) Controller/node-agent aren't built yet — nothing to move for those.
Observatory ingester + query API (ahu-ai-observatory) + TimescaleDB ai-ahu :8300 ahu-gov-01 (192.168.82.121)
Observatory dashboard (ahu-observatory-dashboard) ai-ahu :8320 ahu-gov-01 Port doesn't matter (Efran's call) — kept at 8320, firewall already adjusted to match rather than forcing a repo change.
Internal chatbot (ahu-ai-chatbot — internal-web + internal-agent) + RAG (ai-ahu-rag) + pgvector ai-ahu :3510 / :8110 ahu-int-01 (192.168.82.122) Bundled per the MVP topology (OCR + internal chatbot + RAG on one host).
OCR backend + its Postgres (ahu-ocr-akta-notaris) ai-ahu :3001 ahu-int-01 tidyup branch only — legacy branch will not be deployed to production (§2.4).
Public chatbot (ahu-ai-chatbot — public-web + public-agent) ai-ahu :3500 ahu-dmz-01 (192.168.82.123) RAG cross-zone paths fully resolved (§2.1 ✅ — both gateway and chatbot sides done). Remaining pre-cutover deps: synthesis-tier decision (§2.6) and a real LB/WAF (§4).
Doc classifier (ahu-doc-classifier), vLLM, TEI embeddings, PaddleOCR, Azure DI Layout, cleanup-LLM, gpu-server, node-agent (once built) ai-ahu stays on ai-ahu These are the GPU-resident tenants — nothing here migrates to the 6 new hosts.
DWH compute/storage — ahu-dwh-01 / ahu-dwh-store-01 Engine not built yet — hosts sit idle for now.
Edge TLS/reverse proxy nginx on 103.30.246.154 not part of the production stack — Efran's private server, borrowed short-term. Real prod edge is still an open gap (§4).

2. Architectural blockers — need a decision, not just a config change

2.1 Public chatbot → RAG direct call — RESOLVED end-to-end (2026-07-08)

Status: done, both sides, verified. What shipped:
- /search (the live RAG call) is now Gateway-proxied. New upstream type rag-http + a sync-facade adapter in the Gateway forwards POST /search verbatim, audited as operation=search (gateway tag rag-proxy-v1; search added to the CONVENTIONS.md §5 operation enum). Wired live (ai-ahu-rag → http://192.168.83.20:8110 behind the gateway) and the public chatbot flipped to it (RAG_SEARCH_URL → the gateway at :8200, chatbot commits cf49dbb+bed6b29). Verified end-to-end: a real public "Tanya" query's operation=search event and its synthesis chat event share one trace_id — i.e. the search now rides the same audited path as every other model call, which it never did before.
- /sessions (the other cross-zone RAG path) was closed by deletion, not proxied (chatbot commit f42e355). On investigation it was dead: the RAG /sessions endpoints returned stub [], chat history had already moved to public-web's local ThreadStore, and the UI gated the path to logged-in accounts only. So rather than build a second gateway proxy for a dead call, the passthrough was removed engine-side.

Net: no direct DMZ→Internal-engines path remains for RAG. This fully clears the #1 migration blocker — the public chatbot's cutover is no longer gated on this at all. The remaining public-chatbot cutover dependencies are unrelated (§2.6 synthesis tier, §4 LB/WAF).

The original decision reasoning is kept below for the record, since it's the rationale behind what got built.

ahu-ai-chatbot's own handoff diagram shows public-web -->|/search| rag as a direct call — separate from and in addition to the Gateway-routed calls. Today this works because public-web and ai-ahu-rag are containers on the same Docker host, reachable by container name. Post-split, public-web moves to ahu-dmz-01 and ai-ahu-rag moves to ahu-int-01 — different hosts, different zones — so the Docker-network address stops resolving and the call becomes a new DMZ→Internal-engines path the design doesn't otherwise allow.

Decision (implemented): proxy RAG search through the Gateway, same as every other model/embedding call. Weighing the three options that were on the table:

  • Gateway proxy (chosen). Keeps the DMZ egress rule genuinely absolute — one path out, no exceptions, no asterisks — which matters because that rule is what makes a DMZ compromise contained instead of a foothold into Internal-engines/DWH. The cost is real but bounded: the Gateway needs a new proxied route for RAG search (it already proxies LLM/embedding calls, so this is the same shape of work, not a new capability class), and RAG search traffic now also lands in the audit trail — arguably a feature, since public-surface search queries currently have no audit visibility at all.
  • Narrow firewall exception (rejected, but not unreasonable — this is the "weigh it" part). DMZ isolation exists for security, and a well-scoped exception for a genuine operational need isn't automatically wrong — the apt-mirror exception on ahu-dmz-01 earlier this session is exactly that kind of call, made deliberately. The difference here is what's being exposed: apt-mirror is host-plumbing (patching) with no user data in the path; a firewall hole straight into the Internal-engines zone exposes RAG's search surface (and, transitively, whatever's reachable from that host) directly to a compromised public-facing box. That's a bigger asymmetry between "convenience gained" and "blast radius if this goes wrong" than the apt case, which is why it loses out here — not because exceptions are categorically banned.
  • DMZ-local RAG replica (rejected). Avoids the cross-zone call entirely, but means a second RAG deployment, a sync mechanism, and a second place for the knowledge base to drift out of date — more ongoing operational cost than the Gateway-proxy route for the same outcome.

This was a Gateway feature addition, now built — see the ahu-gpu-manager integration prompt in §7 (marked done).

2.2 Observatory ingester → Redis: Docker-network assumption, but the firewall's already right

ahu-ai-observatory's compose file joins ahu-platform_default (the Gateway's own compose network) specifically so the ingester can reach Redis by container name — comment in the file says as much. Post-split, Observatory (ahu-gov-01) and Redis (ahu-ctrl-01) are on different hosts, so that network trick stops working.

The difference from §2.1: this cross-zone path is already in the request doc's matrix ("Governance host(s) → Control plane (Redis): Redis port"), and the firewall rule for it is already live (ahu-ctrl-01 allows 6379/tcp from ahu-gov-01, applied in the earlier hardening pass). What's left is just config: swap the ingester's Redis address from the Docker network name to 192.168.82.120:6379, and drop the ahu-platform_default external-network dependency from the observatory compose file.

2.3 Dashboard port (8320 vs. 8310) — not worth fighting over

Already flagged in the platform's own handoff notes: the dashboard's compose file deploys on 8320, while CONVENTIONS.md §1 says 8310. Efran's call: doesn't matter which, as long as it's consistent. Resolved the cheap way — the firewall rule on ahu-gov-01 was updated to match the dashboard's existing 8320 default rather than making the repo change the contract requires a repo edit for. No action needed in ahu-observatory-dashboard.

2.4 OCR: legacy branch will not go to production — tidyup is the only path

The compose file for ahu-ocr-akta-notaris's legacy branch hardcodes http://192.168.83.20:8108 (PaddleOCR) and http://192.168.83.20:8003/v1 (cleanup LLM) directly, bypasses the Gateway entirely (no audit trail), and per the repo's own CLAUDE.md: "No authentication — all endpoints are open, no roles/sessions." None of that is production-shaped, and it's now settled that legacy simply doesn't ship.

Decided: only the tidyup branch/worktree deploys to ahu-int-01. It already has the gateway client wired (currently dormant pending env flip) and a security layer (currently shadow-mode per the platform handoff) — flipping the gateway env and taking the security layer out of shadow mode are both part of getting this branch production-ready, not optional extras.

2.5 Hardcoded dev-grade secrets — fix all of them, plus stand up real secrets management

Found checked into compose files:
- ahu-ocr-akta-notaris/docker-compose.yml: POSTGRES_PASSWORD: ahu_dev (legacy branch — moot once §2.4 lands, but the tidyup branch needs its own check for the same pattern).
- ahu-ai-observatory/deploy/compose.yaml: POSTGRES_PASSWORD: observatory — literal, checked in.
- ahu-gpu-manager/deploy/compose.yaml: DASHSCOPE_API_KEY — an external_dev-class upstream key that has no place in a production config once the external-LLM ban (§2.6) is enforced.

Every one of these needs a freshly generated, non-checked-in secret before go-live — no exceptions, matching the "no external LLM calls in prod, full stop" stance.

On the vault question — recommending a two-tier approach rather than one leap:

  • Now, as part of this migration: SOPS + age — secrets get encrypted at rest and committed to git as ciphertext, decrypted to a real .env file only at deploy time on the target host (or right before the scp step). This slots directly into the existing build-and-ship scripts with one extra decrypt step, adds no new running service, no new attack surface, and nothing to patch or back up beyond the age key itself. Matches the platform's general on-prem/self-hosted/low-machinery bias better than reaching for a hosted or heavier option first.
  • Later, if the team wants rotation, access control, or an audit trail of who read which secret when: a self-hosted, open-source secrets manager — Infisical is the best fit (Docker-deployable, so it can run on ahu-ctrl-01 alongside the other control-plane services, has a real UI and RBAC). Would replace the SOPS-encrypted files with a runtime fetch at container start. Worth doing once there's more than one or two people managing secrets day-to-day; probably overkill to stand up before that.

Recommend starting with SOPS+age now (low effort, unblocks the "fix all hardcoded secrets" ask immediately) and revisiting Infisical once the team/secret-count actually justifies the extra moving part.

2.6 External LLM calls are banned in production — the chatbot's synthesis tier has no on-prem home yet (critical)

Confirmed: no external LLM/AI calls in production, no exceptions. That's a clean, enforceable rule for most of the platform — but the chatbot's synthesis tier currently runs on Alibaba Cloud DashScope (Qwen-397B), and per the platform's own topology math, 397B cannot run on the current GPU box at any host-RAM size — it's a hard VRAM ceiling (397B needs 450GB+ even at FP8; the box has 192GB total across both H100s). The only place 397B was ever going to run on-prem is the future B200 cluster, which doesn't exist yet.

So this isn't "swap a secret and redeploy" — the synthesis tier, as currently built, has no production-legal backend. Before the public/internal chatbot can go to production with the ban enforced, one of these needs deciding:
- Ship production with a smaller on-prem model doing synthesis (e.g., the existing 35B-A3B vLLM model) — likely a real quality regression versus staging's 397B output, but on-prem and legal today.
- Hold the affected chatbot surfaces out of the production go-live until the B200 cluster lands.
- Some other on-prem-capable substitute the product side is willing to accept.

Recommend surfacing this to whoever owns chatbot product quality before scheduling a go-live date — it's a capability/quality decision, not something to resolve silently as an infra detail.

3. Production readiness checklist

Things to have ready before (or as part of) cutting each service over, roughly in the order they bite:

  • [ ] Fresh secrets per production host — new DB passwords, new NEXTAUTH_SECRET, new JWT signing keys. Don't copy staging's .env files verbatim; regenerate, and encrypt at rest with SOPS+age per §2.5 rather than plain git-ignored files.
  • [ ] Gateway production config — allow_external_upstreams=false, DashScope/Alibaba entry fully removed from the upstream registry (contract §4.4) — and a resolved answer to §2.6 (synthesis tier has no on-prem model yet) before the chatbot surfaces depending on it go live.
  • [ ] Database provisioning on target hosts — TimescaleDB on ahu-gov-01, Postgres+pgvector on ahu-int-01 for both chatbot and OCR. Run real migrations (Prisma for OCR, whatever the chatbot/observatory use) rather than copying staging data — decide explicitly whether this is a fresh start or a data migration.
  • [ ] TLS + edge — needs real production infrastructure now that 103.30.246.154 is confirmed out of scope (§4) — not just a cert on a borrowed box.
  • [ ] Health checks wired to something that reads them — every platform service already exposes /healthz//api/health per the contract; the disk/monitoring pass from this session gives you node_exporter on all 6 hosts, but nothing is scraping it yet. Stand up Prometheus (or point at whatever the team already runs) before relying on "monitoring" as done.
  • [ ] Backup target — still open from the earlier hardening pass. Non-negotiable before real production data lands in TimescaleDB (audit/PII data) or the OCR/chatbot Postgres instances.
  • [ ] Rollback plan per service — the chatbot already has one (infra/deploy/rollback.sh + tagged images kept on the target host); extend the same pattern (keep N-1 image, a documented one-command revert) to OCR, observatory, and the gateway before calling any of them production.
  • [ ] Classifier audit/heartbeat — still a known gap (works, but doesn't emit audit events per the SDK pattern in contract §4.5). Doesn't block migration (classifier isn't moving), but it does mean the audit trail has a hole in production from day one if left as-is.
  • [ ] Smoke test per cutover — the chatbot's own HERMES-CUTOVER-BRIEF.md (nginx swap → curl health → curl a real query → confirm streaming → rollback instructions) is a good template; write one of these per service before its cutover, not after something breaks.

4. The LB/WAF question is genuinely open (correction from the last pass)

The previous version of this audit treated 103.30.246.154's nginx as a cheap, already-working answer to the LB/WAF gap. That was wrong — it's Efran's own private server, borrowed for this project short-term, not project-owned infrastructure. It cannot be the production edge: if the borrowing arrangement ends, production goes down with it, and there's no ownership/access guarantee appropriate for a production entry point.

This reopens item 7 from the original delivery audit exactly as it stood there: either a managed cloud LB/WAF (the request doc's stated preference) or a dedicated on-prem host fronting ahu-dmz-01. Needs a real decision and, most likely, new infrastructure — not something to resolve by repointing an existing config. Recommend treating this as a blocker for the public chatbot's production cutover specifically (internal-only services don't need it).

5. Recommended migration order

The topology doc's own sequence still holds, now with real service names attached and the Gateway's new RAG-proxy work sequenced in:

  1. ~~Gateway RAG-proxy endpoint (§2.1)~~ — ✅ DONE (2026-07-08). The endpoint is built; the step that gated everything else is cleared.
  2. Observatory + TimescaleDB + dashboard + Keep → ahu-gov-01 — lowest risk of the batch (no public exposure), fixes the Docker-network→real-IP Redis address (§2.2) as part of the move. Keep (currently on ai-ahu, see §A.1) moves with the Observatory in this same step, not separately — they're wired together via the EWS-notifier webhook — and the external-heartbeat caveat from §A.1 must land in this step, not after.
  3. Gateway + Redis → ahu-ctrl-01 — do this alongside or right after step 2, since Observatory can't reach a Redis that hasn't moved yet.
  4. Internal chatbot + RAG + OCR (tidyup branch only, §2.4) → ahu-int-01 — bundle these since they're already meant to share a host.
  5. Public chatbot → ahu-dmz-01 — the RAG cross-zone blocker (step 1) is fully resolved on both sides, so this is unblocked on that axis; remaining pre-cutover deps are the §2.6 synthesis-tier decision and a real LB/WAF (§4) in front of it.
  6. DWH hosts stay idle until that engine exists.
  7. Once all four zones are live and stable, revisit the HA tier (2× control plane, DB standby, 2× DMZ/internal) — the MVP tier you have now is explicitly "no redundancy," worth knowing going into production, not after an incident.

6. Should you build a pipeline?

Recommendation: extend what you have, don't replace it — and don't fully automate deploys yet.

What already works and is worth keeping: build-locally → tarball → scp over SSH → docker load → docker compose up -d, with a tagged-image rollback path and a documented cutover/smoke-test template. It's not a registry-based pipeline, but it has a real advantage for this specific network: it's push-based over the admin SSH path, which is exactly what's still open through the new firewall — a registry-pull-based pipeline would need ahu-dmz-01 (and every other host) to have outbound access to wherever images are hosted, which conflicts with the same gateway-only egress rule from §2.1. Don't reach for a container registry just because it's the default CI/CD pattern — it fits this network worse than what's already there.

What to change:
- Generalize the target host — today's scripts hardcode SERVER2=efran@192.168.83.20. Make it a per-service variable (CTRL_HOST, GOV_HOST, INT_HOST, DMZ_HOST) so the same script pattern works across all 4 active zones instead of one box.
- Add a secrets-decrypt step — once SOPS+age (§2.5) is in place, build-and-ship.sh/deploy-staging.sh-equivalents decrypt the target host's secrets file just before the scp step, instead of expecting a plaintext .env to already exist locally.
- Keep deploys human-triggered for now. These are single-instance hosts with no redundancy (§5.7) — an automatic deploy-on-merge that breaks something takes down production with no failover. Keep the existing model (a person runs the build-and-ship + deploy scripts, reviews the smoke test, decides to proceed) until the HA tier exists.
- CI stays test/typecheck/build-only (as it is today) — that part's fine as a merge gate on GitHub-hosted runners.
- Real deploy automation is a later step, and needs a self-hosted runner. GitHub-hosted runners (ubuntu-latest) can't reach 192.168.82.x at all — it's firewalled to bastion/VPN only, no public path. If/when you want workflow_dispatch-triggered automated deploys, that requires a self-hosted GitHub Actions runner living inside the VPN (e.g., on ahu-ctrl-01) so it can actually reach the target hosts. Worth planning for, not worth building before the manual process has proven itself on real production traffic.

7. Per-repo integration prompts (for their own Claude Code sessions)

Per the hands-off rule, none of the following was implemented — each is a self-contained brief for a Claude Code session running inside that repo:

  • ahu-ai-chatbot: (a) parameterize infra/deploy/build-and-ship.sh / deploy-staging.sh to take DMZ_HOST/INT_HOST instead of the single hardcoded SERVER2, splitting the public/internal stacks to their respective new hosts; (b) ✅ DONE (2026-07-08) — public-web now calls RAG search through the gateway (RAG_SEARCH_URL → :8200, commits cf49dbb+bed6b29), and the dead /sessions passthrough was removed (f42e355); (c) resolve §2.6 — the synthesis tier's model backend needs a production-legal answer before the public/internal surfaces depending on it ship; (d) generate fresh production secrets, encrypted via SOPS+age rather than plaintext infra/env/*.env.
  • ahu-ocr-akta-notaris: (a) the tidyup branch/worktree is the only one that deploys to production — legacy does not ship; (b) as part of hardening tidyup for production, flip its gateway env on (MODEL_GATEWAY_URL → ahu-ctrl-01:8200) and take its security layer out of shadow mode; (c) replace the checked-in ahu_dev-style Postgres password with a SOPS+age-managed production secret; (d) confirm the "no auth" gap from the legacy branch doesn't carry into tidyup's production config.
  • ahu-ai-observatory: (a) change the ingester's Redis address from the ahu-platform_default Docker-network name to ahu-ctrl-01's real IP:6379; (b) replace the checked-in observatory/observatory Postgres credentials with a SOPS+age-managed production secret; (c) point at ahu-gov-01 as the deploy target.
  • ahu-observatory-dashboard: no change needed — keeping the existing 8320 port mapping, firewall already matches. Just deploy to ahu-gov-01.
  • ahu-gpu-manager: (a) ✅ DONE (2026-07-08) — Gateway-proxied RAG-search route shipped (rag-http upstream + sync-facade adapter, operation=search audited, tag rag-proxy-v1, search added to CONVENTIONS.md §5 enum); the public chatbot no longer calls ai-ahu-rag directly (§2.1); (b) production gateway.yaml with allow_external_upstreams=false and the DashScope/Alibaba entry fully removed from the upstream registry (§2.6 — coordinate with chatbot on what replaces it for synthesis); (c) deploy target ahu-ctrl-01; (d) secrets (upstream keys, if any remain for on-prem-only upstreams) via SOPS+age, not checked into deploy/compose.yaml.
  • ahu-doc-classifier: no host move needed (stays on the GPU box), but still owes the platform an audit/heartbeat integration per contract §4.5 (SDK pattern) — worth doing before calling the platform's audit trail "production-complete."

Infrastructure/platform prompts (not repo-scoped — cut across hosts or don't belong to any single engine)

These three came out of the enterprise-readiness follow-up (2026-07-08) and are consolidated here per Efran's request, rather than living in a separate doc. None of them has been implemented — each is meant to be handed to whoever (or whichever session) picks up the work.

A. Wire node_exporter to Power BI for real-time monitoring + alerting

Starting point: prometheus-node-exporter is already installed and running on all 6 hosts (192.168.82.120–125, port 9100) from the earlier hardening pass, but nothing scrapes it. The firewall now allows ahu-gov-01 (192.168.82.121) to reach port 9100/tcp on the other 5 hosts (opened 2026-07-08 specifically to unblock this) — a Prometheus server on ahu-gov-01 is the intended scrape point, matching its existing "governance/monitoring" role alongside the Observatory stack.

The architecture gap to solve, explicitly: Power BI has no native Prometheus-scrape connector — it's a BI/reporting tool that pulls from structured sources (databases, REST/OData APIs, Azure services), not something that speaks Prometheus's scrape protocol. So this is a two-hop pipeline, not a direct connection:

  1. Prometheus server (containerized, on ahu-gov-01) scrapes all 6 node_exporter endpoints on a short interval.
  2. A bridge from Prometheus to Power BI, where the real decision is:
    - Power BI push/streaming dataset (best fit for "real-time") — a small poller service queries Prometheus's HTTP API (/api/v1/query) on an interval and POSTs rows to a Power BI streaming/push dataset via the Power BI REST API. Needs a Power BI Pro/Premium workspace + an Azure AD app registration for API auth. This is the option that actually gets live-updating tiles, not just a periodically-refreshed report.
    - Azure Monitor / Log Analytics bridge, if the org already has an Azure tenant (there are existing x056.ahu-azure.val.id references in this project, worth checking what Azure access already exists) — configure Prometheus remote_write into Azure Monitor managed Prometheus, then use Power BI's native Azure Monitor Logs (KQL) connector. Cleaner if the Azure relationship already exists; not worth standing up an Azure tenant just for this.
    - Scheduled-refresh report via Power BI's generic Web/REST connector hitting Prometheus's query API directly — simplest to build, but Power BI Service's minimum refresh cadence isn't sub-minute, so this does not satisfy "real-time" if that's a hard requirement. Flag this tradeoff explicitly rather than silently building the easy option and calling it real-time.
  3. Alerting is a separate concern from the Power BI visualization — Power BI is not a paging/alerting system (its "data alerts" feature is a narrow, visual-level threshold notice, not incident alerting). Superseded by Keep (2026-07-09, see below): the original recommendation here was standalone Prometheus Alertmanager; now that Keep is deployed and already the Observatory's escalation layer, route Prometheus alerts into Keep instead of standing up a separate Alertmanager — Keep ingests Prometheus alerts natively, so this collapses infra-alerting and EWS-alerting onto one escalation brain, one on-call config, one place to ack. Retire the existing disk-alert syslog script in favor of proper Prometheus alert rules feeding Keep (reuse its 70%/85% thresholds as the first rules).

Decided (2026-07-08): Grafana ships in the same stack, on ahu-gov-01. Even though Power BI is the exec-facing reporting layer, Grafana rides along regardless — it's what an engineer actually opens during an incident, and its query/dashboard authoring is what you'd use to build the Prometheus alert rules that feed Keep. Architecturally it belongs next to Prometheus for the same reason the Observatory dashboard already lives on ahu-gov-01: ahu-gov-01 is the platform's monitoring/governance zone by role, and Grafana constantly reads from Prometheus, so co-locating avoids a cross-zone read dependency for something queried continuously. Concretely: one ahu-gpu-manager/deploy/compose.monitoring.yaml brings up Prometheus + Grafana together (no separate Alertmanager needed, per the Keep decision above) as a coupled unit (they're always deployed as a set in practice — splitting them the way chatbot splits public/internal doesn't apply here, since that split exists because public/internal are different trust zones, and these aren't). Needs its own firewall allow from admin/VPN to Grafana's port (3000 by default) on ahu-gov-01, same pattern as the existing dashboard/observatory-API rules there.

Ask whoever picks this up to confirm first: which Power BI licensing tier is available (Pro/Premium needed for streaming datasets) and whether an Azure Monitor/Log Analytics workspace already exists for this project — the right bridge design depends on the answer, and guessing wrong means rebuilding the pipeline.

A.1 Keep (keephq) — self-hosted incident escalation, adopted 2026-07-09, verdict: right call

Decision: adopt, don't build. Keep was evaluated against building on-call/escalation (schedules, escalation policies, ack/silence, dedup, multi-channel routing, a workflow engine) directly into the Observatory dashboard. Verdict: adopting Keep is correct and not a close call — on-call/escalation is a large, well-understood, commodity problem (it's literally what PagerDuty/Opsgenie/Grafana OnCall are); building it in-house would mean spending engineering effort re-implementing a mature category at the expense of the Observatory's actual differentiated value (the hash-chained audit trail + governance views). The seam is clean: Observatory owns detection (the EWS notifier deriving warnings from the audit stream — bespoke), Keep owns escalation (commodity, adopted). Chosen after Grafana OnCall OSS was found archived (Mar 2026), consistent with the platform's self-hosted/on-prem posture.

Current state (per 2026-07-09-keep-deployment.md): deployed on the GPU host ai-ahu (192.168.83.20), VPN-only, 3 services (keep-frontend/keep-backend/keep-websocket-server), SQLite + file secret-manager, AUTH_TYPE=NO_AUTH (top hardening item — wrong for a government incident tool even VPN-only). Observatory's EWS notifier POSTs firing/resolved events to Keep's generic ingestion endpoint; verified end-to-end (test + live warnings, dedup/auto-resolve by fingerprint working).

Migration decision: move to ahu-gov-01 together with the Observatory in migration step 2 (§5), not as a separate step. Role-fit is clean — ahu-gov-01 is already the governance/monitoring zone, and Keep sits naturally beside the Observatory that feeds it and the Grafana/Prometheus stack landing there per A above. Splitting them across zones mid-migration would stretch the observatory → Keep POST across a zone boundary for no reason.

Caveat that must ship with the move, not be deferred: an external heartbeat. If Keep is what pages you, and it lives on the same host as what it watches, a whole-box failure of ahu-gov-01 takes down the Observatory and the pager that would tell you about it. Fix: one heartbeat check from a different host (e.g. ahu-ctrl-01 polling ahu-gov-01's and Keep's /healthz), or a dead-man's-switch alert whose absence, observed from outside the box, is itself the alarm. This is cheap and doesn't require moving Keep off-box — just don't let this migration land without it. (Note this isn't the same situation as the backup-target call earlier — a backup's whole job is surviving the primary's death, so co-location there is self-defeating; Keep's job is routing, so co-location is fine except for this one narrow "who watches the watcher" gap.)

Resource watch-item, not a blocker: post-migration, ahu-gov-01 co-hosts TimescaleDB (audit) and Prometheus's TSDB (§A), plus Keep's own DB once it moves off SQLite (hardening backlog item below). Fine at 16 vCPU/64GB for MVP; two time-series stores competing for RAM/disk as retention grows is the most likely reason to eventually split TimescaleDB onto its own host, which the topology doc already anticipates.

Hardening backlog carried over from the deployment runbook, in priority order: (1) AUTH_TYPE=NO_AUTH → real auth (DB or platform SSO) — top priority, VPN-only isn't sufficient for a government incident tool; (2) SQLite → Postgres for durability (the target host already runs Postgres/Timescale); (3) pin image tags (currently :latest); (4) configure Keep's own outbound notification providers (email/Slack/webhook) — Keep routes, it still needs a channel to actually reach humans; (5) resource limits + periodic backup of Keep's state dir.

B. Internal service-to-service traffic is plaintext HTTP — needs TLS

The problem, concretely: per the migration survey, OCR's calls to PaddleOCR (http://192.168.83.20:8108) and the cleanup LLM (http://192.168.83.20:8003/v1) are plain HTTP, and this pattern likely repeats across other cross-host calls surveyed earlier (Gateway↔upstreams, Observatory↔Redis, RAG↔Gateway once §2.1 lands). Once these cross real network segments instead of staying on one Docker host's loopback, that's notarial-deed/identity-document content in cleartext on the wire between hosts.

Scope — this is both infra and per-repo work:
- Infra piece (do this first, it's a prerequisite for everything else): stand up an internal CA (a self-hosted step-ca instance is the standard lightweight choice, or openssl if something more manual is preferred) so every internal hostname/IP can get a certificate without hitting a public CA or rate limits — public CAs like Let's Encrypt can't issue certs for private 192.168.82.x addresses anyway. Decide a cert lifetime + rotation mechanism (short-lived certs + automated renewal, e.g. via step-ca's ACME support, beats long-lived certs nobody remembers to rotate).
- Per-repo pieces, once the CA exists — each service that currently calls another service over plain http:// needs: (a) the called service's compose config to terminate TLS (a sidecar like nginx/caddy/stunnel in front of the container is the least invasive way to add TLS without rewriting the app, if the app itself doesn't natively support HTTPS), and (b) the calling service's config updated from http:// to https:// plus trusting the internal CA's root cert. Concretely: ahu-ocr-akta-notaris (PADDLE_OCR_URL, CLEANUP_LLM_URL env vars), ahu-gpu-manager (its upstream registry entries), and anything else the per-repo prompts above end up wiring host-to-host.
- Decide mTLS vs. one-way TLS — one-way (client verifies server cert) is simpler and stops passive eavesdropping; mutual TLS (both sides present certs) additionally stops an unauthorized host from calling in, which matters more given the firewall is the current access-control layer, not per-service auth. Recommend starting one-way (fixes the immediate cleartext-PII problem) and treating mTLS as a follow-up hardening step once the CA/cert-issuance plumbing is proven out — trying to do both at once is a bigger first PR than this needs to be.

C. Backups: provision the target that's been open since the first audit

Why this is now the single most-repeated open item across three audits: there is currently no backup target at all — not "risky," genuinely zero recovery story for the governance/audit TimescaleDB (once it holds real audit/PII data) or the OCR/chatbot Postgres instances once they're seeded with real notarial-deed and identity data.

Decided (2026-07-08): request a dedicated backup-target host — do not co-locate on ahu-gov-01. Putting the backup on the same host as the TimescaleDB it protects defeats the point of having a backup: if ahu-gov-01 dies (disk failure, host compromise, the hypervisor going down), the primary and its backup are lost in the same event. This isn't a new ask, either — the original devops request doc listed "a backup target for the governance host's database" as its own separate line item, distinct from the 6-host table, and it just never got delivered. Follow up on that original request rather than compromising now. If there's urgency before devops can provision it, ahu-dwh-store-01 (idle, 2TB, DWH engine not built yet) is a defensible temporary stopgap — it's at least a different physical host — but shouldn't be the permanent home given that capacity is earmarked for the DWH engine's own storage later, and using it now means opening a firewall path into what's supposed to be the platform's most isolated zone.

Scope:
1. Pick a backend. On-prem S3-compatible object storage (MinIO is the natural fit — same "self-hosted, on-prem-first" posture as everything else in this platform) is the most consistent choice given nothing here relies on external cloud services; a managed cloud object-storage bucket is the alternative if that's more operationally acceptable than running one more stateful service. This is worth a quick decision up front rather than defaulting silently.
2. Back up each stateful service appropriately, not with one generic script:
- TimescaleDB (ahu-gov-01) — pg_basebackup + continuous WAL archiving for real point-in-time recovery (PITR), not just periodic pg_dump, given this is the audit-integrity store the platform's own contract calls "not optional even at MVP."
- OCR's and the chatbot's Postgres/pgvector instances (ahu-int-01) — pg_dump on a schedule is probably sufficient given these are smaller and less write-heavy than the audit stream, but confirm that assumption against actual write volume once real traffic exists.
- Any Docker volumes holding uploaded documents (OCR's uploads volume, per its compose file) — these aren't in a database at all, so they need their own backup path (a straightforward rsync/tar-to-object-storage job), easy to forget since it's not "a database backup."
3. Define RPO/RTO before picking a schedule — "how much data can we afford to lose" and "how long can we be down" should drive the backup frequency and the restore procedure's design, not the other way around. Nobody's defined these numbers yet; that's part of this prompt's scope, not just the mechanics.
4. Restore test, not just a successful backup job. Schedule a periodic (quarterly is reasonable to start) "restore this backup to a scratch instance and verify the data's actually there and queryable" drill. An untested backup is a hypothesis.