Server Delivery Audit — AHU AI Platform Non-GPU Hosts
Date: 2026-07-08
Scope: the 6 hosts delivered at 192.168.82.120–125, checked against 2026-07-06-devops-server-request.md and 2026-07-06-server-topology-and-specs.md.
Method: direct SSH (ai-developer / password from vm_ai.xlsx) into all 6 hosts; specs read from the guest OS, network reachability tested directly.
TL;DR
Devops delivered exactly the MVP tier, 6 hosts, correctly named and mapped to zones, with compute specs at or above every requested range — that part is right. At initial audit (2026-07-08 AM), almost none of the "every host, regardless of tier" baseline requirements were applied: no firewall, no network segmentation, no Docker runtime, no monitoring agent, and the DMZ host had open network access to both DWH hosts — the one path the design docs say must never exist.
Update (2026-07-08, same day): items 2–7 of the recommendation below were remediated directly (SSH password auth was deliberately kept on per explicit decision — see §6). Firewall, Docker, monitoring, and patch scheduling are now live on all 6 hosts, and the DMZ→DWH/Governance/internet paths are confirmed blocked. Remaining open items: LB/WAF approach and the governance-DB backup target still need a devops decision (§4), and several inter-zone firewall rules are placeholders pending real service ports (§6).
1. Host-to-purpose mapping
All 6 hostnames and roles match the MVP tier table in the request doc exactly:
| IP | Hostname | Zone / future purpose |
|---|---|---|
| 192.168.82.120 | ahu-ctrl-01 |
Control plane — Gateway + Controller + Redis |
| 192.168.82.121 | ahu-gov-01 |
Governance — Observatory ingester + query API + dashboard + TimescaleDB |
| 192.168.82.122 | ahu-int-01 |
Internal engines — OCR + internal chatbot + RAG/vector store |
| 192.168.82.123 | ahu-dmz-01 |
DMZ — public chatbot only (internet-facing) |
| 192.168.82.124 | ahu-dwh-01 |
DWH compute — warehouse/lakehouse + ETL/CDC + Domain API + governance/audit |
| 192.168.82.125 | ahu-dwh-store-01 |
DWH storage — operational metastore + lakehouse object storage |
No 7th/8th host was delivered for the LB+WAF or the governance-DB backup target the MVP request also called for (see §4).
2. Compute spec compliance
| Host | Requested | Delivered | Verdict |
|---|---|---|---|
ahu-ctrl-01 |
8 vCPU / 16 GB / 100 GB SSD | 8 vCPU / 15Gi RAM / 100 GB (sda) |
✅ exact match |
ahu-gov-01 |
8–16 vCPU / 32–64 GB / 100 GB OS + 1–2 TB NVMe, growable | 16 vCPU / 62Gi RAM / 100 GB (sda) + 2 TB (sdb, mounted /data) |
✅ top of range |
ahu-int-01 |
16 vCPU / 32–64 GB / 500 GB SSD | 16 vCPU / 62Gi RAM / 500 GB (sda) |
✅ top of range |
ahu-dmz-01 |
4–8 vCPU / 16 GB / 50–100 GB SSD | 8 vCPU / 15Gi RAM / 100 GB (sda) |
✅ top of range |
ahu-dwh-01 |
16–32 vCPU / 64–128 GB / 200 GB SSD | 32 vCPU / 125Gi RAM / 200 GB (sda) |
✅ top of range |
ahu-dwh-store-01 |
8–16 vCPU / 32–64 GB / multi-TB, start ~2–4 TB | 16 vCPU / 62Gi RAM / 2 TB (sda) |
✅ at range floor for disk, rest top of range |
All 6 are QEMU/KVM virtual machines ("QEMU Virtual CPU version 2.5+"), not bare metal — worth confirming with devops whether that was the intended delivery model, since the request doc didn't specify virtualized vs. physical. Disk devices show as generic virtio (sda/sdb); the guest can't confirm whether the governance host's 2 TB volume is backed by actual NVMe media on the hypervisor side — that's only verifiable from the host/hypervisor, not the VM.
Verdict: compute sizing is fully compliant, consistently provisioned at the top of each requested range.
3. Baseline requirement compliance ("every host, regardless of tier")
Checked identically across all 6 — the pattern is the same on every host (clearly one template):
| Requirement | Status | Evidence |
|---|---|---|
| OS: Ubuntu 22.04/24.04 LTS | ✅ Pass | All 6 report Ubuntu 24.04.3 LTS (Noble Numbat) |
| NTP/chrony mandatory | ✅ Pass | systemd-timesyncd active, System clock synchronized: yes on all 6 (chrony itself isn't installed, but systemd-timesyncd satisfies "NTP") |
| SSH key-only (no passwords) | ❌ Fail | sshd -T shows passwordauthentication yes on all 6 — we authenticated with a plain password, which the spec explicitly forbids |
| Firewall: default-deny | ❌ Fail | ufw status → inactive; iptables -L shows Chain INPUT/FORWARD/OUTPUT (policy ACCEPT) with zero rules on all 6 — fully open, not default-deny |
| Per-zone private VLAN/subnet | ❌ Fail | All 6 hosts sit on one flat 192.168.82.0/24 with the same default gateway (192.168.82.1) — no zone separation at the network layer |
| Runtime: Docker + Compose v2 | ❌ Fail | docker: command not found on all 6; no docker package installed |
| Monitoring: disk alerting at 70%/85% | ❌ Fail | No monitoring agent found (checked node_exporter, telegraf, zabbix, netdata, datadog, collectd) — only thermald present, which is unrelated |
| Patching: monthly window (not fully unattended) | ⚠️ Partial/wrong mode | unattended-upgrades is installed and enabled (APT::Periodic::Unattended-Upgrade "1") — the doc explicitly asks for a controlled monthly window, not full auto-apply, because these services have uptime/HA expectations. Current config is the opposite of what was requested. |
Critical finding: DMZ → DWH path exists (should be impossible)
The design doc states, as a non-negotiable: "DMZ / Internet → DWH (any host): Explicitly denied — DWH is the platform's most isolated zone, no path from the public side, ever."
Tested directly from ahu-dmz-01 (192.168.82.123):
- ping to ahu-dwh-01 (.124) and ahu-dwh-store-01 (.125): 0% loss, <0.3ms RTT (same L2 segment)
- ping to ahu-gov-01 (.121): same result
- Direct TCP connect to ahu-dwh-01:22: OPEN
This is the combined effect of the flat subnet + fully-open firewall above, but it's worth calling out on its own: right now, the DMZ host can reach every other zone, including DWH, over any port. This is the one boundary the entire design is built around, and it doesn't exist yet.
4. Not delivered / needs confirmation with devops
- Load balancer + WAF — no dedicated host for this among the 6 IPs. The request allowed for "managed cloud LB/WAF preferred" as an alternative to a dedicated host — need to confirm with devops which path they're taking.
- Backup target for the governance DB — called out as "not optional even at MVP" in the request doc; not present among the delivered hosts.
5. Recommendation (original, 2026-07-08 AM)
Treat this delivery as "right hardware, wrong hardening state" — the sizing/naming work is done and correct, but the hosts need a baseline pass before anything gets deployed on them:
1. Disable SSH password auth, enforce key-only.
2. Turn on ufw/nftables with default-deny + the allow-matrix from the request doc (this alone closes the DMZ→DWH gap).
3. Split the flat /24 into per-zone VLANs/subnets, or at minimum add L3 ACLs between zones if VLANs aren't feasible yet.
4. Install Docker + Compose v2 on all 6.
5. Install a monitoring/disk-alerting agent (70%/85% thresholds) — given the GPU host's repeated disk-full incidents, this is the one they've already been burned by once.
6. Switch unattended-upgrades from auto-apply to a monthly-window model.
7. Confirm LB/WAF approach and stand up (or provision) the governance-DB backup target.
6. Remediation performed (2026-07-08, same day)
Per direct instruction: item 1 (SSH key-only) was explicitly skipped — password auth stays on for easier debugging/access. Items 2, 4, 5, 6 were completed directly; item 3 was implemented as host-based L3 ACLs (true VLAN reassignment isn't reachable from inside the guest OS — see below); item 7 needs a follow-up decision (§7).
Docker (item 4): docker-ce + docker-ce-cli + containerd.io + docker-buildx-plugin + docker-compose-plugin installed via the official Docker APT repo on all 6 hosts. Verified: docker --version (29.6.1) and docker compose version (v5.3.1) on every host, ai-developer added to the docker group.
Monitoring (item 5): prometheus-node-exporter (apt package, port 9100) installed and enabled on all 6 — ready for a future Prometheus to scrape, matching the DCGM-exporter pattern already used on the GPU host. Plus a local disk-alert.timer (systemd, every 15 min) running /usr/local/bin/disk-alert.sh, which logs to syslog via logger -t disk-alert at WARNING (≥70%) and CRITICAL (≥85%) — confirmed silent on all 6 (all well under 70% currently), confirming it only fires on threshold breach. This satisfies the disk-alerting requirement functionally; it does not itself page anyone — wiring disk-alert syslog entries (or the node_exporter metrics) into an actual alert channel (Slack/email/PagerDuty) is a follow-up once a destination is chosen.
Patch window (item 6): unattended-upgrades auto-apply disabled (APT::Periodic::Unattended-Upgrade "0", service disabled) on all 6. Replaced with a monthly-patch.timer (systemd, fires 1st of month 02:00 UTC + up to 30min randomized delay) running apt-get update && apt-get upgrade -y && apt-get autoremove -y, logged to syslog via logger -t monthly-patch. Next scheduled run: 2026-08-01. ahu-dmz-01's egress lockdown initially blocked this too — resolved, see the apt-mirror exception below.
Firewall + segmentation (items 2 & 3): applied ufw per host, default-deny incoming everywhere, verified with a fresh SSH connection after every single host before moving to the next (no hypervisor console access exists as a fallback, so this was done one host at a time, non-DMZ hosts first):
| Host | Inbound allowed | Outbound |
|---|---|---|
ahu-ctrl-01 |
SSH from 10.15.16.0/24 (admin/bastion — the source IP this session's traffic NATs through); 8200/tcp (Gateway) from ahu-dmz-01 + ahu-int-01; 6379/tcp (Redis) from ahu-gov-01; 8210/tcp (Controller) from the GPU host 192.168.83.20 (node-agent) |
default allow (needs to reach GPU-zone model server ports) |
ahu-gov-01 |
SSH from admin/bastion; 8310/tcp (dashboard) + 8300/tcp (Observatory API) from admin/bastion only |
default allow (needs to reach ctrl 6379 for the audit stream) |
ahu-int-01 |
SSH from admin/bastion only | default allow (needs ctrl 8200 + DWH Domain API once it exists) |
ahu-dmz-01 |
SSH from admin/bastion only (no LB/WAF rule yet — see below) | default deny, explicit allow only to: ahu-ctrl-01:8200/tcp (sole app egress path, per spec), 192.168.175.20/.21:53 (the box's actual configured internal DNS resolvers), 123 (NTP) |
ahu-dwh-01 |
SSH from admin/bastion only | default allow (needs outbound to core AHU registry systems — external, read-only pull) |
ahu-dwh-store-01 |
SSH from admin/bastion only | default allow |
Port numbers came from ahu-gpu-manager/docs/CONVENTIONS.md's allocation table (gateway 8200, controller 8210, observatory API 8300, dashboard 8310) plus Redis' standard default (6379) — these are the only ports actually documented anywhere. Several matrix rows reference ports that don't exist yet because no engine is deployed on these hosts (LB/WAF→DMZ "app port", Internal-engines→DWH "Domain API port", DWH-compute→DWH-store storage port): opening a firewall rule for an undefined port isn't meaningful, so those are deliberately left as follow-ups once the real service/port is chosen — noting this rather than guessing a number.
Verified after enabling on all 6:
- Fresh SSH session succeeds on every host (no lockout).
- From ahu-dmz-01: TCP to ahu-dwh-01:22, ahu-dwh-store-01:22, ahu-gov-01:22 → all BLOCKED. TCP to 1.1.1.1:443 (arbitrary internet) → BLOCKED. TCP to 192.168.175.20:53 (allowed internal DNS) → OPEN as intended.
- This closes the critical finding from the original audit: DMZ can no longer reach DWH, Governance, or the open internet — its only path out is the Gateway.
On true VLAN/subnet separation: all 6 hosts remain on the same flat 192.168.82.0/24 at the L2/L3 addressing level — that part of item 3 needs the hypervisor/switch fabric (VLAN tagging, or moving hosts to distinct subnets), which isn't reachable from inside the guest OS over SSH. What's implemented instead is host-based L3 ACLs (via ufw) enforcing the same allow-matrix at the firewall layer, which achieves the same access-control outcome the design cares about, just not physical broadcast-domain isolation. Flag to whoever owns the hypervisor/network fabric if true VLAN separation is still wanted on top of this.
ahu-dmz-01 apt-mirror exception (added after initial hardening, same day): per instruction, added a narrow egress exception instead of push-based patching or leaving it manual. archive.ubuntu.com and security.ubuntu.com (the box's actual configured sources) both resolve, at time of writing, to the same 2 Cloudflare-fronted IPs (104.20.28.246, 172.66.152.176, plus their IPv6 equivalents) — ufw rules allow outbound 80/443 to exactly those 4 addresses, comment-tagged apt-mirror. Verified: apt-get update now succeeds for both Ubuntu repos (Hit:1-4). Deliberately not extended to download.docker.com — that's CloudFront-backed with a much larger, faster-rotating IP pool, so pinning it would either be unreliable or require a far broader (non-narrow) rule; apt-get update shows it failing cleanly (W: Failed to fetch ... Network is unreachable), which just means Docker-engine-specific package updates on ahu-dmz-01 need a manual/bastion-driven update when needed, while OS security patches flow normally.
Caveat: these are today's resolved IPs for a Cloudflare-fronted domain — if Canonical's CDN reassigns edge IPs, apt-get update on ahu-dmz-01 would start failing again silently (the systemd timer doesn't alert on failure). Worth a periodic sanity check (ssh in and confirm monthly-patch logged success) rather than assuming it'll keep working indefinitely.
7. Still needs a decision — not something these SSH sessions can resolve alone
- LB/WAF approach — no dedicated host was delivered; the request allowed for a managed cloud LB/WAF instead. Needs a decision on which path, then (if on-prem) a host to provision or (if managed) an account/service to stand up.
- Governance-DB backup target — "not optional even at MVP" per the request doc; no object storage/PITR service exists yet to point at. Needs a storage destination decided before this can be wired up.
- Placeholder firewall rules — dashboard/API/Domain-API/storage ports between
ahu-int-01↔ahu-dwh-01↔ahu-dwh-store-01and the future LB→ahu-dmz-01path aren't opened yet since those services/ports don't exist. Revisit once each engine's actual listening port is chosen.
8. Developer account (vigilia)
Created on all 6 hosts (password from /home/efran/pass_tmp at time of writing), added to both sudo and docker groups — same access level as ai-developer, since developers will build/run containers and manage packages on these boxes. Verified: SSH login + whoami/hostname succeed on all 6. No firewall change was needed — developers connect through the same VPN/bastion path (confirmed with the user), which is already inside the 10.15.16.0/24 SSH allow rule.