think
16px
820px
← all files

OpenBao (secrets manager) — deployment runbook

Date: 2026-07-10
Host: ahu-ctrl-01 (192.168.82.120, control plane) · VPN/firewall-only
Status: deployed + verified, including request-level audit logging. Fully operational.

Chosen over Infisical specifically for dynamic secrets / DB credential auto-rotation and request-level audit logging — the two capabilities that matter for the platform's PDP-law compliance posture. (Static-secret UX is slightly nicer in Infisical; those two capabilities are OpenBao's reason for existing.)

What's deployed

  • OpenBao v2.5.5, image digest-pinned openbao/openbao@sha256:6150c4a6….
  • Storage: integrated Raft (single node now; can add peers for HA later — matches the "Compose + external HA primitives" substrate decision). Persistent docker volume ahu-openbao_bao-data.
  • TLS: listener is HTTPS with a self-signed cert (SAN: 192.168.82.120, 127.0.0.1, localhost). Verified: HTTPS health endpoint returns clean JSON; plain HTTP is rejected. Transit is encrypted from day one.
  • Ports: API 8250, cluster 8251 — deliberately remapped off the default 8200, because the Gateway owns 8200 on this same host.
  • Seal: Shamir, 5 key shares / threshold 3.
  • Secrets engines enabled: secret/ (KV v2, static secrets) and database/ (dynamic DB credentials — the auto-rotation engine).
  • Audit device: file/ → /openbao/logs/audit.log (persistent volume), declared in config (OpenBao 2.x requires declarative — API-enable is blocked). Sensitive fields HMAC'd (log_raw=false default). Verified capturing (records accumulating; hmac-sha256: values present, confirming secrets aren't stored raw).
  • Compose: ~/ahu-openbao/compose.openbao.yaml on ctrl-01 (project ahu-openbao), cap_add: IPC_LOCK (mlock — keeps secrets out of swap), resource-limited (1g/1cpu), healthcheck = bao status (so a sealed server correctly reads as unhealthy = "needs unseal").
  • Reviewable config copies in workspace poc-ahu-ai/platform-infra/secrets/. Canonical version-controlled home should be ahu-gpu-manager/deploy/ — not yet committed there.

Verified end-to-end (2026-07-10)

  • bao status: initialized=true, sealed=false, HA enabled, raft, v2.5.5.
  • KV round-trip: wrote + read secret/smoke.
  • Both secrets engines listed (secret/, database/).
  • TLS: https://192.168.82.120:8250/v1/sys/health → 200 JSON; http://… → rejected.
  • Audit: file/ device loaded on restart, log capturing (7+ records), 41 hmac-sha256: values present (sensitive fields hashed, not raw).
  • Container healthy.

⚠️ CRITICAL — do this before relying on OpenBao for anything real

The initialization produced 5 unseal key shares + the root token, currently in a 600 file:

~/ahu-openbao/openbao-init.json   on ahu-ctrl-01 (192.168.82.120)

This file is the crown-jewel bootstrap material. Right now it sits in plaintext on the host — that must not be the permanent state. Required actions (owner, not automatable):
1. Retrieve it, distribute the 5 unseal shares to 5 separate people / secure locations (Shamir 3-of-5 means no single person can unseal alone — a feature for a government-adjacent system).
2. Store the root token in a secure location, then consider revoking it after creating proper admin policies/AppRoles (root token isn't meant for day-to-day use).
3. Delete openbao-init.json from the host once the above is done.

Until then, treat OpenBao as bootstrapped-but-not-hardened.

Unseal lifecycle (important operational note)

OpenBao starts sealed after any restart or host reboot, and needs a manual 3-of-5 unseal to become usable again — this is the security model, not a bug. The container healthcheck will show unhealthy while sealed (correct signal: "needs unseal"). Recommended: tie planned unseals to the monthly patch window already configured (a host reboot during that window is exactly when someone's around to unseal), and document the unseal command for whoever holds the shares:

docker exec -e BAO_SKIP_VERIFY=true ahu-openbao bao operator unseal   # run 3× with 3 different shares

Access

  • UI/API: https://192.168.82.120:8250 from the VPN (self-signed cert → browser warning is expected). Firewall allows admin range 10.15.16.0/24 → :8250.
  • CLI: bao talks to it via BAO_ADDR=https://…:8250 + BAO_SKIP_VERIFY=true (self-signed) until a trusted cert is in place.

Hardening / follow-up backlog

  1. ~~Enable the audit device~~ ✅ DONE — declared in config as audit "file" "file/" { options = { file_path = … } } (two labels: type, path). Verified capturing with HMAC'd sensitive fields.
  2. Secure + remove the init key file (the CRITICAL section above) — now the top open item.
  3. Replace the self-signed listener cert — once audit + PKI are sorted, OpenBao's own PKI engine can issue a proper internal cert (this is the internal-CA the TLS-between-services work also needs; OpenBao doubling as CA was part of its appeal).
  4. Per-service AppRole auth — services fetch their secrets from OpenBao at startup via AppRole (role-id + secret-id), instead of plaintext env files. Then migrate the existing plaintext secrets in: the Grafana admin password (currently in ~/ahu-monitoring/grafana.env on gov-01), and the DB credentials once TimescaleDB/Postgres land (use the database/ engine to rotate them, not just store them).
  5. Tighten the TLS key file perms (currently 644 so the container uid can read it, on a firewalled single-tenant host) — or move cert handling into OpenBao PKI.
  6. HA later: add raft peers on a 2nd control-plane host when the HA tier lands.