think
16px
820px
← all files

Keep (keephq) on-call/alerting — deployment runbook (GPU host ai-ahu)

Self-hosted Keep is the escalation/on-call layer for the platform's EWS
warnings, chosen after Grafana OnCall OSS was found archived (Mar 2026).
Deployed on the GPU host ai-ahu (192.168.83.20), VPN-only, on 2026-07-09.
The observatory's EWS notifier POSTs each warning to Keep as a per-alert
firing/resolved event; Keep owns grouping, escalation, ack, and on-call.

What's deployed

  • Compose: ~/keep-deploy/compose.keep.yml on ai-ahu (project keep). Self-contained (not the upstream multi-file compose): telemetry (PostHog/Sentry) disabled, the optional grafana/prometheus profiles omitted (the host already runs its own on :3000/:9090).
  • Services (3):
  • keep-frontend (Next.js UI) — host 192.168.83.20:8330 → :3000
  • keep-backend (FastAPI API + ingestion) — host 192.168.83.20:8331 → :8080
  • keep-websocket-server (soketi, realtime UI) — host 192.168.83.20:8332 → :6001
  • All bound to 192.168.83.20 (VPN-reachable, NOT internet-exposed).
  • State: SQLite + file secret-manager in ~/keep-deploy/state (bind mount). Back up this dir to preserve alerts/config/API keys.
  • Auth: AUTH_TYPE=DB since 2026-08-13 — username/password login on both UI and API. Credentials in ~/keep-deploy/keep-auth.env (mode 600, git-ignored). See 2026-08-13-keep-auth-and-smtp-escalation.md.
  • UI: http://192.168.83.20:8330 (from the VPN) → login page.

How it's wired to the observatory

  • Observatory notifications block (live in ~/ahu-ai-observatory/deploy/observatory.yaml): enabled: true, min_severity: warning, eval_interval_s: 300, single webhook channel → http://192.168.83.20:8331/alerts/event (Keep's generic ingestion). Email channel left disabled — Keep owns escalation/ack, so no double-paging.
  • Ingestion auth: KEEP_ALLOW_MESH_ALERT_INGESTION=true on the backend → the host-internal observatory POSTs tokenless (the backend port is VPN-bound; not open to the internet). No API key to manage for ingestion.
  • Payload: Keep AlertDto — name, status (firing|resolved), severity (critical|warning|info), lastReceived, fingerprint (= the observatory's stable warning fingerprint id|metric|subject), source: ["ahu-observatory"], description, labels (metric, subject, escalated). Keep dedups/auto-resolves by fingerprint.

Verified end-to-end (2026-07-09)

  • Backend up (200); test firing alert → 202 ingest → visible in Keep; same-fingerprint resolved → the alert flipped to resolved (auto-resolve, no duplicate).
  • Live: the observatory notifier evaluated real warnings and Keep received them — e.g. Kemungkinan probing / percobaan jailbreak (sec_error_probing) and Volume panggilan abnormal (sec_volume_outlier), firing/warning, grouped by fingerprint, with the actor as labels.subject.

Ops

  • Status/logs: cd ~/keep-deploy && docker compose -f compose.keep.yml ps · ... logs -f keep-backend
  • Restart/stop: docker compose -f compose.keep.yml restart · ... down (state persists in ./state)
  • Read API (needs a real token since auth was enabled): POST /signin with the keep-auth.env credentials, then send Authorization: Bearer <accessToken>. The old X-API-KEY:keepappkey trick worked only because NO_AUTH accepted any key; it now returns 401.
  • Configure escalation/on-call: in the Keep UI (http://192.168.83.20:8330) — workflows, escalation policies, on-call schedules, ack, silence. This is where the "escalation workflow" lives (deliberately not rebuilt in our dashboard).

Hardening backlog (before real production reliance)

  1. ~~Enable auth.~~ DONE 2026-08-13 — AUTH_TYPE=DB. Note what this closed: NO_AUTH had already seeded a user keep with role admin and the default password keep, so the instance was one AUTH_TYPE flip away from a default-credential admin login. That user is now renamed admin with a generated password, and keep/keep returns 401.
  2. Postgres, not SQLite. SQLite single-node is fine for MVP; point DATABASE_CONNECTION_STRING at a Postgres for durability/scale (the host already runs Postgres/Timescale).
  3. Pin image tags. Currently :latest for keep-ui/keep-api — pin to a released version so a rebuild can't drift.
  4. Notification transports out of Keep. Prepared, blocked on one input. deploy/keep/setup-smtp.sh installs the SMTP provider + the email escalation workflow and verifies delivery in one command — it needs a relay host/credential and a routable recipient, which do not exist on this host yet. Until then Keep has 0 providers and 0 workflows, i.e. alerts are collected and deduped but nobody is paged. See 2026-08-13-keep-auth-and-smtp-escalation.md.
  5. Edge DNS (optional): front the UI with an edge vhost (like the dashboard's x056.ahu-demo.dashboard.val.id) if browser access beyond the VPN is wanted.
  6. Resource limits + backups: add compose mem_limit/cpus and a periodic backup of ~/keep-deploy/state.

Rollback

Disable the observatory notifier (notifications.enabled: false in observatory.yaml + docker compose restart observatory; backup observatory.yaml.bak-pre-keep), and/or stop Keep (docker compose -f ~/keep-deploy/compose.keep.yml down). The observatory is byte-identical to pre-notifier when disabled.