Keep: authentication enabled + SMTP escalation prepared

Date: 2026-08-13 · Host: ai-ahu (192.168.83.20, VPN-only) · Status: auth live; email paging ready, needs one input

You picked SMTP as the paging channel and asked for a UI to set the relay and recipient. The
short answer: that UI already exists — Keep ships a first-class SMTP provider with a form —
but it was sitting behind AUTH_TYPE=NO_AUTH, so anyone on the VPN could have read or abused a
ministry relay credential stored in it. Auth came first; the form is now safe to use.

What you need to supply

One relay + one mailbox. Nothing else is blocking.

Field	Notes
`SMTP_SERVER`	must be reachable from `ai-ahu`
`SMTP_PORT`	587 STARTTLS · 465 implicit TLS · 25 plain
`SMTP_ENCRYPTION`	exactly `TLS`, `SSL`, or `None`
`SMTP_USERNAME` / `SMTP_PASSWORD`	blank both for an IP-allowlisted internal relay
`SMTP_FROM`	many relays reject a From outside their own domain
`ALERT_RECIPIENT`	must be routable. `STAFF_ADMIN_EMAIL` is `@ahu.local`, which does not resolve — a distribution list is better than one inbox

Option A — the UI (Providers → SMTP)

http://192.168.83.20:8330 → log in → Providers → filter smtp → click the tile. The form asks
for exactly: Provider Name, SMTP Server Address, SMTP Port, SMTP Encryption (dropdown), SMTP
Username, SMTP Password → Connect. Name it ahu-smtp — the workflow references that name.
Pressing Connect runs Keep's send_email scope check, which opens a real connection and logs in,
so a green result means the relay genuinely works. You then still need the workflow (Option B
installs it for you, or import deploy/keep/ews-email-escalation.yaml via Workflows → Upload).

Option B — one command (recommended)

Keeps the password out of the browser, the shell history, and ps; installs the workflow too.

cd ~/keep-deploy
cp smtp.env.example smtp.env && chmod 600 smtp.env && $EDITOR smtp.env
./setup-smtp.sh --test-only   # validates the relay, stores nothing
./setup-smtp.sh               # installs provider + workflow, sends a real test email

It refuses to proceed on a non-routable recipient, a bad encryption value, or a world-readable
env file; on a relay failure it surfaces the relay's own error and stores nothing.

The delivery path

flowchart LR A[Observatory notifier
dedup + escalate-once] -->|"POST /alerts/event
tokenless, VPN-bound"| B[Keep ingestion] B --> C{"trigger: source =
ahu-observatory"} C --> D["workflow
ahu-ews-email-escalation"] D -->|provider ahu-smtp| E[SMTP relay] E --> F[on-call mailbox]

Severity filtering happens upstream in the observatory (min_severity, default warning), so
the workflow deliberately has no second severity gate. It fires on resolved too — status is in
the subject line, and on-call needs the all-clear as much as the alarm. To page on firing only,
add if: "'{{ alert.status }}' == 'firing'" to the action.

Verified, not assumed

Auth changes to a live pager and a workflow that silently pages nobody are the two failure modes
here, so both were tested rather than reasoned about.

Check	Result
API rejects unauthenticated reads	`GET /providers` → 401 (was 200 — NO_AUTH accepted any key)
Browser login works	Playwright: form submit → authenticated `/incidents`
Default credential killed	`keep`/`keep` → 401
Live pager pipeline survived the auth flip	`POST /alerts/event` → 202, `"Allowing service alert ingestion from ahu-observatory"`
Workflow YAML parses in Keep	`POST /workflows/test` resolved the provider reference correctly
Email actually delivers	end-to-end through a throwaway Mailpit sink — subject `[AHU AI · warning · firing] Escalation path verification`, body fully templated (name, severity, status, fingerprint, labels)
Re-running is idempotent	2 runs → 1 workflow, 2 emails (not 4)

Why the tokenless ingestion survived: the KEEP_ALLOW_MESH_ALERT_INGESTION bypass lives in
AuthVerifierBase._extract_api_key, and DbAuthVerifier overrides only _verify_bearer_token —
so it is inherited under DB auth. Confirmed by reading the code and by the 202 above.

Bugs the verification caught (all fixed)

POST /providers/test is unusable for notify-only providers — it calls get_alerts(), so it
returns "get_alerts() method not implemented" even on a perfect relay. My first script used it
as a pre-flight, which would have blocked every install. Installing is the credential test.
Duplicate workflows double-page. Keep addresses workflows by internal UUID, so
DELETE /workflows/<slug> 404s silently and each re-run installed another copy — measured as
2 workflows → 5 emails. Now swept by workflow_raw_id, with a hard assert of exactly 1 copy.
Same class as the fingerprint storm bug fixed on 2026-07-09.
False-negative verification — polled a non-existent runs URL and reported failure on a run
that had actually succeeded. Now resolves the UUID and baselines the newest execution id before
firing, so a stale prior success can't be misread as proof.
SMTP_FROM_NAME=AHU AI Observatory unquoted in my own env template broke source. Quoted.

Current state

Auth: live. ~/keep-deploy/keep-auth.env (mode 600) holds username admin + a generated
24-char password; KEEP_FORCE_RESET_DEFAULT_PASSWORD=false so a UI password change persists.
Read it on the host: grep KEEP_DEFAULT ~/keep-deploy/keep-auth.env.
Providers: 0 · Workflows: 0 — deliberately clean. The test rig (provider, workflow, Mailpit
sink and image) was fully removed, and the synthetic test alerts were resolved. A workflow left
pointing at a deleted provider would fail on every real alert, and a sink-backed provider would
swallow real pages.
So nobody is paged yet. Alerts still collect, dedup and auto-resolve in Keep; the last mile
is one ./setup-smtp.sh away.
Backups before the change: compose.keep.yml.bak-pre-auth, state/db.sqlite3.bak-pre-auth.

Rollback

cd ~/keep-deploy
cp compose.keep.yml.bak-pre-auth compose.keep.yml
docker compose -f compose.keep.yml up -d --force-recreate keep-backend keep-frontend

Reverts to NO_AUTH. The admin user and its password stay in the DB (harmless under NO_AUTH); to
restore the pre-auth user row too, also copy state/db.sqlite3.bak-pre-auth back while stopped.