think
16px
820px
← all files

SUPERSEDED (2026-07-09): Grafana OnCall OSS was found archived (Mar 2026). We deployed Keep (keephq) instead and the webhook now emits Keep's AlertDto. See docs/deploy/2026-07-09-keep-deployment.md. This doc is kept only for the batch-vs-per-alert rationale.

Observatory → Grafana OnCall (self-hosted) webhook integration

Guide for wiring the observatory's EWS webhook (internal/notify) to a
self-hosted Grafana OnCall, instead of a bespoke escalation UI. Written from
the observatory side (the webhook is ours); the OnCall deployment/config is
yours. The observatory side (OnCall-native per-alert payload) is already
implemented + deployed dormant — see the payload + setup below.

What the observatory emits (tag obs-oncall-webhook-v1 — Option B is now IMPLEMENTED)

  • Config: notifications.webhook in observatory.yaml — enabled, url, token_env. The notifier is dormant unless notifications.enabled: true + this channel enabled: true.
  • When it fires: on each eval_interval_s tick, after the observatory's own min_severity filter + dedup (dedup_window_s) + escalate-once (escalate_after). The webhook now sends one POST per warning (per alert_uid), not a batch — and emits a resolved POST once a condition clears.
  • Request: POST <url>, Content-Type: application/json, and Authorization: Bearer <token> when the env var named by token_env is set (secret resolved at send time; never in yaml/logs). Per-POST timeout; a failed POST doesn't stop the others; errors logged, never crashes the service.
  • Body (one per alert) — OnCall-native:
    json { "alert_uid": "<stable fingerprint: id|metric|subject>", "state": "firing", // or "resolved" when the condition clears "escalated": false, // true only on the escalate-once tick "evaluated_at": "2026-07-09T12:00:00Z", "severity": "critical", "metric": "tokens_out", "subject": "ahu-chatbot/public/user-x", "title": "...", "message": "<detail + evidence lines>", "evidence": [ { "label": "tokens_out", "value": 58967 } ] }
  • OnCall mapping: alert_uid → grouping key; state → firing/resolved (auto-resolve); title/message → alert title/body. Duplicate firing POSTs for the same alert_uid are idempotent (OnCall groups by it), so the observatory's at-least-once retry is safe.
  • Email channel (if enabled) stays a lumped human digest of firing alerts only — resolved-only ticks send no email.

Recommended OnCall setup (self-hosted — fits the on-prem / no-external-cloud posture)

  1. Deploy Grafana OnCall self-hosted (Helm or docker; it's free and runs fully on-prem).
  2. Create a Webhook integration → it gives you a unique inbound URL (+ optional auth). Put that URL in notifications.webhook.url; if you use a token, set the env var named by token_env (e.g. WEBHOOK_TOKEN) in the observatory compose env — never in the yaml.
  3. In the integration's alert templates, map fields (Jinja over the per-alert JSON above): alert_uid → grouping key, title ← title, message ← message, resolve on state == "resolved", severity ← severity.

Escalation ownership — avoid double-escalation

Once OnCall owns routing/escalation/ack/on-call schedules, let it. Don't run both the observatory's own escalation-to-email and OnCall for the same condition, or an operator gets paged twice by two systems. Clean division:

  • Observatory = detect + emit the signal (per-alert, with firing/resolved state). Consider disabling the observatory email channel (leave only the OnCall webhook enabled), and set escalate_after/dedup_window_s loosely — OnCall's policies, not ours, drive re-notification. (Our dedup doesn't starve OnCall: OnCall escalates on time since firing while unacked, not on repeated POSTs, so suppressing duplicates our side is fine; and we now emit the resolved event on clear, so OnCall auto-closes.)
  • Grafana OnCall = group, route, escalate (tiers/timers), acknowledge, on-call rotation, silence — the whole workflow, battle-tested, with its own UI. This is exactly why we chose it over a bespoke UI.

TL;DR

Option B is shipped (obs-oncall-webhook-v1, deployed dormant): the webhook already emits per-warning alerts with alert_uid + firing/resolved state, so OnCall grouping / ack / auto-resolve work out of the box. Your side: deploy self-hosted OnCall, create a Webhook integration, put its URL in notifications.webhook.url + the token in the token_env env var, set notifications.enabled: true + webhook.enabled: true, and docker compose restart observatory. Consider disabling the email channel so OnCall owns escalation/ack (avoid double-paging).