OnCall-native webhook — implementation report

Branch: feat/oncall-webhook. Implements Option B from
docs/integration/2026-07-09-grafana-oncall-webhook.md: the webhook channel
now POSTs one JSON body per warning, carrying a stable alert_uid
(Fingerprint) and a firing/resolved state, so Grafana OnCall groups by
condition and can auto-resolve. Email stays a lumped human digest of firing
warnings only.

Status

All gates green:
- go build ./... — OK
- go vet ./... — OK
- gofmt -l on all touched files — clean (no output)
- go test -race ./internal/notify/ — PASS (all tests, including race
detector)
- TEST_PG_DSN='postgres://postgres:test@dind:5434/postgres' go test ./...
(after bash scripts/testdb.sh) — PASS on a clean -count=1 run. One
transient failure was observed on a single earlier run
(internal/store.TestInsertEventDedups: deadlock detected (SQLSTATE 40P01)) — confirmed pre-existing DB-contention flakiness unrelated to this
change (internal/store was not touched; the test passes standalone and on
rerun of the full suite). Not caused by, or related to, this diff.

Per-change summary

notify.go — added Alert{Warning, AlertUID, State, Escalated} and
StateFiring/StateResolved consts; Notification is now
{Alerts []Alert, EvaluatedAt time.Time} (dropped Warnings/Escalated).
dispatcher.go — entry gained a last predict.Warning field,
refreshed on every sighting of a fingerprint (firing or suppressed).
Decide now returns []Alert instead of ([]predict.Warning, bool):
firing alerts follow the same state-machine branches as before (new /
dedup-suppress / escalate-once / re-notify-after-window), now each
producing an Alert{State: StateFiring, ...}; a NEW branch walks
d.state for fingerprints absent from the current (filtered) tick and
emits one Alert{State: StateResolved, Warning: entry.last} per one,
deleting the entry immediately after — so a resolve fires exactly once.
Alerts are sorted firing-before-resolved, then severity desc, then
AlertUID ascending.
webhook.go — Send iterates n.Alerts and calls a new sendOne
per alert, building a webhookAlertPayload (alert_uid, state,
escalated, evaluated_at, severity, title, message, metric,
subject, evidence). Failures are collected (not short-circuited) and
returned via errors.Join; ctx cancellation is checked before each POST
and stops the loop (remaining alerts not attempted, already-collected
errors still returned). Bearer auth resolved per-alert via
config.ResolveSecret (unchanged secret-handling contract). Never panics.
email.go — Send filters to StateFiring alerts via a new
firingWarnings helper; if none, returns nil immediately (sender never
invoked — verified by a dedicated test). buildSubject/buildBody now
take (firing []predict.Warning, ...) instead of the whole
Notification. [ESCALATED] prefix triggers off anyFiringEscalated
(only firing alerts are considered, though a resolved alert's Escalated
is always false by construction anyway).
evaluator.go — tick calls safeDecide (now returning
([]Alert, error)), skips only when len(alerts) == 0 (a resolved-only
tick is NOT skipped — it still reaches every channel). Builds
Notification{Alerts: alerts, EvaluatedAt: now}. Retry/timeout/recover
plumbing in dispatch is untouched.
main.go — untouched; confirmed it only references the Channel
interface, NewEmailChannel/NewWebhookChannel/NewDispatcher/
Evaluator, none of which changed shape. go build ./... confirms.

Anti-storm verification

TestDispatcherDedupHoldsUnderChangingEvidence (the C1 regression guard)
adapted to []Alert: a warning recurring every tick with changing evidence
but stable ID/Metric/Subject still fires exactly once (tick 0), suppresses
(ticks 1-2), escalates exactly once (tick 3), suppresses again (ticks 4-5)
— asserted via firingOf(...), plus a new assertion that NO resolved alert
is ever emitted in this test (would indicate the fingerprint moved). PASS.
New TestDispatcherClearedWarningResolvesExactlyOnce: drives a warning to
escalation, clears it, asserts exactly one StateResolved alert with the
right AlertUID, then asserts 5 more empty ticks produce zero further
resolves. PASS.
TestDispatcherClearedThenRecurringSendsFresh extended: clear now asserts
the resolve fires once (not a no-op), a second clear tick asserts it does
NOT repeat, and the recurrence still fires fresh (not escalated). PASS.
TestDispatcherMinSeverityFiltersBelowFloor unchanged in intent — still
filters below-floor severities before they ever reach fingerprinting. PASS.

Judgment calls

Ordering tie-break — AlertUID not ID. The task spec says "severity
desc, then AlertUID." The pre-existing TestDispatcherOrdersDeterministically
hardcoded an ID-alphabetical expectation, which no longer holds once the
tie-break is a sha256 digest (unrelated to the ID string). Rewrote that
test to assert the documented properties (severity bands correctly
ordered; ascending AlertUID within a band) rather than hardcoding an
order that was an accident of using ID as the old tie-break.
Resolved message content. Made it a short fixed string
("condition cleared") rather than replaying stale evidence/detail from
the last firing — showing "tokens_out=133333" on a resolved alert reads
as if the condition were still current. alert_uid + state are what
OnCall needs to act; title/severity/subject still come from entry.last
for context in the OnCall UI.
Per-POST timeout is derived from ctx, not a fresh budget per alert.
context.WithTimeout(ctx, defaultWebhookTimeout) per alert means the
effective deadline is min(parent ctx deadline, now+10s) — the
evaluator's overall SendTimeout (10s today, notifySendTimeout in
main.go) is a shared budget across all alerts in one Send call, not
reset per alert. For a tick with many alerts under a short SendTimeout,
later POSTs can fail on ctx-deadline exceeded. This matches the letter of
the CHANGES spec ("own timeout ... derived from ctx") but is worth flagging
operationally if eval_interval_s ticks start producing large alert counts
— may want to bump notifySendTimeout in main.go if that becomes common.
Retry re-sends the whole batch, including already-successful POSTs.
Unchanged from prior behavior (the evaluator's dispatch retries the whole
Send call on any non-nil error) — if 1 of N alerts 500s, the retry
re-POSTs all N, including ones that already succeeded. This is an
at-least-once, not exactly-once, delivery contract; acceptable for OnCall
(re-firing/re-resolving the same alert_uid+state is idempotent from its
side) and no different from the pre-existing batch-retry semantics — not a
regression introduced here, just worth naming explicitly since it's more
visible now that failures are per-alert.
Resolve-before-firing computation order in Decide. Resolved alerts
are computed by walking d.state (deleting stale entries as found) before
the firing loop walks current. This is safe (disjoint fingerprint sets;
Go permits deleting map entries during range over that same map) and
keeps the state-mutation logic linear/readable rather than needing a
two-pass diff.

Files touched

internal/notify/notify.go, dispatcher.go, webhook.go, email.go,
evaluator.go (implementation)
internal/notify/dispatcher_test.go, webhook_test.go, email_test.go,
evaluator_test.go (tests, adapted + extended)
cmd/observatory/main.go — NOT modified (verified compiles unchanged)