RAG Quality Monitor — design spec

Status: design (approved in brainstorming 2026-07-09; awaiting spec review).
One-liner: measure whether the public chatbot's answers are grounded (backed by the documents it retrieved) and relevant (actually answer the question) — post-hoc, from the audit trail, with zero chatbot code changes — and surface degradation on the dashboard + through the existing Keep pager.

Plain-language summary

Today we can see that the chatbot answered, but nothing about answer quality. This adds an automatic grader: it reads recently-finished public conversations from the audit history, pulls out (question → retrieved documents → answer), and scores each two ways — cheap deterministic checks, plus an on-prem LLM judge (faithfulness / relevance). Scores go to a dashboard "RAG Quality" view and, when quality drops, to Keep as an alert. No changes to the chatbot for the public surface.

The key enabling finding (verified against live data 2026-07-09)

The chatbot's synthesis LLM call (the one that writes the final answer) is audited by the gateway with its full request + response body stored (zstd, hash-chained, in ai_call_bodies, keyed by event_id). That request body inlines everything the grader needs:
- the user question (prefixed PERTANYAAN:),
- the retrieved passages with scores (a HASIL ALAT: section: [RAG-1] (skor 0.79) …),
- a strict grounding system prompt ("setiap klaim WAJIB bersumber dari hasil alat … JANGAN mengarang fakta").
The response body holds the answer. So the RAGAS triad (question, context, answer) is reconstructable per turn from the synthesis call alone — no dependency on the (comparatively rare) operation=search rows, and no engine change. Verified: a real public synthesis body contained the question + two scored RAG passages + the answer.

Scope

• In (v1): the public chatbot's orchestrator synthesis turns (the path that emits the HASIL ALAT grounding prompt). Deterministic metrics + LLM-judge, dashboard view, EWS→Keep alerting, config-driven thresholds.
• Out (phase 2, separate specs): (a) staff/internal chatbot — its synthesis path must be confirmed to inline context the same way; if it does, it's a config addition, if not it needs an engine-telemetry prompt (hands-off). (b) Golden Q&A regression set (Badan Hukum / Yayasan / apostille) run on deploys — reuses the judge harness. (c) Any turn served by a path that does not inline retrieved context into the prompt (documented limitation, below).

Architecture (all inside ahu-ai-observatory — Go; no new container)

A new background quality evaluator + store + API + dashboard view, mirroring the EWS notifier's shape (a Run(ctx) ticker started in main.go, config-gated, dormant unless enabled).

1. Sampler. On a ticker (eval_interval_s, default 300), find public synthesis calls completed since the last run that haven't been scored yet. A synthesis call is identified by a signature in its request body's system message (default marker set in config, e.g. contains "Susun jawaban" + "HASIL ALAT") — robust to model/position, unlike "last chat in trace". Bound each run (max_batch, default 100). Volume today is single-digits/day, so effective sampling is ~100%; sample_rate (default 1.0) is there for when volume grows.
2. Extractor. Decompress the request+response bodies (reuse the existing klauspost/compress/zstd decoder + ai_call_bodies read path). Parse:
   - question ← text after PERTANYAAN:,
   - context ← the HASIL ALAT: section (kept as a blob; passages + skor N parsed opportunistically for the deterministic metrics),
   - answer ← response body choices[0].message.content.
   Parsing markers live in config (prompt-template coupling is the main fragility — see Risks).
3. Deterministic metrics (free, always): had_context (bool), num_passages, top_score / mean_score (from skor N), context_chars, answer_chars, refused (answer opens with the honest-fallback phrase / no context present). These alone catch "answered with no/weak retrieval".
4. LLM judge (on-prem, gateway, batch): one call per turn to the gateway /v1/chat/completions, model qwen-35b (on-prem — prod-legal, no external), X-Tenant-Id: ahu-observatory, X-Surface: system, X-Priority: batch, deterministic Idempotency-Key (= the graded event_id). A single structured-output judge prompt returns three 0–1 scores + short reasons: faithfulness (every claim supported by context), answer_relevance (addresses the question), context_relevance (retrieved context was on-topic). The judge call is itself audited (observatory becomes a visible tenant). Judge failures/timeouts → the turn is recorded judge_status=error (deterministic metrics still stored), never blocks the loop.
5. Store. New hypertable rag_quality (one row per graded synthesis call): event_id (unique), trace_id, ts, engine, surface, question_sha/question_preview, num_passages, top_score, had_context, refused, faithfulness, answer_relevance, context_relevance, judge_status, judge_model. Idempotent on event_id (never double-grade / double-charge the judge).
6. API (internal/api). GET /api/quality/summary (windowed aggregates: mean faithfulness/relevance, grounded-rate = faithfulness ≥ τ, refusal-rate, count) and GET /api/quality/recent (worst-scoring recent turns for human eyeballing: preview + scores + reasons). Role: operator+auditor (rolesCalls).
7. Dashboard ("RAG Quality" view). Trend of faithfulness/relevance over time (toned via the existing thresholds pattern), grounded-rate + refusal-rate tiles, and a "worst recent answers" table (question preview, scores, judge reason) — honest empty state when no graded turns.
8. Alerting (reuse EWS→Keep). Add quality rules to the existing warning evaluation: e.g. rag_low_faithfulness (windowed mean faithfulness < τ over ≥N graded turns → warning/critical), rag_ungrounded_spike (share of turns with faithfulness < floor exceeds X). These flow through the notifier → Keep exactly like the security/early-warning rules (stable fingerprint id|metric|subject), no new alerting path.

Config (new rag_quality: block in observatory.yaml, defaults → dormant)

enabled (default false), eval_interval_s (300), max_batch (100), sample_rate (1.0), judge_model (qwen-35b), judge_tenant (ahu-observatory), synthesis_markers (the request-body signatures), parse markers (PERTANYAAN:, HASIL ALAT:, skor), and thresholds (faithfulness_warn/faithfulness_crit, min_graded, ungrounded_floor). Secrets none (on-prem judge, no key). Dormant + byte-identical when absent, like every other observatory feature.

Data-access / PDP note

The grader reads stored request/response bodies (auditor-scope data that can contain citizen queries) programmatically, and sends question+context to the on-prem judge model through the gateway (audited; never leaves on-prem — prod-legal per the no-external-LLM rule). This is a new automated reader of body data; it should be recorded as a distinct actor and called out to whoever owns the PDP posture. No bodies leave the platform.

Risks / honesty

• Prompt-template coupling (main fragility). Extraction depends on the orchestrator's synthesis prompt format (PERTANYAAN: / HASIL ALAT: / skor). If the chatbot changes that template, extraction degrades. Mitigations: markers in config (no redeploy to adjust); a parse_status per row + a dashboard/alert signal when the parse-success rate drops (so a template change is detected, not silent); faithfulness can fall back to "whole prompt context minus question" if the section markers vanish.
• Judge is an LLM — scores are directional, not ground truth. We mitigate with the deterministic metrics (which are exact) and by surfacing the judge's short reason for human spot-checks; the golden-set (phase 2) adds a fixed yardstick.
• Coverage = turns whose synthesis prompt inlines context (the orchestrator path). A turn answered with no retrieval at all is recorded had_context=false (itself a useful signal), not silently skipped.

Success criteria

• For a real public doc-Q&A turn, a rag_quality row appears within one interval with sensible deterministic metrics + three judge scores + reasons; re-runs don't double-grade (idempotent on event_id).
• Dashboard "RAG Quality" view renders trend + grounded/refusal tiles + worst-recent table from live data.
• Dropping the faithfulness threshold / injecting a low-faithfulness turn raises a warning that reaches Keep via the existing pipe.
• Judge outages degrade to deterministic-only rows, never stall the loop; the whole feature is dormant + byte-identical when rag_quality.enabled: false.