think
16px
820px
← all files

Golden Q&A Regression Set — design spec (RAG Quality phase 2b)

Status: design (approved to build 2026-07-10). Extends the phase-1 RAG Quality judge harness.
One-liner: an on-demand, CI/deploy-triggerable runner that asks the chatbot a fixed set of curated high-stakes questions, scores each answer against an expert-verified expected answer with the on-prem judge, and reports pass/fail + regressions vs the last run — so a deploy or model change that degrades answer quality on Badan Hukum / Yayasan / apostille is caught before users hit it.

Why this, not live faithfulness, for regressions

Live faithfulness (phase 1) grades "is the answer faithful to what was retrieved." It cannot catch "the retrieval/answer is confidently wrong" — including the internal data-agent restating a wrong SQL result faithfully. A golden set with expert-verified correct answers is the only thing that checks correctness against ground truth, and it's the natural regression gate on every deploy/model swap.

HONESTY constraint (shapes the deliverable)

The authoritative "correct answers" for a government legal/apostille system MUST be authored/verified by AHU domain experts — I cannot invent them. So this spec delivers:
1. the harness (runner + scoring + storage + report + CI gate),
2. a small example golden set with every item flagged verified: false (illustrative structure only — NOT authoritative),
3. an SME guide for populating/verifying the real set.
The mechanism ships now; the ground-truth content is a human task the harness enables.

Architecture

A standalone cmd/goldeneval binary in the observatory repo (CI/deploy-triggerable, NOT a background loop), reusing the phase-1 gateway judge, writing to a new golden_results table, surfaced read-only on the dashboard.

  • Golden set file (deploy/golden/golden-set.example.yaml): a list of items:
    ```yaml
    items:
    • id: pt-pendirian-syarat
      surface: public # public (driven now) | internal (schema-ready, drive deferred)
      domain: badan-hukum-pt
      question: "Apa saja syarat mendirikan PT perorangan?"
      expected:
      must_include: ["WNI", "berusia minimal 17/18 tahun", "satu orang", "UMK"] # key facts the answer MUST contain
      must_not_include: ["notaris wajib"] # known-wrong claims that must NOT appear
      rubric: "Answer should name the founder eligibility + that PTP is for micro/small business, online via ptp.ahu.go.id."
      verified: false # SME sets true after review
      ```
  • Runner (goldeneval -set <file> -config <observatory.yaml>):
    1. Assigns a run_id. For each surface: public item, drives POST <chatbot_url>/api/orchestrate with {surface:"public", sessionId:"golden-<run_id>-<item_id>", message: question} and concatenates the event: content SSE deltas into the answer. (chatbot_url from config; the observatory container can reach http://192.168.83.20:3500 — verified.)
    2. Scores via a golden judge (new Judge.ScoreGolden): given (question, expected {must_include/must_not_include/rubric}, answer) → {correctness 0..1, passed bool, reason}. On-prem, X-Priority: batch, deterministic idem-key golden-<run_id>-<item_id>, enable_thinking:false (same Qwen3 handling as phase 1). Deterministic pre-checks first: a missing must_include phrase or a present must_not_include phrase → auto-fail (records which), independent of the judge.
    3. Writes one golden_results row per item; prints a summary (passed/total, list of failures with reasons, regressions vs the previous run_id for the same items); exits non-zero when pass-rate < golden_min_pass_rate (CI gate).
  • surface: internal items are LOADED + schema-valid but v1 does NOT drive them (the internal data-agent endpoint/auth + answer-drift design are unresolved — see Deferred). The runner logs "internal items skipped (drive not wired)".
  • Storage (golden_results table, plain table): run_id, run_at, item_id, surface, domain, question_preview, answer_preview, score, passed, reason, judge_status. Store fns: insert, latest-run summary, per-run items, previous-run comparison.
  • API (GET /api/quality/golden, operator+auditor): latest run summary (run_at, passed/total, pass_rate) + per-item results. Read-only.
  • Dashboard: a "Golden set" panel on the RAG Quality view — latest run pass-rate tile + a per-item pass/fail table (question, score, pass/fail, reason). Honest empty state when no run yet.
  • Config (rag_quality block additions): chatbot_url (default http://192.168.83.20:3500), golden_min_pass_rate (default 0.8), golden_pass_threshold (per-item correctness→pass cutoff, default 0.7). Judge reuses the existing judge_* config.

Deferred (documented, not built)

  • Internal data-agent golden drive: needs (a) the internal query endpoint + staff-auth wired into the runner, (b) drift-stable question design — data-agent "correct answers" change as the DB changes, so golden items must be time-scoped, structural ("names 3 provinces"), or run against a frozen fixture. Schema is surface-aware so this is an additive follow-up.
  • Auto-triggering from CI/deploy hooks (the binary is the integration point; wiring it into a pipeline is ops).

Risks / honesty

  • Content authority: example items are illustrative + verified:false; a red banner/flag in the report + dashboard makes clear the set isn't authoritative until SMEs verify. The harness must never present example-set results as a quality guarantee.
  • Driving live chatbot: generates real (batch-priority) LLM traffic tagged golden-* (excluded from real per-actor metrics the same way smoke actors are; also passively graded, harmlessly). Bounded by the set size.
  • Judge is an LLM: golden scoring pairs the judge with deterministic must_include/must_not_include string checks so hard facts aren't left to judgment; the rubric handles nuance.
  • Answer non-determinism: the chatbot's answer varies run-to-run; the golden gate is about correctness, not exact-match, so the judge + fact-checks tolerate wording changes. Pass-rate trend over runs is the signal, not a single item flip.

Success criteria

  • goldeneval run against the live public chatbot produces a golden_results row per public item with a correctness score + pass/fail + reason; missing must-include facts auto-fail; summary prints + exits non-zero below the pass-rate gate.
  • GET /api/quality/golden + the dashboard panel show the latest run; honest empty state before the first run.
  • Example set + SME guide are clearly marked non-authoritative.
  • Re-running is idempotent per (run_id, item); a new run_id is a fresh run (history retained for regression comparison).