think
16px
820px
← all files

Golden Q&A Regression Set — Implementation Plan (RAG Quality phase 2b)

For agentic workers: REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Steps use - [ ]. Spec: docs/superpowers/specs/2026-07-10-golden-eval-design.md.

Goal: an on-demand goldeneval runner that drives the public chatbot with a curated question set, scores answers against expert-verified expectations (deterministic fact-checks + on-prem judge), stores results, reports pass/fail + regressions, and gates deploys — plus a read-only dashboard panel. Reuses the phase-1 judge.

Repos: ahu-ai-observatory (Go; new cmd/goldeneval + store + judge method + API + config) and ahu-observatory-dashboard (a panel). Content honesty: ship the harness + a verified:false EXAMPLE set + an SME guide — never present example results as authoritative.

Global Constraints

  • Golden answers are SME-authored. The example set is illustrative only; every item is verified: false; the report + dashboard visibly flag an unverified set.
  • On-prem judge only, reusing phase-1 ragquality.Judge (gateway, X-Priority: batch, enable_thinking:false, deterministic idem-key). No external model.
  • Deterministic fact-checks gate the judge: a missing must_include phrase or a present must_not_include phrase → auto-fail with the offending phrase recorded, regardless of the judge score.
  • v1 drives surface: public only. internal items load + validate but are skipped-with-a-log (data-agent drive is a documented follow-up). Don't build internal drive.
  • The runner is a CLI, not a background loop; it writes results via its own DB connection and exits non-zero below the pass-rate gate (CI usable).
  • Config additions live in the existing rag_quality block; dormant/unused until goldeneval is run.

Task 1: golden_results store + golden-set schema/loader + config + example set

Files: internal/store/migrations/005_golden_results.sql, internal/store/golden.go (+test), internal/ragquality/goldenset.go (+test), internal/config/config.go (+test), deploy/golden/golden-set.example.yaml, deploy/observatory.example.yaml.
- Migration (plain table): golden_results(run_id text, run_at timestamptz, item_id text, surface text, domain text, question_preview text, answer_preview text, score double precision, passed boolean, reason text, judge_status text, PRIMARY KEY(run_id,item_id)) + index on (item_id, run_at desc).
- store/golden.go: GoldenRow struct; InsertGolden(ctx,pool,GoldenRow); GoldenLatestRun(ctx,pool) (runID string, runAt time.Time, passed,total int, err); GoldenRunItems(ctx,pool,runID) ([]GoldenRow,error); GoldenItemHistory(ctx,pool,itemID,limit) for regression comparison.
- internal/ragquality/goldenset.go: type GoldenExpected struct { MustInclude, MustNotInclude []string; Rubric string }, type GoldenItem struct { ID, Surface, Domain, Question string; Expected GoldenExpected; Verified bool }, type GoldenSet struct { Items []GoldenItem }, LoadGoldenSet(path string) (GoldenSet, error) (YAML; validate: non-empty id/question, unique ids, surface ∈ {public,internal}). Pure + tested.
- Config: add to RAGQuality: ChatbotURL string (default http://192.168.83.20:3500), GoldenMinPassRate float64 (default 0.8), GoldenPassThreshold float64 (default 0.7). Validate when enabled: rates ∈ [0,1]. (These are only used by goldeneval; harmless when unset.)
- deploy/golden/golden-set.example.yaml: ~6-8 EXAMPLE public items across badan-hukum-pt / yayasan / apostille, each verified: false, with a header comment: "EXAMPLE ONLY — not authoritative; AHU SMEs must verify + expand. See docs."
- [ ] tests (config defaults/validation; goldenset loader incl. dupe-id + bad-surface errors; a DB-gated store round-trip) → implement → go test -p 1 ./... (dind DB) green → commit feat(golden): results store, golden-set schema/loader, config, example set.

Task 2: golden judge scorer

Files: internal/ragquality/judge.go (+ judge_test.go).
- Add type GoldenScore struct { Correctness float64; Passed bool; Reason string } and func (j *Judge) ScoreGolden(ctx, itemID, question string, expected GoldenExpected, answer string) (GoldenScore, error).
- Deterministic pre-check FIRST (pure helper checkFacts(expected, answer) (autoFail bool, reason string)): case-insensitive substring test — any must_include phrase absent → autoFail; any must_not_include phrase present → autoFail; reason names the offending phrase. If autoFail, still call the judge for a correctness score + reason (so the dashboard shows how far off), but force Passed=false.
- Judge call: reuse the gateway POST machinery + enable_thinking:false + idem-key golden-<itemID> (the runner passes a run-scoped id). GOLDEN rubric (system): "Compare ANSWER to the EXPECTED answer for an Indonesian AHU question. Score correctness 0..1 (does the answer convey the expected facts, no contradictions). Output strict JSON: {\"correctness\":0.0,\"passed\":true,\"reason\":\"<=200 chars\"}." Parse leniently (reuse the balanced-brace extractor). Passed = judge-passed AND correctness ≥ GoldenPassThreshold AND NOT deterministic-autoFail.
- [ ] tests (fake gateway): correct answer → passed/high correctness; missing must_include phrase → autoFail even if judge says pass; must_not_include present → autoFail; non-2xx/unparseable → error (retryable vs terminal like phase-1). Commit feat(golden): on-prem golden judge scorer + deterministic fact gate.

Task 3: cmd/goldeneval runner + SME guide

Files: cmd/goldeneval/main.go, docs/golden-set-guide.md, (wire example set path).
- Flags: -set <golden.yaml> (default deploy/golden/golden-set.example.yaml), -config <observatory.yaml> (for DB DSN + rag_quality/judge/chatbot config), -run-id <optional> (else a timestamp-based id passed in — NOTE: time.Now is fine in a CLI, unlike the resumable workflow runtime).
- For each public item: POST {ChatbotURL}/api/orchestrate {surface:"public", sessionId:"golden-<run_id>-<item_id>", message: question}, read the SSE stream, concatenate event: content deltas into the answer (bounded read + timeout). Then checkFacts + ScoreGolden → a GoldenRow; store.InsertGolden. internal items → skip + log.
- After the run: print a table (item, pass/fail, score, short reason), the pass-rate, and a regression list (items that passed in the previous run for the same item_id but fail now, via GoldenItemHistory). Exit non-zero if pass-rate < GoldenMinPassRate OR any verified:true item regressed. Print a clear banner if the set contains verified:false items ("NON-AUTHORITATIVE example set — results are illustrative").
- Robust: a chatbot/judge failure on one item → that item recorded judge_status=error, run continues; never panics.
- docs/golden-set-guide.md: what a golden item is, how SMEs author/verify (must_include = load-bearing facts, must_not_include = known-wrong claims, rubric = nuance), how to run (goldeneval -set ...), how the CI gate works, and the internal-data-agent-drive caveat.
- [ ] tests: drive against an httptest fake chatbot (canned SSE) + a fake judge → rows written, summary + exit code correct; a failing item → non-zero exit; example-set banner shown. Commit feat(golden): goldeneval runner (drive public chatbot, score, gate) + SME guide.

Task 4: API + dashboard "Golden set" panel

Files: internal/api/quality.go + server.go (route); dashboard src/lib/api.ts, src/views/QualityView.tsx (+tests).
- GET /api/quality/golden (rolesCalls): latest run — {run_id, run_at, passed, total, pass_rate, items:[{item_id, domain, question_preview, score, passed, reason, judge_status}], has_unverified bool}. Empty → {total:0,...} honest.
- Dashboard: a "Golden set" section on the RAG Quality view — a pass-rate tile (toned good-when-high via the existing highIsGoodTone) + a per-item pass/fail DataTable (question, score, pass badge, reason). A visible "example set — not authoritative" note when has_unverified. Honest empty state ("No golden run yet — run goldeneval").
- [ ] tests: api (empty → 200 zeroed; seeded run reflected; role gate); dashboard (renders tiles/table from mocked golden summary incl. unverified banner + empty state; the /api/+group_by copy regex assertions). Gates: go test -p 1 ./...; dashboard npx tsc --noEmit && npx vitest run && npm run build. Commit feat(golden): /api/quality/golden + dashboard Golden-set panel.

Post-execution (controller)

  1. Whole-branch adversarial review (focus: deterministic-gate correctness, judge retryable/terminal reuse, runner never-crashes + honest exit code, non-authoritative flagging end-to-end, dormancy).
  2. Merge → master (observatory + dashboard), tag obs-golden-v1.
  3. Deploy observatory (migration auto-applies) + dashboard. Run goldeneval once against the live public chatbot with the EXAMPLE set → confirm rows + dashboard panel + exit-code behavior (expect some example items to fail — they're illustrative). Do NOT present example results as a quality verdict.
  4. Update handoff + memory ([[rag-quality-monitoring-next]] phase-2b). Hand the SME guide to Efran for real content.