Golden Q&A Regression Set — SME guide

Audience: AHU subject-matter experts (SABH / Apostille / PP) authoring and verifying the golden Q&A set that goldeneval uses to catch chatbot answer-quality regressions before they reach real users.

Status of the shipped set: deploy/golden/golden-set.example.yaml ships with every item verified: false. It is illustrative structure only — not a quality verdict on the chatbot. A run against it as shipped demonstrates the harness, not the chatbot's quality. Read this guide, then edit that file (or a copy of it) into a real, SME-reviewed set before treating any run's results as meaningful.

See also: docs/superpowers/specs/2026-07-10-golden-eval-design.md (design) and docs/superpowers/plans/2026-07-10-golden-eval.md (implementation plan) for the full architecture this guide's CLI implements.

What a golden item is

One golden item is a single question the public chatbot (/tanya) should be able to answer correctly, plus an SME-authored contract for what "correctly" means:

- id: pt-pendirian-syarat          # stable, unique — the DB key and the judge idempotency key
  surface: public                  # public (driven today) | internal (loaded, not yet driven — see below)
  domain: badan-hukum-pt           # free-text grouping for the report/dashboard
  question: "Apa saja syarat mendirikan PT perorangan?"
  expected:
    must_include: ["WNI", "berusia minimal 17 tahun", "satu orang"]
    must_not_include: ["wajib akta notaris"]
    rubric: >
      Answer should state the founder must be an Indonesian citizen (WNI) of
      legal age, that a PT perorangan has a single founder, and that a
      notarial deed is NOT required (that's the ordinary-PT path).
  verified: false                  # an SME sets this true after review

Every run scores an item two ways, and both must agree for the item to pass:

1. Deterministic fact gate (must_include / must_not_include) — a plain case-insensitive substring check against the chatbot's answer. No judgment call, no LLM involved. This is the harness's floor: a fluent-sounding wrong answer cannot talk its way past it.
2. On-prem LLM judge (rubric plus the same two lists) — scores 0.0-1.0 "correctness" and its own pass/fail verdict, for the nuance a substring check can't express.

A missing must_include phrase or a present must_not_include phrase auto-fails the item regardless of what the judge says — the offending phrase is recorded as the reason.

Authoring and verifying an item

• must_include — the load-bearing facts an answer MUST state. Keep these short, specific, and truly required. A phrase that's merely "nice to have" belongs in rubric, not here — putting it in must_include will fail a correct answer that says the same thing in different words. Because this is a literal substring match, pick a phrase you're confident any correct answer would contain verbatim (a number, a named entity, a required document/term), not a paraphrase-prone sentence.
• must_not_include — known-wrong claims: a common misconception, an outdated rule, a superseded number. If any such phrase appears in the answer, the item auto-fails regardless of the judge.
• rubric — free text handed to the judge alongside the question and the must_include facts. Use it for nuance a substring check can't carry: what the answer should emphasize, what scope is expected, why a plausible-looking answer would actually be wrong.
• verified — false until an AHU SME has reviewed the item: the question is realistic and unambiguous, must_include/must_not_include are correct under current regulation, and rubric is fair. Set it true only after that review. This flag has a real, load-bearing consequence, not just a documentation one — see "The CI gate" below: an unverified item's regression is always visible in the report, but it can never by itself fail the exit-code gate, while a verified item's regression can.
• domain — group related items (e.g. all PT items under badan-hukum-pt) so the report/dashboard can be scanned by area.
• surface — public or internal. See "surface: internal items" below.

Caveat — matching is case-insensitive SUBSTRING matching. Both must_include and must_not_include test whether the phrase appears anywhere inside the answer text (after lower-casing both sides). There are no word boundaries and no stemming, so a short token matches far more than you intend. Two consequences for authors:
- Prefer specific, multi-word phrases a correct answer would contain verbatim (a number, a named entity, a required term) — not a single common word.
- Especially avoid short must_not_include tokens. "PT" matches inside adaptor, kapten, and optik; "tidak" matches inside ketidakpastian — either would auto-FAIL a perfectly correct answer. A must_not_include phrase should be a distinctive multi-word claim (e.g. "wajib akta notaris"), never a common word or a 2–3 letter fragment.

Keep id stable once shipped: golden_results is keyed by (run_id, item_id), and regression detection tracks an item's pass/fail trend across runs by item_id. Renaming an id starts that item's history over from nothing.

Running it

go build -o goldeneval ./cmd/goldeneval
./goldeneval -set deploy/golden/golden-set.example.yaml -config /etc/ahu-observatory/observatory.yaml

Each run: drives every surface: public item against {chatbot_url}/api/orchestrate, scores each answer (deterministic gate + judge), writes one golden_results row per item (including skipped/errored ones), and prints a report — a per-item pass/fail table, the scored pass rate, and any regressions versus the most recent scored run for each item. The report ends with a loud banner whenever the loaded set contains any verified: false item.

A dashboard panel and GET /api/quality/golden also surface the latest run read-only (see the design spec) — this guide focuses on the CLI you actually run.

The CI gate

goldeneval exits non-zero — and prints exactly why, as its last line — when any of these is true:

1. The scored pass rate is below rag_quality.golden_min_pass_rate (default 0.8). The pass rate is passed / scored, where "scored" means items the judge actually evaluated (judge_status: ok). surface: internal items (always skipped) and any item that errored while driving or judging are excluded from that denominator — they can neither dilute nor inflate the signal a pass rate is meant to convey. (A deterministic must_include/must_not_include violation is NOT an error: it is a definitive fail and counts in the denominator, even if the judge call itself was down.)
2. Any verified: true item regressed — the most recent past run that actually scored it (judge_status: ok) passed, and it fails (also with a real ok score) now. A run where the item errored or was skipped (e.g. the judge hit gateway backpressure) is neither a regression nor a baseline: a transient error can never by itself trip this gate. A verified: false item's regression is still printed (so an SME sees the trend) but never fails the gate by itself. That is the whole point of shipping an example set separately from a real one: illustrative results must never be able to block a deploy on their own.
3. The set contains any verified: false item and -allow-unverified was not given. An example/unverified set is not a certifiable quality pass — its expectations are illustrative and may be wrong — so a run against one exits non-zero regardless of pass-rate, with a distinct reason, unless you pass -allow-unverified to acknowledge you are deliberately running it (e.g. a local smoke run). This stops a goldeneval || exit 1 pipeline from going green just because the chatbot "answered the example questions well". A fully-verified set is unaffected.

Per-item pass/fail also depends on rag_quality.golden_pass_threshold (default 0.7): the judge's own pass verdict, AND correctness >= golden_pass_threshold, AND no deterministic auto-fail, must all hold.

Wire the binary into CI/deploy by running it and checking the exit code, e.g.:

./goldeneval -set deploy/golden/golden-set.example.yaml -config deploy/observatory.yaml || {
  echo "golden eval gate failed — see the report above"
  exit 1
}

surface: internal items — not driven yet

Items with surface: internal load and validate normally, but v1 of goldeneval does not drive them: they are recorded judge_status: skipped and never call the chatbot or the judge. This is deliberate, not an oversight. Driving the internal staff surface (Tanya Data / the data-agent) needs two things this phase doesn't build:

1. The internal endpoint + staff auth, wired into the runner. The internal surface is a different backend (ahu-ai-agent-internal's native Agno SSE protocol), not the public /api/orchestrate this runner drives today.
2. Drift-stable question design. The public chatbot answers from relatively stable legal/procedural knowledge; the internal data-agent answers from the live database, so "the correct answer" changes as the data changes. An internal golden item needs to be time-scoped, structural (e.g. "names exactly 3 provinces matching X"), or run against a frozen fixture — otherwise an item that was correct yesterday can fail today for a reason that has nothing to do with answer quality.

Feel free to author internal items now for planning purposes, but don't expect them to run (or gate anything) until that follow-up ships.

Honesty reminder

The example set exists to prove the harness works end to end, not to certify the chatbot. Every one of its items is verified: false. Do not report a goldeneval run against the unmodified example set as a quality result to anyone outside the team building this — expand and verify the set with real AHU SME review first.