Audit guardrail refusals as status: "refused" — completion notes

Date: 2026-07-09
Spec: specs/2026-07-09-refusal-audit-status-design.md
Plan: plans/2026-07-09-refusal-audit-status.md
Status: implemented, gated green, deployed + flipped ON + verified
end-to-end (2026-07-09).

What changed

• packages/orchestrator-types/plan.ts — RefusalCode union + optional
  code on Refusal.
• guards/input-patterns.ts — now imports the canonical Refusal (removed
  the duplicate local interface); sets GUARDRAIL_INPUT_PII for PII patterns,
  GUARDRAIL_INPUT_ENTITY for entity/off-topic/custom blocks.
• steps/plan.ts — NO_TOOLS, GUARDRAIL_PLAN_REFUSED, PLAN_UNPARSEABLE.
• lib/audit/emit.ts (new) — buildRefusalEvent (pure, schema-v1) +
  emitRefusalAudit (fire-and-forget XADD to ahu.ai.audit).
• orchestrate.ts — captures turnStartMs; emits at the isRefusal branch
  (surface, user_id=req.sessionId, trace_id=turnTraceId, code,
  total_ms, config_version=policy version) BEFORE streaming the refusal copy;
  void … .catch(()=>{}) so it can never delay or break the stream.
• infra/env/public.env.example — AUDIT_REDIS_URL (dormant) + AUDIT_STREAM.
• infra/compose.public.yaml — public-web joins external
  ahu-platform_default (verified present on Server 2; holds ahu.ai.audit).

Design choices / findings

• One choke point. All five refusal paths flow through plan() → the
  isRefusal branch, so a single emit covers them. No per-path emit.
• Emitter placement. public-web-only today → lives in the app
  (apps/public-web/src/lib/audit/), not a package, per the boundary rule.
• Separate Redis. AUDIT_REDIS_URL is a distinct instance from the
  anon-limits REDIS_URL — not shared. ahu-platform-redis (network
  ahu-platform_default) confirmed as the holder of ahu.ai.audit; flip value
  is redis://ahu-platform-redis:6379/0.
• Never PII. The event carries only metadata — no request_body/response_body
  and no message text (asserted in tests).
• event_id = uuidv7 (uuid v13's v7), never blank.

Out of scope (flagged)

The staff/internal native-Agno path has no structured refusal signal
(refusals there are prompt-driven text), so it cannot emit refused events yet.
Not attempted — we will not string-match agent output. Follow-up ticket if staff
refusal attribution is wanted.

Tests

• guards.test.ts — PII→GUARDRAIL_INPUT_PII, entity/off-topic→GUARDRAIL_INPUT_ENTITY.
• plan-refusal-codes.test.ts — NO_TOOLS / GUARDRAIL_PLAN_REFUSED / PLAN_UNPARSEABLE.
• audit/emit.test.ts — schema shape + no body columns; dormant (no redis touch)
  when unset; exactly one XADD with correct stream/fields + fail-fast client
  opts; custom AUDIT_STREAM; never throws on Redis failure.
• orchestrate-refusal-audit.test.ts — one schema-v1 event with trace_id =
  turn X-Request-Id, user_id = session, no message text; dormant when unset;
  full refusal stream survives a dead Redis; non-refusal turn emits nothing.

Gate: pnpm check green (public-web + internal-web 65 / streams 17 + typecheck +
conventions).

Deploy + flip + verification — DONE 2026-07-09

Rebuilt ahu-ai-chatbot-public (0bb0aee), recreated ahu-public (joined
ahu-platform_default, resolves ahu-platform-redis), and flipped
AUDIT_REDIS_URL=redis://ahu-platform-redis:6379/0 on Server 2's live
public.env. All four acceptance checks pass:

1. ✅ Blocked query on public Tanya (NIK / "siapa direktur … alamat …") → the
   normal refusal copy streams (UX unchanged).
2. ✅ docker exec ahu-platform-redis redis-cli XREVRANGE ahu.ai.audit shows the
   events: status:"refused", error_code:"GUARDRAIL_INPUT_PII",
   engine:"ahu-chatbot", surface:"public", correct user_id/trace_id,
   uuidv7 event_id, no message text.
3. ✅ ai_calls: 16 refused rows for the actor, 16 distinct traces, matching.
4. ✅ GET /api/security-warnings (operator token) →
   sec_error_probing fires: "ahu-chatbot/public/: 100% dari 16
   panggilan ditolak/gagal non-infra … pola percobaan menembus guardrail",
   evidence probe_calls:16, total_calls:16, threshold_pct:30.

⚠ Deploy-hygiene caveat (env drift)

The flip was applied in place to Server 2's live infra/env/public.env
(appended AUDIT_REDIS_URL/AUDIT_STREAM) to avoid reverting Efran's earlier
RAG_SEARCH_URL flip, which lives only on Server 2 (not in the git-ignored local
public.env). Consequence: the local public.env is now behind Server 2 by
two keys (RAG_SEARCH_URL, AUDIT_REDIS_URL). Targeted deploys (rebuild image
+ compose up) are unaffected, but a full deploy-staging.sh scps the local env
over Server 2's and would revert both flips. Reconcile by adding these two
keys to the local public.env before any full deploy-staging.sh run.