Completion notes — Audit Hardening (2026-07-07)

Spec: specs/2026-07-07-audit-hardening-design.md · Plan: plans/2026-07-07-audit-hardening.md
Source prompt: https://x056.think.val.id/prompt-ai-ahu-chatbot-audit-hardening.md

All four items shipped, TDD throughout, verified against the installed pins
(agno 2.4.7 + openai 2.16.0) in a dedicated scratch venv — not assumed.

1. Embedder tenant attribution ✅

apps/{internal,public}-agent/dash/embedder.py — gateway branch now passes
client_params={"default_headers": gateway_default_headers("interactive")}.
Verified plumbing: agno 2.4.7's OpenAIEmbedder.client_params merges into
both the sync and async OpenAI client constructors (empirical). Priority
interactive — the instance serves query-time knowledge search while a user
waits (offline reingest shares it; commented). Legacy no-EMBEDDER_BASE_URL
branch stays byte-identical. Tests assert X-Tenant-Id on the constructed
client; the test loader registers a stub dash package + real gateway.py
by file path so the heavy dash/__init__ never runs.

2. 403 → friendly degradation (public) ✅

• LlmHttpError gains code (parsed from the JSON error body:
  error.code | code) — carries EXTERNAL_UPSTREAM_FORBIDDEN for
  observability.
• orchestrate.ts: 403 from LlmHttpError (non-stream) or
  ComposeUpstreamError (mid-stream event: error) → SSE error
  "Layanan AI sedang tidak tersedia untuk permintaan ini." Tests assert
  no orchestrator failure: leak and no raw code string in the frame.

3. Staff surface degradation ✅ (best-effort within agno, as prompted)

• Not forced: agno agents still don't opt into X-Queue-Events — a
  client-level header would leak foreign event: queue SSE frames into
  openai-python's chunk parser. Documented in
  packages/streams/src/friendly-errors.ts for the future agno upgrade.
• Bounded retry already exists beneath agno: openai-python retries twice
  with exponential backoff honoring Retry-After (verified on 2.16.0) before
  RunError surfaces. Documented, not reimplemented.
• New friendlyAgentErrorMessage() maps 429/rate-limit/503 shapes →
  "Sistem sedang sibuk. Silakan coba lagi sesaat lagi." and
  403/EXTERNAL_UPSTREAM_FORBIDDEN → "Layanan AI sedang tidak tersedia untuk
  permintaan ini." Applied at both NativeProvider failure sites (non-ok run
  POST; RunError events). Raw text preserved in console.error; unrelated
  errors pass through unchanged. Benefits both surfaces (shared package);
  staff /app/data was the target.

4. Per-turn trace_id — FEASIBLE, implemented (the prompt's "attempt" landed)

The finding the prompt asked for: agno 2.4.7 has no per-run header API, but
openai-python re-evaluates the default_headers mapping on every request
build — verified empirically on the pinned openai 2.16.0 (and 2.44.0), for
chat and embeddings. So:

• dash/gateway.py: request_id_var (ContextVar) + GatewayHeaders
  (lazy Mapping: static identity + X-Request-Id when the var is set) +
  RequestIdMiddleware (pure ASGI, framework-free) seeding the var from the
  incoming X-Request-Id. gateway_default_headers() now returns
  GatewayHeaders, upgrading all existing call sites (3 OpenAILike models,
  knowledge client, embedder) with zero further changes. One
  add_middleware(RequestIdMiddleware) line in each agent's app/main.py.
• Senders: DataTool (public orchestrator, server-side) forwards the turn's
  existing id via a new optional requestId ctor arg —
  turnRequestId(req) = ${sessionId}-t${turn} shared with buildLlms, so
  the orchestrator↔agent fan-out for one turn joins one trace.
  NativeProvider (staff, browser → same-origin proxy → agent) sends the
  same derivation; the proxy forwards headers verbatim, no CORS impact.
• Guard: test_openai_client_resolves_request_id_per_request drives the real
  pinned client's _build_request across ContextVar changes — an openai
  upgrade that snapshots the mapping fails the image's build-time pytest
  instead of silently reverting to random trace ids.
• Caveats documented in-code: ContextVar propagates through asyncio tasks
  (agno's async run path) and anyio to_thread; a raw ThreadPoolExecutor
  hop would drop it (header simply omitted → gateway assigns, as today).
  No-header requests always reset the var (no cross-request leakage).

Verification

• Workspace: typecheck clean; 139 tests green (streams 16, public-web 103,
  internal-web 20); check-conventions.mjs OK.
• Agents: 13/13 DB-free tests per app on the pinned venv; mirrors
  byte-identical (gateway.py, embedder.py, agents.py, knowledge.py,
  tests); both main.py carry the middleware (files are variant-specific,
  same addition applied).
• Dormancy: gateway env unset ⇒ unchanged behavior; all additions are
  additive X-* headers (ignored by OpenAI/vLLM) or error-message copy.

Interpretations (prompt asked)

1. Any 403 from the LLM endpoint gets the friendly copy (code distinguishes
   on-prem refusal on the error object; users never see raw codes).
2. Embedder priority interactive (query-time dominant use; reingest shares).
3. Staff "bounded retry honoring Retry-After" = the openai SDK's built-in
   retry, documented rather than duplicated at another layer.
4. NativeProvider's X-Request-Id also flows on the public doc-RAG surface
   (ai-ahu-document-rag backend ignores it harmlessly — additive).

Out of scope (unchanged)

Synthesis on-prem move = gateway-side deploy/gateway.yaml decision (Efran).
This repo now degrades cleanly the moment allow_external_upstreams flips off.