Audit Hardening — post-flip gateway follow-up (2026-07-07)

Source: https://x056.think.val.id/prompt-ai-ahu-chatbot-audit-hardening.md
(follow-up to the 2026-07-04 gateway integration; cross-repo audit found the
public egress path wire-correct, these close the audit-completeness +
graceful-degradation gaps). Contract: ../ahu-gpu-manager/docs/CONVENTIONS.md v1.0.

Hard constraint: dormant — gateway env unset ⇒ byte-for-byte identical
behavior (new additive X-* headers excepted). Everything verified against the
installed pins: agno 2.4.7 + openai 2.16.0 (empirically, in a scratch venv).

Out of scope: moving synthesis on-prem (gateway-side deploy/gateway.yaml
decision Efran owns). Only the 403 degradation (#2) relates to it here.

1. Embedder tenant attribution (both agents)

build_embedder() sends no identity headers → query-time embedding traffic is
audited as tenant unknown. Fix: client_params={"default_headers": gateway_default_headers("interactive")} in the EMBEDDER_BASE_URL branch.

Verified plumbing (agno 2.4.7): OpenAIEmbedder.client_params merges
into both OpenAI(**…) and AsyncOpenAI(**…) constructors; headers land on
client.default_headers (empirically confirmed).
Priority = interactive: the embedder instance is wired into
dash_knowledge/dash_learnings used by query-time search_knowledge_base
while a user waits. Reingest jobs share the instance — accepted (comment).
Legacy fallback branch (no EMBEDDER_BASE_URL) stays bare — byte-identical.
Test (file-path loader): constructed embedder carries X-Tenant-Id: ahu-chatbot.

2. 403 → graceful degradation (public orchestrator)

Gateway returns 403 {"error":{"code":"EXTERNAL_UPSTREAM_FORBIDDEN"}} when
on-prem enforcement refuses an external_dev upstream — today that leaks to
users as orchestrator failure: ….

LlmHttpError gains code?: string, parsed from the JSON error body
(error.code | code) for all non-ok responses.
orchestrate.ts catch: status 403 (from LlmHttpError or
ComposeUpstreamError, i.e. non-stream and mid-stream) → SSE error
"Layanan AI sedang tidak tersedia untuk permintaan ini." — no raw leak.
Any 403 from the LLM endpoint gets the friendly copy; code is carried on
the error object for observability, not shown to users.

3. Staff surface degradation under saturation

Agno agents can't opt into X-Queue-Events (client-level headers would leak
foreign event: queue SSE frames into openai-python's chunk parser) → live
queue-position UX is not forced, per the prompt. What ships:

Bounded retry exists already at the SDK layer: openai-python
max_retries=2 with exponential backoff honoring Retry-After (verified
2.16.0) — documented in-code, not reimplemented.
New packages/streams/src/friendly-errors.ts:
friendlyAgentErrorMessage(raw) maps 429/rate-limit/503-shaped messages →
"Sistem sedang sibuk…", 403/EXTERNAL_UPSTREAM_FORBIDDEN → "Layanan AI
sedang tidak tersedia…", else null (raw preserved).
NativeProvider applies it at both failure sites (non-ok HTTP on the run
POST; RunError events mid-stream). Raw message still goes to
console.error for debugging. Both surfaces benefit (shared package), staff
/app/data is the target.
In-code documentation of what agno 2.4.7 does/doesn't allow, for a future
upgrade to add real queue frames.

4. Per-turn `X-Request-Id` for agent calls — FEASIBLE (implemented)

Agno 2.4.7 has no per-run headers, but openai-python re-evaluates
default_headers per request when given a lazy Mapping (verified on
openai 2.16.0 and 2.44.0, chat + embeddings): a ContextVar-backed mapping
gives true per-turn ids through agno's cached client, no per-turn client
construction, no agno internals touched.

dash/gateway.py: request_id_var: ContextVar, GatewayHeaders(Mapping)
(static identity + X-Request-Id from the ContextVar when set);
gateway_default_headers() now returns it → all existing call sites
(3 OpenAILike models, knowledge client, embedder) upgrade automatically.
Plus RequestIdMiddleware — pure ASGI, no framework imports, sets the
ContextVar from the incoming X-Request-Id header. One add_middleware
line in app/main.py (both agents).
Senders: public path — DataTool gains requestId?; orchestrate.ts
passes the turn's existing id (${sessionId}-t${turn}) so agent calls join
the orchestrator's trace. Staff path — NativeProvider sends
X-Request-Id: ${sessionId}-t${history.length} on the run POST (same-origin
proxy forwards headers verbatim; no CORS impact).
ContextVar propagates through asyncio tasks (agno's async run path).
Documented caveat: raw ThreadPoolExecutor hops would drop it (anyio's
to_thread preserves context; acceptable).
A pytest exercises the real pinned openai client's _build_request twice —
so an openai upgrade that breaks the lazy-mapping behavior fails the image's
build-time pytest instead of silently reverting to random trace ids.

Success criteria