Integration prompt — ai-ahu-chatbot (audit-hardening / post-flip)

Paste into a Claude Code session inside the ai-ahu-chatbot repo. This is a follow-up to prompt-ai-ahu-chatbot.md — the initial gateway integration is DONE and LIVE. A cross-repo audit against the real gateway code found the core public egress path is wire-correct (headers, event: queue/position+eta_ms SSE parse, event: error, 429/503 degradation, deterministic idempotency, model ids all match). This prompt closes the audit-completeness + graceful-degradation gaps it found. None require a gateway change.

Hard constraint (same as before): every change is backwards-compatible and DORMANT — with the gateway env unset, behavior is byte-for-byte identical to today. Verify every agno/openai API against the INSTALLED versions (agno 2.4.7 is pinned; do not assume newer behavior).

Out of scope for you (a gateway-side config decision Efran owns): synthesis currently routes through the gateway to a class: external_dev DashScope cloud model, allowed only because the live gateway has allow_external_upstreams: true. Moving synthesis to an on-prem model for production is a deploy/gateway.yaml change on the gateway side — NOT a chatbot change. Your only related task here is #2 (degrade gracefully on the 403 that on-prem enforcement will return).

1. Attribute embedding calls to ahu-chatbot (audit-completeness — Important)

apps/internal-agent/dash/embedder.py + apps/public-agent/dash/embedder.py — build_embedder() constructs the OpenAIEmbedder with base_url from EMBEDDER_BASE_URL but sends no identity headers, so query-time embedding traffic through the gateway is audited as tenant unknown instead of ahu-chatbot. The RAG retrieval step is invisible in the audit trail by tenant.

• Pass the existing gateway_default_headers(...) (from dash/gateway.py:13) into the embedder's underlying OpenAI client (verify the installed OpenAIEmbedder accepts default_headers / client kwargs — agno's OpenAIEmbedder wraps the openai client which takes default_headers; confirm the exact plumbing, do not guess). Use X-Priority: interactive (a user is waiting on the retrieval) — or batch if the embedder is only used in an offline reingest path; pick per how it's actually called.
• Dormant-safe: only add headers. When EMBEDDER_BASE_URL is unset (legacy OpenAI fallback), the added X-* headers are ignored by OpenAI — behavior unchanged. Add a test asserting the constructed embedder carries X-Tenant-Id: ahu-chatbot.

2. Degrade gracefully on 403, not just 429/503 (Important)

apps/public-web/src/lib/orchestrator/llm/client.ts throws LlmHttpError on 429/503 (with Retry-After/X-Queue-Depth) and the orchestrator degrades to "sistem sedang sibuk". But a 403 ({"error":{"code":"EXTERNAL_UPSTREAM_FORBIDDEN"}}, which the gateway returns when on-prem enforcement refuses an external_dev upstream — i.e. the moment synthesis is prod-hardened) is NOT handled, so it surfaces to users as a raw orchestrator failure: ... string.

• Handle 403 in the LlmClient / orchestrate.ts / steps/compose.ts path: surface a clean, user-facing degradation (e.g. "Layanan AI sedang tidak tersedia untuk permintaan ini") via the existing SSE error/status channel, not a raw error. Read the error.code if present to distinguish an on-prem refusal from other 4xx.
• Add a test: a 403 from the LLM endpoint yields the friendly degradation, no raw-error leak to the browser stream.

3. Internal/staff surface — degrade under saturation (Important, best-effort within agno)

The staff surface (apps/internal-web + apps/internal-agent agno agents) has no queue UX: agno agents don't opt into X-Queue-Events, so under gateway saturation staff get a dead spinner then a generic error.

• Live queue-position UX may not be feasible through agno's SDK-managed client — do not force it. The realistic, valuable fix: ensure a gateway 429/503/403 surfaced by the agent path degrades to a clear staff-facing "sistem sedang sibuk / tidak tersedia, coba lagi" state (a bounded retry with backoff honoring Retry-After, then a friendly message), instead of a spinner-to-generic-error. Investigate whether agno surfaces the upstream HTTP status to the caller; wire the busy/unavailable state off that.
• Document in-code what agno 2.4.7 does and doesn't allow here, so a future agno upgrade can add real queue frames.

4. Per-turn trace_id for agent calls (Important — attempt, else document)

Agno 2.4.7 can't set per-run headers, so the internal/public agents send a client-level static header set with no per-turn X-Request-Id → the gateway assigns a random trace_id per agent call, and the orchestrator↔agent fan-out for one turn can't be reconstructed in the audit trail.

• Attempt a per-turn X-Request-Id: check whether constructing the OpenAILike/model with a per-request default_headers (a fresh lightweight client per turn carrying that turn's request id) is feasible without a heavy refactor, or whether a newer agno affords per-run headers. The request id should match the one the orchestrator already generates for the turn (so agent calls join the same trace).
• If genuinely infeasible in agno 2.4.7: document the limitation clearly and do the minimal improvement — at least a stable per-session id in the header set (better than fully random per-call for grouping). Do not over-engineer; a documented constraint + per-session grouping is an acceptable landing.

Verification

• Run the existing suites (TS + Python); nothing regresses. Type-check + lint clean.
• Dormancy: with gateway env unset, confirm request URLs/bodies/headers (minus the new additive X-*) are unchanged.
• New tests: embedder carries X-Tenant-Id; 403 → friendly degradation (public); staff path degrades on 429/503/403.
• Report back: files changed, the agno per-run-header finding (feasible or documented-limitation), and anything in the header/degradation contract that needed interpretation.