Chatbot ↔ GPU/LLM-gateway integration review

Scope: cross-repo wire-compat audit of ai-ahu-chatbot (engine, READ-ONLY) against
ahu-gpu-manager (authoritative gateway). Chatbot is LIVE (flipped 2026-07-04, staging).
Any fix lands gateway-side (config or code); no chatbot edits proposed here.

Bottom line: the public Tanya egress path is wire-correct (headers, SSE queue/error
frames, 429/503, idempotency, model ids all parse against the real gateway). But there are
two live production issues (external cloud egress of user data; embeddings audited as
unknown) and three completeness gaps (staff-surface queue UX, agent-path trace
correlation, latent prod 403 on synthesis). Only one is a gateway-side fix — a
route_prefixes.synthesis config change; the rest are engine/library limitations.

Compatibility matrix

Findings

A. CRITICAL (LIVE) — user data egresses to Alibaba DashScope on every public synthesis; latent prod 403

route_prefixes.synthesis → ext-dashscope-397b (deploy/gateway.yaml:10-11), class: external_dev,
endpoint …maas.aliyuncs.com (:35-41). Every public-Tanya answer composition (compose.ts →
SYNTHESIS_GATEWAY_URL → /synthesis/v1) ships the user question plus retrieved RAG passages /
DB tool results (compose.ts:37-67) to Alibaba cloud.

• Right now (staging): works because the deployed config has allow_external_upstreams: true
  (gateway.yaml:2). This is a direct violation of the on-prem constraint ("cloud is dev-phase
  only"; chatbot CLAUDE.md "No external API calls in production paths"). It is a live
  data-egress issue, not a latent one.
• On prod flip (allow_external_upstreams: false): the gateway returns
  403 EXTERNAL_UPSTREAM_FORBIDDEN before admission (handler.go:188-192) → synthesis hard-fails
  for every turn.

Assessment: config/deploy, not a code bug. Failure scenario: with allow_external_upstreams=false,
a user asks any question → planning + RAG succeed → compose POSTs to /synthesis/v1 → 403 → no answer.

Gateway-side fix: register an on-prem synthesis upstream and repoint the prefix, e.g.

route_prefixes:
  synthesis: qwen-35b        # or a dedicated on-prem large model when B200 lands

Until an on-prem "big" model exists, synthesis should fall back to qwen-35b (already registered,
on_prem) rather than DashScope. Keep DashScope only in a dev config with allow_external_upstreams: true.

B. IMPORTANT — chatbot doesn't gracefully degrade on 403 (compounds A)

orchestrate.ts:217-223 degrades only on status===429||503. A synthesis 403
(EXTERNAL_UPSTREAM_FORBIDDEN) falls through to the generic orchestrator failure: LlmHttpError…
branch (:225-227) — the user sees a raw error string, not "sistem sedang sibuk". Engine-side; only
bites once A's prod flip happens. Noting so the gateway team knows a clean 403 won't be handled
prettily downstream. (No gateway fix; fixing A removes the trigger.)

C. IMPORTANT (LIVE) — embedding calls audited as tenant unknown

build_embedder() constructs OpenAIEmbedder(base_url=EMBEDDER_BASE_URL, …) with no
default_headers (embedder.py:17-22). Query-time knowledge/learnings search embeds through the
gateway (…:8200/v1/embeddings) with no X-Tenant-Id/X-Surface/X-Priority. Gateway routing still
works (model Qwen/Qwen3-Embedding-4B resolves to tei-embeddings), but MetaFrom sets tenant
unknown (handler.go:70-72), so operation:embed audit events are not attributed to
ahu-chatbot. Live now (trust-mode tokens empty, so unknown isn't rejected).
Not gateway-fixable — the client sends no header. Engine follow-up: pass
default_headers=gateway_default_headers("interactive") to OpenAIEmbedder (verify the ctor accepts
it in the installed agno).

D. IMPORTANT — Agno agent calls have no per-turn trace_id (audit tree broken at orchestrator↔agent boundary)

Agno 2.4.7 exposes only client-level default_headers; Agent.run(user_id=…) never reaches the HTTP
layer (agents.py:36-40). The agents therefore send no X-Request-Id and no X-User-Id, so the
gateway generates a fresh random trace_id per agent call (handler.go:73-74). The orchestrator's
own planning/synthesis calls carry X-Request-Id=${sessionId}-t${turn}, but the public_dash /
data-agent tool calls they fan out to (each of which independently hits the gateway for SQL
planning + output_model synthesis) land under unrelated trace_ids. A single user turn's audit
fan-out cannot be reconstructed. Confirmed known gap; blocked on upstream agno. Not gateway-fixable
today. (Note: also means the internal-agent's output_model synthesis — when configured via
SYNTHESIS_GATEWAY_URL — sends X-Priority: interactive and no trace, same as planning.)

E. IMPORTANT — staff/internal surface has no queue UX and no 429 degradation (flip-day gap, still OPEN)

Queue narration only exists in the TS orchestrator (client.ts streamEvents + compose.ts +
orchestrate.ts:209 queueStatus), used by public Tanya. Staff Tanya Data (apps/internal-web)
streams the Agno agent directly via NativeProvider. The agents never send X-Queue-Events: on
(gateway.py:13-25 omits it), so under saturation the gateway holds silently to the hold budget
(45 s interactive) then returns 429 (handler.go:211-219) — which the Agno client raises as a
plain error. Staff users get a spinner-then-generic-error, never "Dalam antrean — posisi N"
(violates CONVENTIONS §3.5) and no "sistem sedang sibuk". Engine-side; not gateway-fixable.

F. MINOR — knowledge-template call has no trace/idempotency/user headers

knowledge.py:421-431 via knowledge_llm_config() sends X-Tenant-Id/X-Surface/X-Priority but no
X-User-Id/X-Request-Id/Idempotency-Key (gateway.py:43-47). Audit event is attributed to
ahu-chatbot (good) but with a server-generated trace_id. Acceptable for a low-volume admin
template-generation call; non-streaming, so idempotency is optional. No action required.

The two earlier flip-day gaps — status

• Internal/staff surface has no queue UX → STILL OPEN (finding E). Impact: on GPU
  saturation, staff see a dead spinner then a generic failure. Cosmetic/UX under load, not data loss.
• X-Request-Id not end-to-end → STILL OPEN (finding D), root-caused to agno 2.4.7. Impact:
  audit attribution/tracing is per-call-random for all agent traffic; per-turn correlation exists
  only for the TS orchestrator's direct planning/synthesis calls.

Verdict

(a) Is the chatbot integration correct/complete?
The public Tanya egress path is wire-correct and complete — header set, X-Priority
legacy-mapping, the position/eta_ms queue frame, the {status,code} error frame, : ka
keepalive tolerance, 429/503 graceful degradation, deterministic idempotency keys, and all four
model ids resolve against the real registry. Incomplete on: embedder identity headers (C),
agent-path trace correlation (D, agno-blocked), staff-surface queue UX (E), and 403 handling (B).

(b) Is the gateway compatible as-is, or does it need changes?
Wire-compatible as-is — the gateway's frames/headers/model resolution match exactly what the
chatbot sends and parses; no proxy code change is required for correctness. The one required
gateway-side change is configuration: route_prefixes.synthesis must point at an on_prem
upstream before allow_external_upstreams is set false for prod (finding A), otherwise synthesis
403s on every turn. Findings C/D/E cannot be fixed gateway-side (the clients don't send the needed
headers / don't opt into queue events).

(c) Any LIVE production issue right now?
Yes, two:
1. On-prem violation (A): every public-Tanya synthesis is shipping the user's question + retrieved
RAG/DB content to Alibaba DashScope, live, because the deployed gateway config has
allow_external_upstreams: true. This is the "cloud is dev-phase only" constraint being violated
in a live surface. Repoint route_prefixes.synthesis to an on-prem model.
2. Audit blind spot (C): all embedding traffic is recorded under tenant unknown, so
embed-operation volume/attribution for the chatbot is missing from the audit stream.

The prod-flip 403 (A, latent) and the two open flip-day gaps (D, E) are correctness/observability
debt to close before prod, but are not breaking staging today.