Spec — Gateway integration prep (dormant) for ahu-gpu-manager

Date: 2026-07-04
Source: integration prompt https://x056.think.val.id/prompt-ai-ahu-chatbot-filled.md
(filled variant; gateway live at http://192.168.83.20:8200).
Canonical contract: ../ahu-gpu-manager/docs/CONVENTIONS.md v1.0 — on any
conflict, that file wins.

Goal

Prepare every model-egress path in this repo for the central GPU/LLM gateway
while staying backwards-compatible and dormant: with no new env vars set,
behavior is byte-for-byte identical to today (modulo the explicitly sanctioned
new static identity headers, which current vLLM/DashScope ignore).

Requirements

1. Agno agent headers — apps/{internal,public}-agent/dash/agents.py:
   models built by _resolve_model() / _resolve_synthesis_model() /
   _build_model_from_slot() (OpenAILike) send
   X-Tenant-Id: ahu-chatbot, X-Surface (from SURFACE env),
   X-Priority: interactive. Verified against agno==2.4.7 (the requirements
   pin shipped in the images): OpenAILike supports client-level
   default_headers; there is no per-run header hook, so per-run
   user_id → X-User-Id is impossible today → # TODO(gateway) note.
2. Close bare-OpenAI egress — apps/{internal,public}-agent/app/api/knowledge.py:421
   constructs OpenAI() calling gpt-4o (api.openai.com, bypasses everything).
   Reroute via gateway-resolution logic; model from new env
   KNOWLEDGE_LLM_MODEL (default gpt-4o).
3. Queue-awareness — apps/public-web/src/lib/orchestrator/llm/client.ts:
   - 429/503 → structured error carrying Retry-After / X-Queue-Depth so
   callers degrade with a friendly "sistem sedang sibuk" message.
   - Opt-in LLM_QUEUE_EVENTS=on → send X-Queue-Events: on on streaming
   requests; parse event: queue frames ({"position":N,"eta_ms":M})
   arriving before the first chat chunk; surface to the browser SSE
   status channel; handle post-commit event: error frames; ignore unknown
   SSE event types.
   - Deterministic Idempotency-Key on every request (never
   random-per-attempt).
4. Env prep (commented, not active) — commented flip blocks in
   infra/env/{internal,public,shared}.env (+ committed *.env.example
   per repo mandate), incl. note that SYNTHESIS_API_KEY becomes obsolete
   after the flip. Canonical model names stay as-is (gateway forwards bodies
   verbatim).
5. Verification — suites green, type-check/lint clean, dormancy confirmed,
   optional safe live check against the real gateway.

Decisions (with Efran, 2026-07-04)

• knowledge.py gating: MODEL_GATEWAY_URL is already set in staging
  env files (pointing at plain vLLM), so "reroute when MODEL_GATEWAY_URL set"
  is not dormant. Decision: gate the reroute on the new env
  KNOWLEDGE_LLM_MODEL being set (Efran, option 2). Unset → bare OpenAI()
  byte-for-byte. KNOWLEDGE_LLM_MODEL=Qwen/Qwen3.6-35B-A3B-FP8 goes in the
  commented flip block.
• Queue status wording: prompt says "Antrian ke-N (±Xs)"; CONVENTIONS §3.5
  says "Dalam antrean — posisi N (±X menit)". Conventions win →
  Dalam antrean — posisi N (±M menit) (minutes rounded up, min 1).
• Queue-event surfacing: implemented as a typed event stream
  (LlmClient.streamEvents()), not a bare callback — queue frames arrive
  while the synthesis stream is held, and only a yielding generator can reach
  the browser SSE channel before the first content chunk. stream() remains
  as a delta-only wrapper for compatibility.
• X-Request-Id: added to the orchestrator's base headers
  (${sessionId}-t${turn}) — required by CONVENTIONS §2/§8; one turn = one
  trace.
• Legacy X-Priority: planning|synthesis in orchestrate.ts stays — the
  gateway maps legacy values during transition (§2); not part of this change.
• Python placement: new light module dash/gateway.py (no agno/DB
  imports); tests load it by file path like tests/test_embedder.py does,
  because dash/__init__.py imports the heavy dash.agents.

Constraints

• Engine mandate: changes dormant until env flip; no live env value changes —
  commented entries only. Mirrors (internal-agent ↔ public-agent) stay
  byte-identical. pnpm check + agent pytest suites stay green.