LLM Gateway — Priority Queue + Request Log (2026-07-04)

A thin OpenAI-compatible proxy in front of the model backends. Two jobs only:
bound GPU contention with priority queueing, and emit one structured log line
per LLM request (the seam later systems — chat auditing, usage metering —
build on). A component, not a platform.

Why now

All LLM traffic already flows through two env-var URLs
(MODEL_GATEWAY_URL → local ahu-vllm:8000, SYNTHESIS_GATEWAY_URL) and
every caller already sends X-Tenant-Id / X-Surface / X-User-Id /
X-Priority headers (Plan C). Inserting the gateway is an env flip; removing
it is the same flip back.

Component

apps/llm-gateway — Python 3.12, FastAPI + httpx, ~250 lines.

• Passthrough: POST /v1/* (chat/completions, completions, embeddings)
  proxied verbatim to the default upstream (GATEWAY_UPSTREAM_URL, the local
  vLLM). POST /synthesis/v1/* proxied to GATEWAY_SYNTHESIS_UPSTREAM_URL
  with the synthesis API key injected server-side. Response streaming (SSE)
  passes through chunk-for-chunk.
• Queue: a global slot pool per upstream (GATEWAY_MAX_CONCURRENT,
  default 16 for vLLM; synthesis pool default 8). A request holds its slot
  until the upstream response — including the full stream — completes (that
  is what occupies the GPU). When the pool is full, waiters queue in priority
  order, FIFO within a class. GATEWAY_QUEUE_TIMEOUT_S (default 120) →
  503 {"error":"queue_timeout"}.
• Priority classes from X-Priority: background → class 1; anything
  else (planning, synthesis, missing) → class 0 (interactive). Known
  limitation, accepted: requests the agents make on behalf of background jobs
  arrive without the header and ride as interactive — per-run propagation
  through agno is out of scope.
• Request log: one JSON line to stdout per request:
  {ts, method, path, upstream, priority, tenant, surface, user, model, status, queue_ms, upstream_ms, streamed} — no prompt/response bodies (that
  is the chat-audit system's job, designed separately).
• Health: GET /healthz → {ok:true, in_flight, queued} per pool.

Deployment

• New service ahu-llm-gateway in infra/compose.shared.yaml, image built +
  tested + shipped by build-and-ship.sh like the other five, port
  192.168.83.20:8200 → 8000, on ahu-net (reaches ahu-vllm by docker
  DNS).
• Env flips in public.env + internal.env (+ examples):
  MODEL_GATEWAY_URL=http://ahu-llm-gateway:8000,
  SYNTHESIS_GATEWAY_URL=http://ahu-llm-gateway:8000/synthesis.
  Gateway's own env carries the real upstream URLs + synthesis key.
• Rollback: flip the two env vars back to the direct upstreams; the
  gateway container can stay up idle.

⚠ Recorded finding (decision deferred to the owner)

The live SYNTHESIS_GATEWAY_URL points at an Alibaba Cloud MaaS endpoint
(*.aliyuncs.com, model qwen3.5-397b-a17b) — an external API in the
production path. The on-prem rule as written names OpenAI/Anthropic, so the
convention gate does not flag it, but the spirit is debatable. The gateway
makes this traffic visible (every synthesis call logged) and gives a single
choke point to cut it later; this spec does not change the upstream itself.

Testing

• Unit (pytest, in-image, runs in the ship gate): priority ordering (a
  background waiter never passes an interactive waiter), FIFO within class,
  slot release on stream completion and on upstream error, queue timeout.
• Live: flip envs on staging, run a public Tanya aggregate + a staff Tanya
  Data question end-to-end, confirm gateway log lines show both tiers and
  queue_ms≈0 at idle.

Out of scope

• Prompt/response body capture (chat-audit system, separate spec).
• Per-model routing tables, retries, failover, token metering.
• Changing the synthesis upstream (finding recorded above).