Task 12: Live-fire verification against real vLLM upstream (P0 exit) — Report

Date: 2026-07-04
Branch: feat/gateway-p0
Commit: 10608fe — "test(p0): smoke, real-client compat, saturation and restart drills"
Tag: p0-gateway
Real upstream: qwen-35b → http://192.168.83.20:8001/v1, model Qwen/Qwen3.6-35B-A3B-FP8 (confirmed reachable via GET /v1/models before starting)

Status: DONE — all drills passed, P0 exit gate cleared

Setup

cp deploy/gateway.example.yaml deploy/gateway.yaml
# added /deploy/gateway.yaml to .gitignore (runtime config, never committed)
docker compose -f deploy/compose.yaml up -d --build

Build succeeded (Go 1.25-alpine → alpine:3.20 multi-stage, ~33s compile). Containers:

NAME                     STATUS
ahu-platform-gateway-1   Up (0.0.0.0:8200->8200/tcp)
ahu-platform-redis       Up (redis:7-alpine, appendonly yes)

Gateway log on boot: 2026/07/04 08:58:23 gateway listening on :8200 (4 upstreams) — all 4 configured upstreams (qwen-35b, cleanup-3b, tei-embeddings, ext-dashscope-397b) loaded without config errors.

Known, expected non-issues (per task adjustments, not chased):
- DASHSCOPE_API_KEY unset → ext-dashscope-397b (external_dev, DashScope) will fail auth/probes. No drill exercises this upstream. Confirmed harmless: queue/status?upstream=ext-dashscope-397b responds normally (registry entry present; only outbound calls would fail).
- cleanup-3b (:8003) and tei-embeddings (:8100) health state on x056 not verified — out of scope; only qwen-35b needed to be healthy for these drills, and it was.

Step 1 — Smoke test

GW=http://localhost:8200 MODEL="Qwen/Qwen3.6-35B-A3B-FP8" ./scripts/smoke.sh

Result: PASS (all 4 sections succeeded).

Cosmetic script artifact (not a gateway bug): smoke.sh's streaming section pipes to head -20; under set -o pipefail, head closing the pipe early triggers a SIGPIPE in curl, which returns exit 23 (write error) and aborts the script under -e before it reaches the final XLEN line. I isolated this by re-running the identical streaming request without the head truncation:

$ curl -sfN ".../v1/chat/completions" ... | (full SSE stream) ... data: [DONE]
exit: 0

— clean completion, exit 0, [DONE] sentinel received. The gateway's audit record correctly reflects this: the head-truncated run produced an audit event with "status":"error","error_code":"UPSTREAM_200" (client disconnected mid-stream after resp.StatusCode=200; streamErr != nil in handler.go:236-238) — this is correct drop-detection behavior, not a defect: the gateway accurately recorded that the client (curl, killed by head) went away mid-response. Verified against source (internal/proxy/handler.go:227-241).

Audit stream depth: XLEN ahu.ai.audit went from 0 → 2 → 3 (2 events from the truncated smoke run, +1 from the clean re-run) — satisfies "≥ 2" expectation.

Step 2 — Real-client compatibility (openai-python via uv)

uv run --with openai python3 - <<'EOF'
from openai import OpenAI
c = OpenAI(base_url="http://localhost:8200/v1", api_key="EMPTY",
           default_headers={"X-Tenant-Id": "compat-check", "X-Priority": "interactive"})
r = c.chat.completions.create(model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[{"role": "user", "content": "Say OK"}], max_tokens=5)
print("non-stream:", r.choices[0].message.content)
s = c.chat.completions.create(model="Qwen/Qwen3.6-35B-A3B-FP8", stream=True,
    messages=[{"role": "user", "content": "Count to 3"}], max_tokens=20)
print("stream:", "".join(ch.choices[0].delta.content or "" for ch in s if ch.choices))
EOF

Output:

non-stream: Here's a thinking process
stream: Here's a thinking process:

1.  **Analyze User Input:** The user said "

Result: PASS. Both calls succeeded, no SDK parse errors, no exceptions. Confirms the gateway's SSE framing/keepalive/queue-frame handling is fully tolerated by the real openai-python client the engines use (ai-ahu-chatbot, ahu-ocr-akta-notaris, ahu-doc-classifier).

Step 3 — Saturation / loadcheck (created scripts/loadcheck.sh)

Script (as specified in the brief, committed verbatim):

#!/usr/bin/env bash
# Fires 40 concurrent batch requests at a pool of 16; verifies queueing + 429s are clean.
set -euo pipefail
GW="${GW:-http://localhost:8200}"; MODEL="${MODEL:-Qwen/Qwen3.6-35B-A3B-FP8}"
seq 40 | xargs -P40 -I{} curl -s -o /dev/null -w "%{http_code}\n" \
  "$GW/v1/chat/completions" -H 'Content-Type: application/json' \
  -H 'X-Tenant-Id: loadcheck' -H 'X-Priority: batch' \
  -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":8}" \
  | sort | uniq -c

Run:

GW=http://localhost:8200 MODEL="Qwen/Qwen3.6-35B-A3B-FP8" ./scripts/loadcheck.sh

Distribution: 40 200 — all 40 requests returned 200. Zero 429, zero 5xx, zero connection resets. Wall clock: ~0.6s for the whole burst (real vLLM continuous batching on H100 handled the 8-token completions very quickly; the 16-slot / 8-batch_max pool did not need to hold requests long enough to hit the 120s batch budget). This is a PASS per the brief's own framing ("all-200s is a PASS").

gateway_queue_wait_ms histogram (GET /metrics) confirms real queueing/admission activity occurred (not just pass-through):

gateway_queue_wait_ms_bucket{le="10"} 13
gateway_queue_wait_ms_bucket{le="50"} 13
gateway_queue_wait_ms_bucket{le="100"} 13
gateway_queue_wait_ms_bucket{le="500"} 45
...
gateway_queue_wait_ms_sum 8692
gateway_queue_wait_ms_count 45

(count 45 = 5 pre-loadcheck requests + 40 loadcheck requests, cumulative since boot; most of the 40 batch requests queued in the 100–500ms bucket while funneling through the 16-slot pool — mean wait ≈ 193ms across all recorded requests.)

Audit XLEN: 5 → 45 (exactly +40, one event per request, no drops; audit_dropped: 0 in healthz throughout).

Step 4 — Kill-the-gateway drill

docker compose restart gateway & sleep 0.2
curl -s -o /dev/null -w "%{http_code}\n" localhost:8200/healthz || true
sleep 3
GW=http://localhost:8200 MODEL="Qwen/Qwen3.6-35B-A3B-FP8" ./scripts/smoke.sh
docker exec ahu-platform-redis redis-cli XLEN ahu.ai.audit

Gateway log across the restart:

2026/07/04 08:58:23 gateway listening on :8200 (4 upstreams)
2026/07/04 09:00:30 draining...
2026/07/04 09:00:31 gateway listening on :8200 (4 upstreams)

(Container Created 08:58:22, StartedAt [restart] 09:00:30 — confirms the restart cycle actually happened, ~1s drain-to-relisten.)

At the 0.2s probe point, healthz still returned 200 — the drain/restart cycle was fast enough (~1s total) that my narrow probe window didn't land inside a hard outage; no connection error was captured on this particular attempt. This is not a hard assertion in the brief ("brief connection errors... engines' retry contract handles this" is descriptive, not a pass/fail gate) — if anything it demonstrates a faster recovery than expected.

Post-restart smoke.sh (with canonical model, matching Step 1's invocation): full recovery confirmed — healthz ok, queue status normal, non-streaming and streaming chat both completed successfully against the real upstream.

Audit stream: XLEN 45 → 53 (strictly grew across the restart + re-verification calls). ✅
Spool volume: docker compose exec gateway ls -la /var/lib/ahu-gateway/spool → directory empty (only ./..) both before and after the restart — Redis never became unreachable, so nothing was ever spooled to disk. ✅

Result: PASS.

Side finding during this step (not a bug — documented, not chased)

The brief's own example command for Step 4 omits the MODEL override (GW=http://localhost:8200 ../scripts/smoke.sh), which falls back to smoke.sh's default MODEL=qwen-35b — the alias, not the canonical model string. Running that literally against the real x056 vLLM produces:

< HTTP/1.1 404 Not Found
{"error":{"message":"The model `qwen-35b` does not exist.","type":"NotFoundError","param":"model","code":404}}

Audit event correctly captured this too: "model":"qwen-35b","status":"error","error_code":"UPSTREAM_404".

Root cause (verified against source, not assumed): internal/registry/registry.go Resolve() maps any configured alias in an upstream's models: [...] list to the upstream for routing purposes only (byModel map). internal/proxy/handler.go:210 then forwards the original, unmodified request body to the upstream (h.forward(ctx, up, path, r, body)) — the gateway never rewrites the model field. This matches the design doc exactly (§4.3a: "model maps through the registry to a healthy replica" — routing only, no body rewriting; true OpenAI passthrough). The real vLLM instance only registers Qwen/Qwen3.6-35B-A3B-FP8 (confirmed via GET /v1/models), not the short alias qwen-35b that's listed in the example config for convenience. So: not a gateway defect — it's expected passthrough behavior, and the example config's alias entry doesn't happen to also be a name the real backend recognizes. All drills after this were re-run using the canonical model string (consistent with Step 1) and I did not modify any code.

Commit + tag

git add scripts/loadcheck.sh .gitignore
git commit -m "test(p0): smoke, real-client compat, saturation and restart drills..."
git tag p0-gateway

10608fe test(p0): smoke, real-client compat, saturation and restart drills   (tag: p0-gateway)
db63788 feat(deploy): dockerfile, compose stack, x056 example config, smoke script

Files changed: scripts/loadcheck.sh (new), .gitignore (+/deploy/gateway.yaml).

Stack left running (as instructed). Stop command: docker compose -f deploy/compose.yaml down

Summary table

Concerns for follow-up (non-blocking)

1. ext-dashscope-397b untested (no API key) — expected, out of scope per task instructions.
2. cleanup-3b / tei-embeddings health state on x056 not verified — out of scope per task instructions.
3. Model-alias-vs-canonical-name passthrough behavior (documented above) is working as designed, but worth a one-line callout in engine integration docs so engine authors know to send the exact upstream-recognized model string, not just any alias listed in gateway config.