Demo runbook — 28 August 2026

Rehearsed 2026-08-24. Every command below was run and verified on that date.

30 minutes before: warm everything

Cold services are the main way this looks slow. vLLM captures CUDA graphs on first
use and paddle's 16 workers each load their model lazily, so the first request
to a cold service can be seconds instead of milliseconds.

# 1) LLMs + embeddings + RAG (run on the H100 box, 192.168.83.20)
H='-H Content-Type:application/json -H X-Tenant-Id:warm -H X-Surface:warm -H X-User-Id:ops -H X-Priority:interactive'
for m in qwen3.5-397b-a17b Qwen/Qwen3.6-35B-A3B-FP8 cleanup-3b; do
  for i in 1 2 3; do
    curl -s -o /dev/null --max-time 120 -X POST http://127.0.0.1:8200/v1/chat/completions $H \
      -H "X-Request-Id: w-$m-$i-$(date +%s)" -H "Idempotency-Key: w-$m-$i-$(date +%s)" \
      -d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"Jelaskan pendirian PT singkat\"}],\"max_tokens\":64}" &
  done
done; wait

# 2) paddle-ocr's 16 workers (run on node 2, 192.168.83.91)
for i in $(seq 1 24); do
  curl -s -o /dev/null --max-time 40 -X POST http://192.168.83.91:8108/layout -F "file=@/tmp/ocr-test.png" &
done; wait

5 minutes before: one-line health check

for u in https://x056.ahu-demo.ocr.val.id/api/health \
         https://x056.ahu-azure.val.id/api/gpu/health \
         https://x056.ahu-azure.val.id/api/classifier/ready; do
  printf "%-55s %s\n" "$u" "$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 "$u")"
done

All three must return 200.

Measured performance (what to expect on the day)

Flow	Measured
RAG retrieval (`/search`)	13–23 ms
397B synthesis, 300 tokens	~2.0 s (TTFT 27 ms)
35B, 400 tokens	1.46 s (TTFT 48 ms)
OCR layout (paddle)	66–71 ms
Full OCR job pipeline	~2 s end to end

Under sustained stress (24 concurrent, 45 s): 1,881 jobs completed, 0 rejected, 0 errors.

If something goes wrong

Symptom	Cause	Action
A service is slow on the first request only	Cold start	Normal. Warm it (above).
OCR uploads hang	gpu-server overloaded	Wait ~30 s — `ahu-autoheal` restarts it automatically. It is labelled and watched.
One LLM answer is slow, others fine	A single vLLM replica degraded	Nothing to do — the gateway health-skips bad endpoints and round-robins the other five.
Everything LLM is slow	Check `docker ps` on node 2 — how many `ahu-vllm-35b*` are up?	Should be 6. `cd ~/node2-models && docker compose -p node2-models up -d`
Chatbot answers with no sources	RAG or Milvus down on `ahu-int-01`	`ssh vigilia@192.168.82.122 'cd ~/milvus && docker compose -p milvus up -d'`

Rollback points (if a change needs undoing)

What	Backup
Gateway slots	`gateway.yaml.bak-pre-demoslots`, `.bak-pre-slotresize`
nginx `/api/gpu` limit	`ahu-azure.conf.bak-pre-limitconn`
paddle healthcheck	`paddle-ocr-service/docker-compose.yml.bak-pre-demo`

Remove autoheal entirely with: ssh ubuntu@192.168.83.91 'cd ~/autoheal && docker compose -p autoheal down'

What changed for this demo (2026-08-24)

Gateway admission slots resized off the old single-H100 sizing: qwen-35b 96→512,
paddleocr 4→16, vllm-397b-onprem 32→128, cleanup-3b 8→64,
tei-embeddings 8→64, doc-classifier-svc 4→16, ai-ahu-rag 8→32.
gpu-server deliberately left at 8 — it is the one service that collapses rather
than queues, so its low budget is protection, not a bottleneck.
ahu-gpu-server 1 → 8 uvicorn workers; celery worker-llm 4→16, worker-akta 2→8.
paddle-ocr 4 → 16 workers, and given a healthcheck (it previously had none).
ahu-autoheal deployed on node 2 — restarts containers that go unhealthy.
restart: unless-stopped only fires on process exit and does nothing for a wedged process.
nginx limit_conn ahugpu 12 on /api/gpu/ — that path bypasses gateway admission.