GPU facts and the app-side optimisation plan

Date: 2026-08-24 · Companion to the benchmark report

The one-sentence finding: we never hit a GPU limit today. Every bottleneck we found was an application or configuration limit, and most of them were sized for the old single-GPU H100 box.

Part A — What we now know about the GPUs

Hardware, verified

They are not "B200 NVL". NVL is specifically the two-GPU bridged variant; these are SXM parts on an 8-GPU NVSwitch baseboard, which is a strictly stronger interconnect.

NVLink is per-node. Node 1 ↔ node 2 is ordinary networking. This is why the 397B must fit inside one node at TP=8 — a cross-node tensor-parallel split would run over Ethernet.

Measured performance ceilings

Fleet scaling is 89% efficient at matched per-replica concurrency (7,928 vs 8,946 tok/s per GPU). The cluster scales well; nothing about the hardware is the problem.

Three things that are NOT true

1. "The B200 is much faster per card." It is ~21–31% faster than an H100 NVL on this model. The 6.1× fleet gain is GPU count and memory, not per-card speed.
2. "FP4 will double throughput." The nvidia/…-NVFP4 checkpoint is MIXED_PRECISION with W4A16 experts — 4-bit weights, 16-bit activations. It never touches the FP4 tensor cores and measured 4–11% slower.
3. "More power = more efficiency." Perf-per-watt got worse: 9.7 tok/s/W on H100 vs 6.3 on B200 for this small-MoE workload.

Part B — Every bottleneck we found was app-side

Six were found today. None were the GPU.

Items 5 and 6 are the important ones, and they share a cause.

The gateway's admission slots were never resized after the migration

internal/pool/pool.go:123 — if p.active >= p.slots.Total { return } — is a hard concurrency cap. Requests beyond it queue.

Every slot budget still reflects the world where one H100 GPU served everything:

qwen-35b is the costly one. At its 96-slot cap the production fleet delivers 19,184 tok/s. The same fleet at concurrency 512 delivers 58,764 tok/s with p50 still at 2.2 s — so the cap is leaving 3× on the table with no latency justification.

Part C — The optimisation plan

Ranked by measured value per unit of risk.

1. Resize the gateway slot budgets — config only, biggest win

qwen-35b: 96 → 512. Justification is measured, not guessed:

512 is the point where throughput is 3× and p50 is still comfortably under the interactive hold budget of 45 s. Going further trades latency for throughput that no current workload needs.

paddleocr: 4 → 16. The service peaks at 47.7 req/s at concurrency 8 with 16 workers; a slot budget of 4 makes that unreachable. 16 gives headroom without pushing past the measured knee.

Do NOT bulk-raise the rest. Measure first (method below). A slot budget above real capacity converts a clean queue into an overload collapse — exactly what happened to gpu-server at concurrency 24.

2. Measure each remaining upstream before resizing

Method, using the harness at /tmp/aload.py:

1. Run from ahu-int-01 (192.168.82.122) — it serves nothing.
2. Sweep concurrency until throughput plateaus.
3. Confirm client_cores_used stays near zero, or you are measuring the client.
4. Set slots at the knee, NOT the plateau — the knee is where
   latency is still acceptable.

Priority order: vllm-397b-onprem (synthesis tier, user-facing), cleanup-3b, tei-embeddings, doc-classifier-svc, ai-ahu-rag.

3. Fix the app-side call pattern, not just the caps

Raising slots only helps if callers actually issue concurrent requests. Two things to verify per engine:

• HTTP connection pooling and keep-alive. A client that opens a new TCP connection per request will not reach high concurrency regardless of slot budget. The async harness used keep-alive and drove 1,536 concurrent on 0.05 of a core; the thread client could not.
• Sequential loops over documents/pages. If OCR extraction iterates pages one at a time awaiting each LLM call, the fleet sees concurrency 1 no matter how many replicas exist. Batching or asyncio.gather over independent items is where the app-side gain is.

4. Add saturation metrics so this is visible next time

None of these caps were observable — they were found by benchmarking. Worth exporting from the gateway per upstream:

• slots_active / slots_total (are we at the admission cap?)
• queue depth and time-in-queue
• rejections / Retry-After responses

If slots_active sits at slots_total, the cap is the bottleneck and the dashboard should say so.

5. Re-check placement only after the above

Consolidation freed GPUs and worked, but the current layout (6× 35B, one card shared by three aux services, one for paddle) is sized against capped demand. Once slots are correct and apps issue real concurrency, re-measure before moving cards again.

What NOT to spend time on

• FP4 — measured slower on the available checkpoint. Revisit only for a true W4A4 build.
• Moving the gateway for throughput — the earlier "gateway costs 25%" finding was a benchmark-client artifact. At scale it matches direct (130,613 vs 126,847 tok/s). Move it to ahu-ctrl-01 for zone architecture, not speed.
• Buying more GPUs — at present the cluster is 3× under-driven by its own admission config.

Summary

The cluster can serve 127,055 tok/s. Production is configured to admit 19,184. The gap is a config file, not hardware — and the same pattern (a limit set for the old world) explains five of the six bottlenecks found today.

APPLIED 2026-08-24. qwen-35b slots 96 → 512 (batch_max 8 → 48), paddleocr 4 → 16.
Measured through the gateway on the production 6-replica fleet: ~15,700 → 53,619 tok/s (3.4×),
p50 2,466 ms — far inside the 45 s interactive hold budget. Backup: gateway.yaml.bak-pre-slotresize.

Also corrected while verifying: the gateway's real overhead is −8.6% at conc 512
(58,644 direct vs 53,619 through it), not the 25% claimed earlier nor the 0% claimed after that.
Both prior figures were measurement artifacts — see Part 7 of the benchmark report.

Next actions, in order: (1) measure and resize the remaining upstreams — vllm-397b-onprem 32,
cleanup-3b 8, tei-embeddings 8, doc-classifier-svc 4, ai-ahu-rag 8; (2) verify engines use
HTTP keep-alive and do not loop pages sequentially; (3) export slots_active/slots_total so the
next cap is visible without a benchmark.