H100 NVL vs B200 — controlled A/B benchmark

⚠ Parts 1–5 client-limited; Part 6's gateway figures were cache-contaminated — see Part 7

Every throughput number in Parts 1–5 was produced by a thread-based load
generator. Part 6 replaces it with an async client and shows the old harness was
capping results at roughly half the real figure, and that its "gateway costs
25%" finding was an artifact rather than a property of the gateway.
Part 6 supersedes all throughput numbers below, and Part 7 corrects Part 6's
gateway figures — those were inflated by Idempotency-Key replay. The relative
findings in Parts 1–3 (B200 vs H100 per card, FP4 slower, paddle worker cap) still
hold, because each compared two sides measured with the same client.

Date: 2026-08-24 · Model: Qwen/Qwen3.6-35B-A3B-FP8 (identical weights, identical vLLM args, TP=1 on both)

Method

Hardware: H100 NVL 95,830 MiB / 400 W limit · B200 183,359 MiB / 1000 W limit.

Burst results

Zero failed requests on either side at every level.

Sustained load — 45 s @ concurrency 16 (the utilisation test)

Both cards sat at ~96–97% utilisation, which confirms the comparison was GPU-bound and not limited by the client or the network.

Analysis

1. The B200 is ~21% faster on this model — real, but modest for a generational jump.
Throughput +21%, latency −17%, TTFT −37% at low concurrency.

2. The reason it isn't larger: this model barely exercises a B200.
Qwen3.6-35B-A3B is a mixture-of-experts with roughly 3B active parameters per token. At 256-token generations the workload is dominated by memory bandwidth and per-kernel overhead, not dense math. The B200's headline advantages — FP4/FP8 tensor throughput and vastly more HBM — are largely idle here. What we measured is essentially the HBM bandwidth delta.

3. Performance per watt is WORSE on the B200 for this workload.

The B200 drew 86% more power for 21% more throughput. On small-MoE inference the H100 is the more efficient card. This is not an argument against the B200 — it is an argument against judging the B200 on this workload.

4. The B200 batches more uniformly.
At concurrency 32 its TTFT p50 is worse (136 ms vs 104 ms) but its p95 is better (144 ms vs 192 ms) — a 8 ms spread versus 88 ms. The H100 favours some requests and produces a long tail; the B200 admits the batch evenly. For an interactive surface, predictable p95 beats a faster median.

5. The real justification is capacity, not per-token speed.
The 397B model needs ~450 GB+ at FP8. The H100 box had 192 GB total and could not run it at any setting — which is why synthesis ran on Alibaba DashScope, a cross-border dependency with PDP exposure. A B200 node holds 1.4 TB across 8 cards and runs it on-prem at TP=8. That is a categorical capability change, not a percentage.

Caveats

• Single GPU per side (TP=1). Fleet-level comparison is now in Part 4, including NVLink topology.
• One prompt shape, 256-token outputs, ~46 s sustained. Not a full sweep across input/output length ratios.
• FP4 was not tested in Part 1. Answered in Part 2 — it is slower on this checkpoint.
• The H100's ahu-vllm was restarted solely for this benchmark and stopped again afterwards.

Part 2 — Scaling the B200 cluster (same day)

The Part 1 conclusion was that a single B200 gives ~21% over an H100 on this model, and that the model barely exercises the hardware. Part 2 acts on that: instead of chasing per-GPU speed, use more of the 8 GPUs.

FP4 — tested and rejected

nvidia/Qwen3.6-35B-A3B-NVFP4 (22 GB vs FP8's 35 GB) deployed on a spare B200, benchmarked with the identical harness and args.

It is slower, and the checkpoint explains why. Its own hf_quant_config.json declares quant_algo: MIXED_PRECISION with the MoE experts at W4A16_NVFP4 — 4-bit weights, 16-bit activations. The math still runs at 16-bit and pays a dequantization step on top, so the FP4 tensor cores are never used. This is a memory optimisation wearing an FP4 label.

Quality showed no regression — deterministic must_include / must_not_include fact-check (the same gate goldeneval applies before its judge): FP8 7/20, FP4 10/20, zero must_not_include violations either side. Treat the 3-assertion gap as noise, not a win. Two caveats matter more than the number: the shipped golden set is verified: false on every item, and this tested the raw model without RAG.

Its one real benefit is ~13 GB more KV-cache headroom per card. Worth revisiting only under KV pressure, or if a true W4A4 checkpoint appears.

Replica scaling — the actual win

The gateway already round-robins across endpoints with per-endpoint health skipping (internal/registry/registry.go:118, eps[(st.next+i)%n]), so this needed no gateway code — only more endpoints in the qwen-35b upstream.

All measured through the gateway, so routing and admission overhead is included — this is what a caller actually receives, not a synthetic ceiling. Distribution was verified even at every step: 6/6/6/6 on a 24-request probe, then 393/393/393/393/392/393 across six under sustained load.

Scaling is sub-linear (4.46× from 6×) because of gateway overhead and per-replica queueing, but latency did not degrade — p50 at 6 replicas / conc 96 is marginally better than 4 replicas / conc 64.

paddle-ocr — was capped at 4-way parallelism

The OCR workhorse. PADDLE_OCR_WORKERS defaulted to 4, and throughput flatlined there.

At 4 workers everything past concurrency 4 simply queued: latency grew linearly while throughput did not move. At 16 workers the peak is 47.7 req/s (+26%) with lower latency at that point (131 ms vs 173 ms).

+26%, not 4× — paddle is genuinely GPU-bound, not merely worker-bound, so additional workers add contention. Concurrency 8 is the knee; 16 is worse. It also degrades gracefully — zero errors at every level, it just queues.

Benchmarking trap: with 16 workers each loads its model lazily, so a run started immediately after restart measures warmup (a misleading 401 ms p50). Fire ~144 warmup requests first.

OCR job path — one uvicorn process on 256 cores

ahu-gpu-server shipped with no --workers flag: a single process fronting the whole OCR job API. worker-llm was --concurrency=4, worker-akta --concurrency=2.

Now 8 uvicorn workers / 16 / 8. Multi-worker is safe here because job state lives in Celery+Redis via AsyncResult (app/main.py:411), not an in-process dict — verified before changing.

Throughput falls past the knee — overload collapse. Two honest caveats: this test bypassed the gateway by hitting :8000 directly, and the gateway's slot admission is what prevents this in production by queueing excess load. Also, an earlier run showing a flattering 29 ms p50 was measuring the result cache — identical payloads hit cache, so payloads must vary to measure anything real.

Final GPU allocation (os-b200-02)

GPU 0,1,3,5,6,7  →  35B replicas ×6      (ports 8001/8011/8014/8012/8015/8013)
GPU 2            →  cleanup-3b + tei-embeddings + signature-verify   (35.5 GB of 183)
GPU 4            →  paddle-ocr, alone    (16 workers, 11.9 GB)

Consolidation freed 2 cards: tei (8.6 GB) and signature-verify (1.7 GB) were each occupying a whole 183 GB B200. paddle deliberately keeps its own card — it is the only one of the four that is actually GPU-bound.

What this says overall

The Part 1 finding holds and gets sharper: on this workload the B200's value is not per-token speed (~21% over H100, and worse perf-per-watt at 6.3 vs 9.7 tok/s/W). It is capacity — 8 cards of 183 GB let you run six full replicas plus every auxiliary service on one node, which is where the 4.46× came from. The quantisation lever was a dead end; the parallelism lever was not.

Part 3 — Full ledger: every model and service moved or improved

Parts 1–2 cover what was benchmarked. This part accounts for everything that changed on 2026-08-24, including services that moved without a throughput story.

Models now serving from the B200 cluster

H100 GPU 0 is now at 0 MiB. GPU 1 holds only the classifier (~1.5 GB).

Two models needed a Blackwell rebuild — and the failure was silent

Both images predate Blackwell (compute capability 10.0). Both started, served /health 200, and looked healthy while being incapable of inference.

Never validate a GPU migration on health checks. Paddle logged the arch failure as a warning and kept answering probes. Only a real inference request exposed it.

Accuracy was preserved, and measured — not assumed

paddle-ocr — same document, both hosts:

H100:  "REPUBLIK INDONESIA\nNIK : 3174012345670001\nNama : BUDI SANTOSO..."
B200:  byte-identical

Warm latency 56–69 ms (B200) vs 57–63 ms (H100).

signature-verify — the AUC-sensitive one, so both the matching and the discriminative path were tested:

The identical model string confirms the fine-tuned signet-ft.pth was baked in — without it the build silently falls back to a weaker GPDS checkpoint (AUC 0.766 vs 0.798). Only bounding boxes differ, by ~0.003 px of floating-point noise. A self-match alone would not have proven anything, which is why the mismatched pair was tested too.

Application services relocated

Correctness and security fixes made along the way

• RAG was pointing at a model that exists nowhere — LLM_MODEL=Qwen/Qwen3.5-27B-FP8 produced 62 model-not-found errors/24 h and a permanently 500-ing /answer. Nobody noticed because only /search is used. Now Qwen/Qwen3.6-35B-A3B-FP8; errors 62 → 0.
• Shared Redis had no authentication at all. Acceptable while docker-internal; not acceptable once the public stack reached it across the LAN. Now --requirepass, verified: PONG with password, NOAUTH Authentication required. without.
• A migration decision had silently reverted. The gateway RAG-proxy (built July, enforcing that the DMZ surface must not call Internal-zone RAG directly) had served 10 calls ever, all on 2026-07-08/09 — every consumer was still using Docker DNS. Now genuinely enforced and carrying live traffic.
• ahu-gpu-server ran a single uvicorn process on a 256-core host. Now 8 workers; Celery raised 4→16 and 2→8.

Honest summary of the day

Two things were tested and produced negative results worth keeping: FP4 is slower on this checkpoint, and perf-per-watt is worse on B200 for small-MoE inference (6.3 vs 9.7 tok/s/W). Neither is an argument against the hardware — both are arguments against the assumptions we started with.

The wins were: 13,014 tok/s on the 35B (4.46×), 47.7 req/s on paddle (+26%), the 397B running on-prem instead of cross-border, and every model verified to produce identical output after moving — not merely to return 200.

Part 4 — Fleet vs fleet: 2× H100 NVL against 16× B200

Parts 1–3 compare one card against one card. That understates the change, because the estimate was never "we bought a faster GPU" — it was "we replaced a 2-GPU box with two 8-GPU nodes." This part measures the whole infrastructure.

Are the B200s "NVL"? No — they are better

Worth settling, because the naming misleads.

"NVL" is specifically the two-GPU bridged variant. These B200s are SXM parts on an 8-GPU baseboard with a full NVSwitch fabric — every GPU talks to every other GPU at ~956 GB/s, versus the H100 NVL's single bridged pair at ~319 GB/s. So they are not NVL; they are a strictly stronger interconnect.

One limit worth stating plainly: NVLink is within a node. Node 1 ↔ node 2 is ordinary networking, not NVLink. That is exactly why the 397B must fit inside a single node at TP=8 — a cross-node tensor-parallel split would run over the network and be far slower.

Fleet capacity

Fleet throughput — measured, not extrapolated

Both fleets running the identical model (Qwen3.6-35B-A3B-FP8), 45 s sustained, ignore_eos, from the same neutral load generator.

The B200 figure uses only 6 of its 16 GPUs. The other ten are busy: eight run the 397B on node 1, and two carry paddle-ocr, cleanup-3b, tei-embeddings and signature-verify. The old fleet's number uses both of its GPUs — everything it had.

Superseded by Part 5. The 397B was later stopped to turn 14 of 16 GPUs onto this model: 19,641 tok/s through the gateway (4.24×) and 26,069 direct (5.62×).

The honest per-GPU picture

Per GPU, a B200 behind the gateway is slightly below an H100 hit directly. The fleet win is almost entirely GPU count, not per-GPU speed — the ~21% per-card advantage from Part 1 is partly eaten by gateway routing and shared host resources once six replicas contend on one machine.

Power tells the same story. At measured draw the H100 fleet does ~9.3 tok/s/W; six B200 replicas do ~4.7. The new fleet is roughly half as power-efficient per token on this workload, and buys throughput with hardware and watts rather than efficiency.

What the old fleet simply could not do

This is the part no percentage captures. The 397B could not run on the old fleet at any setting, which is why synthesis lived on DashScope with the PDP exposure that carried. Measuring it at 2,052 tok/s on-prem is not a speedup over 4,637 — it is a capability that previously had no on-prem number at all.

Fleet verdict

Three findings, and they do not all point the same way:

1. 2.81× the throughput on the same model, using 6 of 16 GPUs against the old fleet's 2 of 2.
2. 15.3× the memory, which is what actually unlocked the 397B and let every auxiliary model move on-prem.
3. ~Half the power efficiency per token, and per-GPU throughput behind the gateway that is not better than a directly-hit H100.

The fleet was worth it for capacity and concurrency — not for per-token speed or efficiency. Anyone citing a single headline multiplier from this document should cite 2.81× (measured, same model, fleet to fleet) or 15.3× (memory), and should not imply the B200 is ~8× faster because there are 8× as many GPUs.

Part 5 — Whole-fleet measurement: 14 B200s on one model

Part 4 compared fleets but the B200 side used only 6 of 16 GPUs, because the other ten were serving the 397B and the auxiliary models. This part stops the 397B and turns almost the whole cluster onto the same 35B, so the fleet ceiling is measured rather than inferred.

Why 14 and not 16. Node 2's GPU 2 (cleanup-3b + tei-embeddings + signature-verify) and GPU 4 (paddle-ocr) stayed in production. Displacing the OCR workhorse to add two more replicas would have made the number prettier and the measurement less honest.

The 397B was stopped for ~10 minutes (12:25:53 → 12:35:46 UTC) and fully restored afterwards.

Results

The gateway is now the bottleneck

The same 14 replicas, the same client, the same load — the only difference is whether requests route through the gateway:

At 6 replicas this overhead was invisible. At 14 it removes a quarter of the fleet's capacity.

The likely cause is placement, not code: the gateway runs on the H100 box, which has 12 CPU cores, and at this scale it is streaming 224 concurrent SSE responses through a single Go process on that machine. The B200 nodes have 256 threads each. The production migration plan already calls for moving the gateway to ahu-ctrl-01 — this measurement turns that from tidiness into a throughput argument.

Per-GPU efficiency falls as the fleet grows

Two separate effects stack here. Gateway routing accounts for the 64% → 48% step. The 100% → 64% step is real scaling loss — more replicas contending for host CPU, memory bandwidth and PCIe on the same machines.

Honest caveat on this run: the load generator was node 1, which was simultaneously hosting 8 of the 14 vLLM replicas. Its 256 threads were shared between generating 224 concurrent requests and serving half the fleet. No genuinely neutral host was available once the whole cluster was in use, so the 14-replica numbers likely understate the fleet slightly. The 25% gateway delta is unaffected — both sides of that comparison ran under identical client conditions.

What this changes

• The defensible whole-fleet figure is 4.24× (19,641 vs 4,637, both through their production path) or 5.62× raw capacity (26,069 direct).
• Neither approaches the 8× the GPU count would suggest. Scaling is sub-linear and the gap widens with fleet size.
• The next throughput win is not more GPUs. It is moving the gateway off a 12-core box — worth ~25% of fleet capacity, more than any model or quantisation change tested today.

Part 6 — Corrected fleet benchmark (supersedes Parts 1–5 throughput figures)

Parts 1–5 all used a thread-based load generator. At high concurrency that client was itself the bottleneck: hundreds of OS threads contending on the GIL, run from a host that was also serving replicas. Every fleet number it produced was too low, and one conclusion drawn from it was simply wrong.

What changed in the method

The client now reports its own CPU. Across every run below it used 0.0–0.05 of 16 cores. The measurements are GPU-bound, and that is demonstrated rather than asserted.

Proof the old client was the constraint: identical fleet, identical target, concurrency 256 → thread client 26,069 tok/s, async client 47,605 tok/s. The harness was hiding 45% of the fleet.

Corrected results — 2× H100 NVL vs 16× B200

Both fleets, same model, same async client, swept to saturation.

Both fleets were driven past their knee until throughput stopped rising. GPU utilisation on node 2 during the fleet run averaged 92% across its 8 cards with samples at 99% — genuinely GPU-bound.

The two numbers that matter

Peak to peak: 6.1× — 127,055 vs 20,813 tok/s.

At comparable latency: 10.0× — and this is the fairer comparison, because the H100 fleet only reaches its peak by letting latency collapse:

The B200 fleet delivers ten times the throughput at lower latency than the H100 fleet achieves at half that load. The old fleet buys its peak with a 5.7-second median; the new one never has to.

Correction: the gateway does NOT cost 25%

Part 5 concluded that routing through the gateway cost 25% of fleet throughput, and recommended moving it off the 12-core H100 box on that basis. That finding was an artifact of the thread client. Re-measured:

At full scale the gateway matches direct — within noise, in fact slightly ahead. It shows a real deficit only in the mid range (conc 256), where per-request routing overhead is not yet amortised. The gateway is not the fleet bottleneck, and the throughput argument for relocating it is withdrawn. Moving it to ahu-ctrl-01 remains right for zone architecture; it is not worth 25%.

Corrected per-GPU scaling

At matched per-replica concurrency the fleet scales at 89% efficiency — far better than the 48% that Parts 4–5 reported. That figure was measuring the client, not the cluster.

Method notes for anyone repeating this

• Check your load generator's CPU. A thread-per-request client silently caps fleet benchmarks. Ours reported client_cores_used on every run for exactly this reason.
• Never generate load from a host under test. Parts 4–5 did, and it cost roughly half the measurable throughput.
• Sweep to the plateau. A single concurrency point cannot distinguish "saturated" from "under-driven" — the H100 fleet looked like 12,688 tok/s at conc 128 and was actually capable of 20,813.
• Warm every replica. vLLM loads lazily per worker; a run started immediately after boot measures warmup.

Service restoration

The 397B was stopped 12:40:53–12:53 UTC, and node 2's paddle-ocr, cleanup-3b, tei-embeddings and signature-verify from 12:40:18. All were restored and verified: 6× 35B replicas, all four aux services, 397B (answering through the gateway), gpu-server, classifier and the public OCR demo all returning 200, with node 2's GPU map back to its production layout.

Part 7 — Correcting Part 6's gateway numbers, and the slot resize

Part 6 concluded that the gateway "matches direct at scale" (130,613 vs 126,847 tok/s). That was wrong too, for a different reason than Part 5.

The bug: Idempotency-Key replay

The harness generated keys as fb-{worker}-{n}. Those repeat across runs, so the gateway correctly replayed the earlier response instead of calling a model — exactly what Idempotency-Key is specified to do in CONVENTIONS §2.

The tell was a physically impossible latency. Back-to-back on the same 6 replicas:

A 256-token generation cannot complete in 27 ms. The audit reported cache_hit=f throughout, so the replay was happening at the idempotency layer rather than the response cache — which is why a cache-hit check did not catch it.

Fix: a per-process run ID in the key (fb-{run}-{worker}-{n}).

Honest gateway overhead

So the gateway costs about 9% at high concurrency. Both earlier claims were wrong in opposite directions: Part 5's "−25%" was a thread-client artifact, Part 6's "0%" was idempotency replay. −8.6% is the measured figure.

That overhead is a reasonable price for admission control, audit, routing and refusal caching. It does not justify relocating the gateway on throughput grounds.

The slot resize — applied and verified

gateway.yaml, backup at gateway.yaml.bak-pre-slotresize:

batch_max was raised in the same proportion (~8% of total) so the interactive-vs-batch starvation guarantee is unchanged rather than loosened.

Result, measured through the gateway on the production 6-replica fleet:

The cap was costing 3.4× of available LLM throughput, and p50 remains far inside the 45 s interactive hold budget.

Verified after the change: paddle via gateway 200, 397B via gateway answering, public OCR demo 200.

Method lesson

Three separate measurement bugs were found in one day, each of which made a benchmark look fine:

1. Thread-based client — capped the fleet at ~55% and produced a false "gateway costs 25%".
2. Load generated from a host under test — cost roughly half the measurable throughput.
3. Repeated Idempotency-Keys — replayed responses and produced a false "gateway is free".

All three were caught by sanity-checking a number against physics rather than by the harness reporting an error. If a latency or throughput figure is better than the hardware can do, the harness is wrong. Check p50 against the minimum possible generation time before believing any result.