Qwen3.6-35B-A3B vs Qwen3.5-27B — Extraction A/B (Stage 1, cloud)

Date: 2026-06-29 · Question: should we swap the dev-box extraction model
(Qwen3.5-27B-FP8 @ x056:8001) for Qwen3.6-35B-A3B?

Setup

Endpoint: Alibaba MaaS (ap-southeast-1), OpenAI-compatible, all models at the
provider's native precision (≈BF16) — Stage-1 screening (cloud needs no FP8; see
"why two stages"). temperature=0, enable_thinking=false, response_format=json_object.
Inputs: 6 real OCR'd dev documents (3× Surat Pernyataan PP ~1.5k chars, 3× Akta
~13k chars), replayed through the real gpu-server/prompts/*.txt.
Models: qwen3.5-27b (baseline = current prod), qwen3.6-35b-a3b (candidate),
qwen3.6-27b, qwen3.5-122b-a10b (refs). Judge: qwen3.7-max.
Harness: backend/src/scripts/llm-extract-ab.ts (+ llm-extract-adjudicate.ts).

Results — robustness, latency, agreement

Model	JSON-ok	p50 latency	agree-vs-27B
qwen3.5-27b (baseline)	6/6	22.6 s	—
qwen3.6-35b-a3b	6/6	15.6 s	87%
qwen3.6-27b	6/6	31.1 s	86%
qwen3.5-122b-a10b	6/6	14.1 s	78%

Compliance (the swap's main risk): clean. The documented 3.5-35B-A3B
structured-output bug does not appear on the 3.6 model with thinking off.
Latency: ~31% faster than the 27B (cloud round-trip incl. network; relative
ordering matches the 3B-active prediction, not an on-prem absolute).
Agreement: 35B-A3B tracks our trusted 27B more closely than any other candidate.

Quality — oracle adjudication of the 13% disagreements

9 scalar disagreements → qwen3.7-max (blind, position-swapped). Raw tally:
candidate-correct 3 / candidate-wrong 4 / equivalent 0 / both-wrong 2. But the
tally is dominated by confounds, not quality:
- 2 both-wrong = confidence.email metadata scores (not extraction values) → noise.
- 6 of 7 akta diffs = referensi_sebelumnya.* on a prompt–document mismatch:
the test "AKTA" is an Akta Pernyataan Keputusan Rapat (amendment), but was routed
to the akta_pendirian prompt. Both models chose internally-consistent but
different prior deeds (27B→2013 establishment, 35B-A3B→2024 latest amendment); the
judge itself flip-flopped because "previous reference" is genuinely ambiguous.
- Where the prompt fits (all SP-PP docs): zero real disagreements — full equivalence.
- Coverage balanced: candidate-only 142 fields vs baseline-only 128 (neither drops fields).

=> No clean quality difference; no regression detected.

Verdict

Swap is justified for the dev box (reversible, low-risk): compliance clean, ~31%
faster, no quality regression. Confirm with Stage 2 = on-prem FP8 (serve
Qwen3.6-35B-A3B-FP8 on GPU1, A/B vs the live 27B-FP8) before promoting — that's the
real ship number. FP8≈BF16 for this task, so Stage-1 is a strong predictor.

Bugs the benchmark surfaced (independent of the model choice)

Prompt routing: documentType="AKTA" blanket-maps to akta_pendirian; amendment/
PKR deeds need their own prompt (akta_perubahan). Mis-routing degrades both models.
No labeled golden set. Dev DB has only ~27 clean SP-PP/AKTA confirmed fields
(KTP/carrier labels are dev junk with no OCR text). A real ~100-field verifikator-
checked set across akta/SP/apostille would make this authoritative.

Caveats

Synthetic dev docs, small N (6). Cloud precision ≠ on-prem FP8 (Stage 2 closes this).
String-anchor accuracy was discarded (identical 1/10 across all models = scorer artifact).

Stage 2 — on-prem FP8 confirm (x056 GPU1)

Result: arch-supported + loads, but serving is BLOCKED on a vLLM upgrade.

Served Qwen/Qwen3.6-35B-A3B-FP8 on GPU1 (CUDA_VISIBLE_DEVICES=1, port 8002), untouched 8001/8003.
It downloads, loads, and initializes: the hybrid arch (Gated DeltaNet linear
attention + FP8 MoE) IS supported by the box's vLLM 0.19.0 — GDN kernel, FP8 MoE,
KV cache all allocate (~34 GB weights + 12 GB KV; fits GPU1 alongside the akta worker
with room, 32× concurrency @ 16k ctx).
BLOCKER: the box's ~2-month-old vLLM 0.19.0 hangs at API-server bringup
(after KV-cache allocation), reproducibly in BOTH cudagraph-capture and
--enforce-eager modes. Also no tuned MoE kernel config for H100_NVL
(E=256,N=512,fp8) → "performance sub-optimal" warning.
=> On-prem FP8 confirm needs a newer vLLM image on x056 first. Weights are now
cached, so a retry is fast. Latency was never going to be representative here
(eager + untuned kernel), so no on-prem latency is reported.
Candidate container torn down; GPU1 + the live 27B restored, healthy throughout.

Bottom line: the swap verdict from Stage-1 stands (35B-A3B ≈ 27B, clean
compliance, faster). Stage-2 adds an operational precondition: upgrade vLLM on
x056 before any on-prem cutover, then re-run llm-extract-onprem.ts for the real
FP8 numbers.

Addendum — small-model choice: 3.6-35B-A3B vs 3.6-27B (refusal/robustness)

Prompted by a separate AGENTIC tool-calling benchmark where 3.6-35b-a3b showed a 33%
refusal rate and lower answer quality than 3.6-27b. Tested whether that transfers to
our single-shot extraction. n=24 real docs (12 SP-PP + 12 AKTA), cloud, thinking off:

Model	JSON-ok	refuse	p50 latency	agree-vs-27B
qwen3.5-27b (baseline)	24/24	0/24	39.3 s	—
qwen3.6-27b	24/24	0/24	56.1 s	86%
qwen3.6-35b-a3b	24/24	0/24	26.6 s	86%

The agentic 33% refusal does NOT manifest in our extraction task (0/24). On every
axis the two candidates tie (refusal, compliance, 86% quality-agreement) except speed,
where 35b-a3b wins 2.1× (and ~9× less active compute). 3.6-27b is the slowest even
with thinking off (dense tax) and ships thinking-ON by default (latency footgun).

Decision: 3.6-35B-A3B for the small/first-pass + cleanup role. Stack = 397b-a17b
(large/escalation) + 35b-a3b (volume). The agentic-bench concern was task-specific.