Signature Matching R&D — C5 audit, honest re-eval, and the C6 overnight run

Date: 2026-07-14 (overnight) · GPU 1 (handed over by Efran) · commit f7f6a12e

Efran: "if you'd like to use GPU 1 to train or fine tune a model or general RnD for our signature spesimen matching, go ahead."

I audited C5 before training anything new. That audit is the headline: the deployed model was never actually selected on merit, and its reported improvement is not statistically established.

1. Four bugs in C5's model selection

Bonus: c5_eval.py compared _state["signet"] (labelled "v3") against a checkpoint — but the C5 deploy overwrote the baked signet-ft.pth with the c5 weights, so that row was c5. It was comparing c5 to itself and printing identical rows. I hit exactly that when reproducing the baseline.

2. Honest re-evaluation (82 truly-disjoint officers, production ensemble)

c5 is directionally better (+6.1pp), but with n=82 the standard error on a ~40% proportion is ≈5.4pp — so +6.1pp is ≈1.1 SE: not statistically established. The previously-recorded +10.5pp (35.8→46.3) was measured with the 13 leaked officers and is too optimistic.

The evaluation set, not the model, is currently the bottleneck — 82 officers cannot separate these models.

3. The real root cause: C5's learning rate was ~7.5× too high

First I reproduced C5's failure with a corrected val (fixed per-officer seeds, production ensemble, 227 held-out officers) at C5's own lr=1.5e-4:

warm start (v3):  70.0%
epoch 1: 65.2%   epoch 5: 60.4%   epoch 10: 58.1%   epoch 20-30: ~62%
loss:    0.225 → 0.084            (memorising, never recovers)

My first read was "the clean set is too small — fine-tuning can't work." That was wrong. Re-running at lr=2e-5 inverts the result:

warm start (v3):  72.2%
epoch 1: 71.8   epoch 3: 73.1   epoch 5: 74.0   epoch 6: 74.4 *best*   → plateaus ~74

Fine-tuning works fine; C5 just cooked it. The high LR blew the warm start apart in a single epoch, the curve never recovered, and because best.pth was never written (bug #1) the most-overfit epoch shipped anyway. Bugs #1 and the LR compounded: a broken selector hid a broken hyperparameter.

(Caveat on the two warm-start numbers, 70.0 vs 72.2: my val seeded synthetic docs with Python's hash(), which is salted per process, so the val set differed between runs. Within-run deltas are valid — and they're what the table reports — but I fixed it to md5 so runs are byte-comparable from here on.)

3b. …but the LR win does NOT (yet) show up on production-like data

Re-running the honest disjoint eval with the LR-fixed model (deterministic seeds now, so these are reproducible):

The whole spread (36.6–40.2) sits inside one standard error (≈5.4pp at n=82). So: the LR fix is real on the 227-officer synthetic val, but on the production-like corpus no model here is distinguishable from any other — including c5 vs v3. I am not claiming c6-small beats c5.

This is the same wall from §2 seen from the other side: with 82 officers we cannot measure a few points of detect@p95. Which is the strongest argument for the corpus expansion — it buys a reliable held-out set (~1,600+ officers, SE ≈1.2pp) as much as it buys training data.

4. And still: expand the corpus (running now)

Data remains the second lever — the clean set is tiny (1,391 officers) and the registry has 21,699 multi-registration officers; we were training on 4,704.

• New manifest: 23,574 officers / 47,148 sheets — with the frozen-eval officers properly excluded this time (fixing bug #4).
• Prep running detached on GPU 1 at ~75–100 sheets/min, 0 errors → ~7–8h ETA for the 40,480 new sheets. Resumable (meta.jsonl checkpoint). ~1.6GB disk (66GB free).
• An unattended chain (chain-c6.sh, nohup'd on the host so it survives my session) then runs: wait-for-prep → re-clean → train at lr 2e-5 AND 5e-5 (25 epochs, fixed-val ensemble selection) → honest disjoint eval of v3 / c5 / c6-small / both c6-big variants. Two LRs because the optimum can shift with ~6× more data.

5. Mid-run checkpoint (de-risking the chain, on partial data)

Rather than discover a bug at hour 5, I dry-ran the chain's clean + train steps on the partial corpus (11,609 sheets). Both work, and the numbers are encouraging:

Clean yield — already beats C5's full set, from a partial corpus:

Extrapolating to the 35k target: ~10,000 officers / ~20,000 crops ≈ 7× C5's training data.

Does more data help? Trained on the 2,041-officer set at lr 2e-5:

(Absolute values differ because the val officer pools differ — only deltas compare.) The mid run's loss falls slowly (0.248→0.197) and its best epoch is the last one — it hasn't converged, where the small run had already memorised and turned over. That's the healthy signature of more data. It also means the epoch budget mattered: I raised the chain from 25 to 40 epochs, which is free given best-checkpoint selection.

6. The decisive test: the REAL Darsani anchor — and it changes the recommendation

Everything above (val, calib2) scores synthetic documents made by to_document. The only real production data we have is the actual Kutipan document vs the four "Darsani" registry specimens. c5_anchor.py only ever scored the genuine one — it never tested ranking, which is the actual failure mode. So I wrote c6_anchor.py:

Two hard findings:

1. No model ranks the genuine specimen first. Every one puts the wrong-person sheet (291092585, Galih Yudha Praja) on top. This is the quantitative proof that "match against best-of-N specimens" would actively pick the wrong file — the exact reason the score must stay advisory and the verifikator's eyes are the mechanism.
2. The incumbent c5 is the best of the four on real data (only one to get the genuine to #2), and my LR-fixed models are worse here (#4) despite winning on the synthetic val.

Synthetic-document gains do not transfer to the real scan. That is the ceiling of this whole approach, and it's now measured rather than suspected. It also means the earlier "+2.2pp LR win" must be read narrowly: it's a win on synthetic docs, not evidence of a better production model.

Recommendation: do not ship c6. Keep c5 deployed. The chain still runs the big-data variants and now ends with this same real-anchor ranking (step 5) — if c6-big finally ranks the genuine #1, that would be the first genuine signal; if it lands #4 like the others, it confirms synthetic augmentation is exhausted and the only real lever left is real document↔specimen pairs from the attestation flow.

7. Why it fails: the bottleneck is the FRONT-END, not the model

I opened up what the model actually sees on the real anchor. This is the most important thing I found tonight.

Two independent front-end failures:

1. Isolation destroys the document's signature. isolate_signature does k-means(k=3) and keeps only the single darkest cluster. The real Kutipan crop contains printed black text, a blue-black signature, a purple stamp and a green security pattern — so the darkest cluster is the printed text, and the blue pen is dropped. The raw crop has a perfectly clear signature; the isolated one is 1% ink of disconnected fragments.
2. The detector accepts printed text as a signature. Galih's winning "signature" crop is the printed name "Galih Yudha Praja", rotated 180°.

So the 0.9142 dino cosine that puts Galih above the genuine isn't a signature judgement at all — it's sparse-thin-strokes matching sparse-thin-strokes. The model is comparing garbage to garbage, and the genuine (a dense, real signature at 4.5% ink) looks less like the mangled doc than the text does.

This reframes C5/C6 entirely. Synthetic-document augmentation trains the metric model on cleanly isolated synthetic crops — a domain production never reaches, because production's isolation is broken on exactly the documents that matter. That is why synthetic-val gains don't transfer, and why more data can't rescue it: we were tuning the wrong layer.

A prototype fix — and an honest negative result

I tried the obvious repair: keep every cluster meaningfully darker than paper (k=5) instead of only the darkest. It recovers the document's signature (ink 1.0% → 12.8%, the full "Naromu" visible) — but it does not fix the ranking:

Because v2 also retains the stamp, the printed text and the security pattern, and Galih's crop is printed text — so the confound now exists on both sides. Keeping more ink is not the answer.

A real fix needs two things, and neither is a hyperparameter:
- a detector that rejects printed text as a signature candidate (Galih's crop should never have been proposed), and
- isolation that separates pen ink from stamp + print on a real stamped document (colour-aware, not "darkest cluster").

That is a proper piece of CV work with its own validation (it must not regress the clean-specimen path, which currently works well). I deliberately did not hack it in unattended — it sits on the production scoring path for every comparison.

Where to look in the morning

ssh efran@192.168.83.20
tail -40 ~/sigtrain/chain-c6.log          # chain progress + final eval table
cat ~/sigtrain/runs/c6big/best.json       # best epoch by the corrected metric
wc -l ~/sigtrain/crops/meta.jsonl         # corpus size (47,148 = complete)

The chain ends with (a) the disjoint-officer table (+ ensemble-weight sweep), and (b) the REAL Darsani ranking anchor — the decisive one: if c6-big ranks the genuine 2655 #1 it is the first real signal; if it lands #4 like the rest, synthetic augmentation is exhausted.

Deliberate non-actions

• No auto-deploy. The chain only trains and evaluates. Shipping a model is a decision to make on the numbers in the morning — I won't push weights to the live service unattended.
• No calibration change. The knots would need re-fitting after a model is chosen.
• I did not add test-time flip/mirror augmentation — measured earlier today: it breaks discrimination (genuine 22.9% vs contaminated 47.8%) because mirroring never occurs in a real scan and only gives impostors extra chances to match.

8. Overnight chain results (completed 00:27) — and the fix that actually works

The chain finished. Prep reached 32,470 sheets → 5,954 clean officers / 12,235 crops (4.3× C5), giving a 988-officer val (SE ≈1.6pp — finally reliable).

Training (big corpus): lr 2e-5 → 50.1% → 53.0% (+2.9pp); lr 5e-5 → 52.0% (+1.9pp). Confirms 2e-5 and that more data helps a little.

Disjoint eval (n=82, production 50/50): v3 36.6 · c5 40.2 · c6-small 39.0 · c6-big-2e5 39.0 · c6-big-5e5 39.0 — still all inside noise. No ensemble gain from 4.3× data.

But two real signals:
- SigNet alone genuinely improved with data: xDOMAIN AUC 0.722 → 0.755, detect@p95 26.8% → 37.8%. The frozen dino (0.664) now caps the 50/50 blend — dino_w=0.3 consistently gives the best xDOMAIN. The production weight is stale.
- Real anchor: c6-big rose to genuine #2 (from #4 for c6-small), matching c5 — but still nobody reaches #1.

The actual fix: the BOX, not the model

Testing the front-end directly on the real document finally produced a #1:

And a padding sweep shows a clean monotonic dose-response, so it's causal, not luck:

pad -15..+5 → genuine #1 ✅ | +10 → #2 | +20 → #3 | +30..+60 → #4
Galih's score RISES 0.327 → 0.649 as padding grows; genuine stays flat (~0.41)

Because Galih's crop is printed text: the more stamp/printed-name you let into the document box, the better the WRONG official matches.

Root cause, precisely: the YOLOS detector returns the whole signature block — (482,1239,875,1489) = 393×249 — when the signature is only ~175×105. It localises "signature + stamp + nama tercetak + NIP", ~5× too much area. The loose "manual" box was just the detector's own output.

Shipped: the draw-box modal (already available to both roles) now tells the user to draw tight around the strokes and exclude stempel / nama tercetak / NIP. Zero risk to the scoring path, and it converts the finding into user value today.

Not shipped — two honest failures. Auto-tightening the box needs to separate a signature from the stamp it is written over:
- connectivity can't: Otsu+bridge merges signature+stamp into one 417×210 component, so dropping the "ring" drops the signature too;
- colour clustering can't (naively): the stamp ring is dark enough to survive in the two darkest clusters and still bridges to the strokes.

That needs real CV (hue-based stamp suppression, or a detector retrained to box strokes only) with its own validation — it sits on the production scoring path for every comparison, so it doesn't go in unattended.

What I'd do next (in priority order)

1. Tighten the box automatically — the highest-value item, now quantified (loose→tight flips the genuine from #3 to #1). Two routes: (a) retrain/fine-tune the detector to box strokes rather than the signature block; (b) hue-based stamp suppression so the pen separates from the purple ring, then re-box onto the handwriting. Must be validated against the clean-specimen path (which works well today) plus the real anchor. My two naive attempts failed for documented reasons — this needs real CV work, not a heuristic.
2. Make the detector reject printed text. It proposed an upside-down printed name as Galih's "signature". Cheap partial win: an aspect/density/OCR-ish veto on candidates.
3. Re-tune the ensemble weight. SigNet has now outgrown the hardcoded 50/50 blend (signet xDOMAIN AUC 0.755 vs the frozen dino's 0.664); dino_w≈0.3 measured better on every model. Cheap, no training — but re-check on the 988-officer val before touching _ensemble_score.
4. Re-measure everything only after (1). Every metric here — val, calib2, the anchor — is computed on crops the front-end produced. Fixing the box changes the inputs, so the model comparison should be redone on clean inputs.
5. Keep lr 2e-5 for any future training run (C5's 1.5e-4 is actively harmful), and keep the C6 harness's fixed-seed/ensemble/best-checkpoint selection.
6. Real document↔specimen pairs from the attestation flow remain the durable lever for the metric model itself.

Honest bottom line

Do not ship c6 — keep c5. 4.3× data moved the reliable val +2.9pp and genuinely improved SigNet (xDOMAIN 0.722→0.755), but it did not improve the production ensemble (39.0 vs c5's 40.2, inside noise) and did not fix the real anchor.

The night's real return isn't a model — it's a diagnosis with a proven fix: the score was never mostly about the signature. With the detector's loose block-box, the AI compares stamp + printed text on both sides, which is why a wrong official (Galih) beat the genuine specimen. Tighten the box to the strokes and the genuine ranks #1 and Galih falls to last — no retraining involved.

The score stays advisory either way; the verifikator's visual comparison + manual verdict remain the mechanism. But the ceiling everyone was attributing to the model was substantially a front-end ceiling.

Lesson worth keeping

C5's story is a cautionary tale about selection, not modelling: a broken checkpoint-selector (best seeded to the warm start → best.pth never written) hid a broken hyperparameter (LR 7.5× too high). The training curve was screaming — val fell from epoch 1 while loss collapsed — but with no best-checkpoint to compare against, last.pth shipped and a leaky eval reported it as a win. Fix the measurement before touching the model.