Apostille Signature Spesimen Matching: Audit and Recommended Fast-Track Approach

Date: 2026-07-15

Scope: Apostille/Legalisasi signature-image matching for verifier assistance

Non-goal: forgery detection or automatic authenticity decisions

Executive conclusion

The next accuracy investment should not be another generic image encoder. The current system's highest-impact failure happens before the matcher: it often crops the entire signature block, then keeps the darkest pixels. On real Apostille documents this can preserve printed names while deleting blue pen strokes or retaining stamps. A stronger matcher cannot reliably recover a signature it never receives.

The recommended system is:

1. localize a tight signature region using document layout and a detector trained on AHU examples;
2. segment handwritten ink, printed text, stamp, and background at pixel level;
3. compare the cleaned document signature with every verified reference using a local-structure pair matcher;
4. aggregate multiple references and abstain when crop quality, registry quality, or model evidence is weak;
5. show a high-precision quick-review signal, never a forgery verdict, and keep the verifier's confirmation authoritative.

The best first matcher challenger is DetailSemNet (ECCV 2024), because it compares local stroke structures instead of relying only on global embeddings. Its official implementation is MIT-licensed and includes pretrained weights for CEDAR and BHSig datasets. However, the first experiment should keep the current SigNet/DINO scorer and replace only the crop/segmentation stage. That isolates whether the measured production bottleneck has actually been fixed.

Do not deploy a new model from the completed C6 training run. On the honest 82-officer evaluation its production ensemble did not beat deployed C5, and no difference is statistically meaningful at that sample size. C6 is useful research infrastructure, not a validated replacement.

What was inspected

The image-matching implementation is present on the active refactor/tidyup-foundation branch at commit 0358f34d, not on repository master. The legacy Apostille application at /home/efran/remote-development/ahu-codebase/ahu-apostille supplies the registry/master-spesimen workflow but does not contain this image-matching service. This assessment therefore inspected the active implementation read-only and did not modify its service or UI.

The relevant implementation is:

• signature-verify-service/app/main.py: detection, crop isolation, DINOv2/SigNet ensemble, calibration, and inference;
• signature-verify-service/app/signet.py: GPDS SigNet loader;
• signature-verify-service/sigtrain/: registry manifest, augmentation, training, calibration, and C6 experiments;
• backend/src/services/apostille-signature-verify.ts: service integration and persistence;
• backend/src/services/apostille-cross-validator.ts: advisory SPESIMEN_TTD_MATCH rule;
• backend/prisma/schema.prisma: score, crop, box, model artifacts, and verifier verdict;
• frontend/src/components/apostille/signature-compare-modal.tsx: verifier comparison experience.

The overnight C6 training chain has now finished; there was no active training process at the time of this assessment. The live service still reports C5 weights through model version sigverify-v5.1-det-ens30-lrcal-signetc5.

Current pipeline and where it fails

What is already correct

• The result is advisory and the validation rule has a WARNING ceiling; it does not block or reject a case.
• The verifier can redraw the document box and compare raw crops side by side.
• The verifier's signatureVerdict is persisted, providing the beginning of a real feedback corpus.
• Model version, crop artifacts, and the document box are persisted.
• A low model score is not treated as proof that signatures differ.

These are good foundations and should be retained.

Primary failure: the model is often not comparing signatures

The real Darsani test case exposed two independent front-end failures:

• The generic detector returned a roughly 393 × 249 signature block, while the handwritten strokes occupied about 175 × 105. The extra area contained stamp, printed name, and NIP.
• The isolation code used RGB k-means and retained one darkest cluster. On a multicolour scan this preserved black print while discarding blue-black pen strokes. One incorrect registry candidate was represented by an upside-down printed name rather than a signature.

A tight, strokes-only document box changed the genuine specimen from rank 3 to rank 1 and moved the printed-text candidate to last. Increasing padding caused the wrong candidate's score to rise monotonically. That is direct evidence that localization/segmentation is the first lever.

This matches published real-world findings. Formal documents contain stamps, ruling lines, boxes, and printed text over signatures. A CVPR workshop study found that a stamp-cleaning stage reduced equal error rate on stamped targets from 0.33 to 0.23, while an unstamped target scored 0.18; the important lesson is the magnitude of the front-end effect, not the older VGG/CycleGAN architecture itself. See Offline Signature Verification on Real-World Documents and its official code.

The current score is not a probability

Version 5.1 maps ensemble cosine knots to 0–100 and describes the 80 point as a likelihood ratio near 3. This does not mean an 80% probability of a correct match, and it does not by itself establish that a case is “safe.” Posterior probability depends on the operational prior, candidate-generation process, crop quality, and registry contamination. The calibration corpus is dominated by synthetic document transformations and random registry impostors rather than the exact production candidate distribution.

The current >=80 UI tier should therefore be treated as an experimental score band, not “aman untuk jalur cepat,” until production precision and its confidence interval have been measured. This matters especially because a roughly 1% random-impostor tail can still be too large for any automatic bypass. It may be acceptable for prioritizing a human comparison, but not for approving or skipping the required decision.

To avoid confusion with the existing paid Jalur Permohonan.PERCEPATAN, the signature output should be called a quick-review signal or sinyal cocok kuat, not a route assignment. It must never change the applicant's lane automatically.

More registry-sheet training has reached diminishing returns

C6 corrected important experimental defects: checkpoint selection, deterministic validation, evaluation leakage, learning rate, and production-ensemble evaluation. It also expanded the clean set to 5,954 officer groups and 12,235 crops. The result was useful but not a deployable win:

All measurements used only 82 truly disjoint officers, so their differences are within sampling noise. On the larger synthetic validation set, C6 improved modestly, but that gain did not transfer to the one real production anchor. A local SigLIP2 bake-off also did not improve over the signature-trained model, which is expected: language-aligned semantic encoders recognize “handwriting” but are not optimized for stroke identity.

The training labels are not reliable person identities

The manifest groups specimen registrations using normalized officer_name. This can merge homonyms and can label successor/migration contamination as a genuine pair. The normalization keeps only letters a-z, which further collapses distinct identities. With only two sheets in many groups, a consistency filter can determine that they disagree but cannot identify which sheet belongs to the named person.

Synthetic document augmentation then starts from a clean registry crop and reuses the same isolation pipeline. It does not reproduce the production failure in which the detector first includes printed metadata and the isolator removes the signature. Random different-officer pairs are also easier than operational hard negatives such as:

• same or similar name;
• same institution, position, and form template;
• multiple registrations for one officer across time;
• a registration whose PDF contains a successor's signature;
• printed text falsely detected as a signature.

This means the next durable gain requires real document-to-reference pairs and stable identity labels, not just more registry sheets.

Recommended target architecture

1. Tight localization and multi-layer segmentation

Use document-type layout anchors and OCR labels to propose a signer region, then run a detector trained to box handwritten strokes, not the entire signer block. Within that ROI, predict overlapping layers for:

• handwritten/signature ink;
• printed text;
• stamp/seal;
• background or security pattern.

Do not force these into mutually exclusive colours before considering overlaps; a signature can physically overlap a stamp or print. The ICCV 2023 Handwritten and Printed Text Segmentation: A Signature Case Study is directly relevant. Its SignaTR6K dataset was derived from only 200 manually pixel-annotated crops from genuine legal documents, and its four-class formulation explicitly models printed, handwritten, background, and overlap pixels. That makes a 200–500 crop AHU annotation pilot realistic.

Practical bake-off:

• lightweight baseline: U-Net or SegFormer-B0;
• research-aligned challenger: the paper's mixed fine-feature/semantic-path design;
• optional benchmark only: CycleGAN-style cleaning, because generative removal may erase or invent strokes and is harder to audit than a segmentation mask.

Emit a crop-quality record, including signature-ink fraction, printed/stamp leakage, truncation at ROI edges, blur, and segmentation confidence. Bad input must produce ABSTAIN, not a low similarity score.

2. Local-structure pair matching

After segmentation, compare both global shape and aligned local stroke patches. The first challenger should be DetailSemNet because its structural matching explicitly addresses the information loss of global-only representations and produces local correspondences that can be inspected. Fine-tune it on AHU real pairs; public benchmark numbers should not be assumed to transfer to stamped Indonesian documents.

Recommended matcher bake-off:

Variant E can use skeleton/keypoint or RootSIFT-style local alignment with geometric verification. This is complementary evidence that actual stroke fragments correspond; it should not replace learned matching.

3. Multi-reference, quality-aware comparison

Do not reduce an officer to one name-derived prototype or blindly take the maximum score over contaminated files. Maintain an explicit, reviewed person_cluster_id with reference-level states such as VERIFIED, UNCERTAIN, and QUARANTINED.

For a document signature:

1. score every verified reference separately;
2. weight by reference quality and recency/tenure relevance;
3. require consistency across references where several exist;
4. abstain if the best reference is isolated from the rest or if registry identity is unresolved;
5. show the ranked references and the selected reference to the verifier.

The simplest safe initial product is a reference ranker: preselect the most plausible verified specimen and show why, while the human still confirms it. This provides useful time savings before binary thresholds are trustworthy.

4. Selective prediction rather than forced classification

The system should optimize “how many cases can be highlighted at a measured precision?” rather than equal error rate or a forced match/mismatch decision. Paired Contrastive Feature for Highly Reliable Offline Signature Verification applies learning with rejection and top-rank learning specifically to suppress ambiguous signature comparisons. Its exact architecture is optional; its selective-prediction objective is the important fit for this product.

Recommended outputs:

• STRONG_MATCH_SIGNAL: eligible for quick human confirmation;
• CANNOT_CONFIRM: normal manual comparison, no negative implication;
• BAD_INPUT_OR_REFERENCE: crop/scan/registry problem; redraw or choose another reference.

There should be no AI-generated “different,” “suspicious,” or “forged” state.

Data and evaluation plan

Build a small production gold set first

Start with approximately 500–1,000 real document-to-specimen comparisons sampled across document types, scan quality, institutions, time periods, and outcome bands. For 200–500 of the hardest crops, annotate handwriting, print, stamp, and background masks. Each pair should retain:

• stable person/registration identity, not just normalized name;
• document ID and selected specimen ID;
• raw page, proposed ROI, corrected ROI, and masks;
• verifier verdict and, for the gold test set, adjudication;
• scan/template/date/institution metadata;
• exact model and preprocessing version.

Use the existing signatureVerdict, box redraw, and selected-specimen actions as capture points. Do not automatically treat every historical click as ground truth; sample and adjudicate uncertain or conflicting cases.

Split to prevent leakage

• No person/officer may occur in both train and test.
• Also stratify or hold out time periods and form templates so the test does not merely repeat layout.
• Keep repeated uploads and identical file hashes in one split.
• Construct hard negatives from the actual candidate generator, especially same-name and same-institution cases.
• Freeze the end-to-end test pages before model development and never synthesize over them.

Measure the product objective

Bootstrap confidence intervals by officer, not by pair, because pairs from one officer are correlated. Report results separately for clean, stamped/overlapped, low-resolution, and contaminated-registry cohorts.

For the first release, the acceptance criterion should be a pre-agreed lower confidence bound on precision for the highlighted cohort, plus a measurable reduction in reviewer time. Coverage is secondary: abstaining on most cases is acceptable if the highlighted subset is genuinely reliable. Any future proposal to skip human confirmation would require a separate legal/product decision and a substantially stricter validation program; it is not recommended here.

Implementation sequence

Work package 0 — fix measurement

• Freeze current C5 as baseline A.
• Create stable person/reference IDs and quarantine known contaminated specimens.
• Assemble and adjudicate the first real-pair test set.
• Replace percentage language with an experimental score band until calibrated on that set.

Work package 1 — solve the front end

• Annotate the initial AHU crop masks.
• Train U-Net/SegFormer and the SignaTR6K-style challenger.
• Compare A versus B using the same C5 matcher.
• Add explicit crop-quality abstention and expose masks in the verifier UI.

Proceed to a new matcher only if B improves real end-to-end ranking and reduces printed/stamp leakage.

Work package 2 — matcher bake-off

• Fine-tune DetailSemNet on real, officer-disjoint AHU pairs.
• Compare C against B; optionally evaluate SURDS as D.
• Mine operational hard negatives rather than random registry negatives.
• Evaluate multiple references independently and add reference-quality aggregation.

Work package 3 — shadow calibration and rollout

• Run the winner in shadow mode while preserving the verifier's normal workflow.
• Fit calibration/rejection thresholds only on real shadow data.
• Publish precision-coverage curves and reviewer-time results.
• Enable only the high-precision quick-review highlight; retain manual comparison for all abstentions.
• Monitor drift by document type, scan source, institution, and registry cohort.

Immediate decisions and non-actions

1. Keep C5 deployed for now. C6 did not establish a production win.
2. Do not spend the next cycle on CLIP/SigLIP or a larger generic ViT. Local results already show that semantic pretraining is not the missing capability.
3. Do not treat the current 80 score as 80% probability or “safe.” Recalibrate after a real-pair gold set exists.
4. Do not train positives by normalized name alone. Introduce reviewed person/reference clusters first.
5. Build learned ink/print/stamp segmentation before changing the matcher. This is the best-supported and highest-leverage next experiment.
6. Use DetailSemNet as the primary challenger after segmentation. Its local structural matching is a better fit than global cosine for handwritten strokes.
7. Ship ranking and abstention before automation. The verifier remains the decision-maker, consistent with the stated fast-track purpose.

References

• Gholamian et al., Handwritten and Printed Text Segmentation: A Signature Case Study, ICCV 2023.
• Shih et al., DetailSemNet: Elevating Signature Verification through Detail-Semantic Integration, ECCV 2024; official MIT-licensed code and weights.
• Engin et al., Offline Signature Verification on Real-World Documents, CVPR Biometrics Workshop 2020; official code.
• Lai et al., SURDS: Self-Supervised Attention-guided Reconstruction and Dual Triplet Loss for Writer Independent Offline Signature Verification, 2022; official code.
• Ji, Suehiro, and Uchida, Paired Contrastive Feature for Highly Reliable Offline Signature Verification, Pattern Recognition 2023.