Apostille Signature Fast-Track: Execution Plan

Date: 2026-07-15

Purpose: turn the signature-spesimen research into a production-measurable verifier accelerator

Non-goals: forgery detection, automatic approval, or changing the paid Jalur Permohonan.PERCEPATAN

The short answer

We should execute this in four controlled increments:

1. Preserve real comparisons and human outcomes. The current database stores only the latest score/crops and clears the verdict on a rerun. Add append-only comparison history before collecting more labels.
2. Replace the front end in shadow mode. Keep the current C5 matcher, but add learned handwriting/print/stamp segmentation and compare its score/ranking with the existing k-means path.
3. Only then test a new matcher. If segmentation improves real end-to-end ranking, fine-tune and evaluate DetailSemNet against the segmented C5 baseline.
4. Calibrate a selective quick-review signal. Roll out ranking first, then a strong-match band only when real shadow data proves its precision and reviewer-time benefit.

The first implementation PR should be comparison-history capture and a reproducible real-pair exporter. It changes no model decision and gives every later experiment trustworthy data.

Desired product behavior

The AI never emits FORGED, SUSPICIOUS, or an authoritative MISMATCH. It emits evidence for a strong match, abstains, or reports that the input/reference is unusable. The verifier remains the decision-maker.

Work package 0 — preserve evidence and establish the baseline

Why this must happen first

SignerMatch currently holds the latest signatureScore, crop paths, box, and signatureVerdict. runSignatureVerify() writes new crop files and resets signatureVerdict to null whenever a comparison reruns. That behavior is correct for the current UI state, but it destroys the relationship between an earlier model output, the exact images shown, and the human outcome.

Training from that mutable record would produce incomplete or incorrectly paired labels. Keep SignerMatch as the latest-state projection, but add immutable comparison and review records.

New persistence seam

Add an append-only model conceptually shaped as follows; exact Prisma names can follow repository conventions:

SignatureComparison
  id
  submissionId
  documentId
  signerMatchId
  spesimenId
  modelVersion
  status
  rawCosine
  displayScore
  docFileHash
  specimenFileHash
  docBoxJson
  docBoxSource          AUTO | MANUAL
  specimenDetectionJson
  docCropPath
  specimenCropPath
  docMaskPath           nullable until segmentation exists
  specimenMaskPath      nullable
  qualityJson           nullable
  abstainReason         nullable
  inferenceMs
  createdAt

SignatureComparisonReview
  id
  comparisonId
  verdict               MATCH | MISMATCH | CANNOT_ASSESS
  reason                 nullable enum/text
  selectedSpesimenId     nullable
  reviewer identity      use the existing auth/audit actor seam
  reviewDurationMs       nullable
  createdAt

SignerMatch.signature* fields remain populated for backward-compatible projection. The new row is written in the same backend operation after a successful or soft-failed /verify response. A model rerun creates another row; it never edits the previous one.

Persist fields the service already returns but the backend currently drops:

• model;
• raw cosine;
• specimen_detection, including rotation/box;
• request and reference file hashes;
• inference duration.

Improve the verifier label without changing authority

Extend the current two-button outcome with Tidak dapat dinilai/CANNOT_ASSESS. When the outcome is not MATCH, capture a short reason:

• wrong document crop;
• signature obscured/scan too poor;
• wrong or contaminated specimen;
• correct specimen unavailable;
• visually not consistent;
• other.

This separation is essential. “The crop contains printed text” must not become a negative signature pair, and “wrong registry PDF” must not teach the model that a correct signer is different.

When the verifier selects another specimen, store the selected spesimenId on the review/comparison. That action is valuable ranking ground truth.

Exporter and immutable manifest

Add a read-only script such as:

backend/scripts/export-signature-comparisons.ts

It should produce a JSONL manifest rather than copying data ad hoc:

{
  "comparison_id": "...",
  "person_cluster_id": "reviewed identity key",
  "document_id": "...",
  "spesimen_id": "...",
  "verdict": "MATCH",
  "reason": null,
  "doc_crop": "...",
  "specimen_crop": "...",
  "doc_box": {},
  "model_version": "...",
  "raw_cosine": 0.61,
  "split_group": "stable officer identity"
}

The exporter must:

• include only adjudicated/usable records;
• verify that referenced files exist and hashes match;
• deduplicate identical documents and specimens by hash;
• keep all data on-premise;
• generate deterministic train/validation/test assignments by stable officer identity;
• exclude CANNOT_ASSESS from pair labels while retaining it for quality-model training.

Registry quality for the observed subset

Do not attempt to clean all 132K officers before the pilot. Create a review state for specimen references encountered in the dataset:

VERIFIED | UNCERTAIN | QUARANTINED

Cluster them using a stable reviewed identity based on spesimenId, NIP where available, institution/position, and tenure—not normalized name alone. The pilot only needs trustworthy clusters for its sampled officers.

Work package 0 completion gate

• Every new comparison is immutable and linked to the exact model, crop, box, reference, and human outcome.
• Rerunning inference no longer destroys training evidence.
• MATCH, visual MISMATCH, and CANNOT_ASSESS are distinguishable.
• The exporter reproduces the same manifest and split from the same database snapshot.
• The live score and validation behavior are otherwise unchanged.

Work package 1 — build the real-pair and mask dataset

Collection targets

Collect approximately 500–1,000 usable real document-to-reference comparisons. The important property is diversity, not reaching the number by duplicating easy cases. Sample across:

• Apostille and Legalisasi;
• document types and signer positions;
• clean scans, phone photos, low resolution, rotation, and old documents;
• clean signatures and signatures overlapping print/stamps;
• single-reference and multi-reference officers;
• known same-name, same-institution, and contaminated-registry hard cases.

Freeze an officer-disjoint, adjudicated end-to-end test set before training. It should contain untouched document pages, the actual candidate reference set, corrected tight boxes, and final selected reference—not only pre-cropped positive pairs.

Pixel annotation pilot

Annotate 200–500 difficult ROIs with channels for:

• signature/handwritten ink;
• printed text;
• stamp/seal;
• background/security pattern.

Allow signature and stamp/print channels to overlap. Store masks as lossless PNG plus a JSON/COCO-style annotation manifest. Use the current manual box tool to seed tight-box labels, then correct the pixel masks in an on-prem annotation tool.

Start with a 50-image annotation calibration exercise. Have two annotators label a shared subset, resolve disagreements, and write a one-page annotation guide before scaling to hundreds of masks. This prevents inconsistent definitions of flourishes, initials, stamp overlap, and printed names.

Dataset completion gate

• No officer identity crosses train/test.
• Duplicate file hashes stay in one split.
• At least the frozen test cohort has adjudicated person/reference identity.
• Hard and low-quality cohorts are explicitly tagged.
• Mask agreement and a sample visual audit show that signature-overlap pixels are labeled consistently.

Work package 2 — fix the front end in shadow mode

Lowest-risk architecture change

Keep YOLOS as the coarse page/region proposal initially. Within its proposed block:

1. run a learned multi-channel segmenter;
2. derive a tight box from the handwriting mask;
3. retain overlap pixels that belong to the signature;
4. compute crop-quality features;
5. feed the cleaned signature to the unchanged C5 DINO/SigNet matcher.

This directly tests the observed failure without simultaneously changing the matcher.

Implement the segmenter behind a narrow module such as:

signature-verify-service/app/segmentation.py
  segment_signature(crop) -> masks, tight_box, quality, abstain_reason

Use an environment mode:

SIGVERIFY_SEGMENTER=off | shadow | on

shadow computes and records the candidate result but leaves the existing score/UI untouched. Include the segmenter version in the cache/model version so baseline and candidate results cannot collide.

First model bake-off

Train two small candidates before attempting a large custom network:

• U-Net baseline;
• SegFormer-B0 or equivalent lightweight semantic segmenter.

If both plateau, evaluate the SignaTR6K-style mixed fine-feature/semantic-path architecture. The test is end-to-end; the most sophisticated segmenter is not automatically the winner.

Quality and abstention

At minimum, return:

• handwriting pixel fraction;
• predicted print/stamp leakage into the cleaned crop;
• signature touches ROI edge;
• blur/resolution indicator;
• mask confidence;
• BAD_CROP, LOW_INK, TRUNCATED, HEAVY_OVERLAP, or BAD_REFERENCE reason.

If quality fails, do not compute a reassuring display band. Return BAD_INPUT_OR_REFERENCE and guide the verifier to redraw or select another specimen.

Front-end experiment matrix

Evaluate on untouched pages and real candidate sets. Do not choose B1/B2 using synthetic document transformations.

Work package 2 completion gate

Proposed initial engineering gates:

• tight signature-region recall at least 95% on the frozen test set;
• no material regression on clean documents;
• a clear reduction in printed/stamp leakage on the overlap cohort;
• at least a 10 percentage-point top-1 ranking gain on the difficult/contaminated cohort, with an officer-level confidence interval;
• bad crops are rejected rather than assigned a misleading similarity band.

If B does not beat A end to end, stop and inspect masks/candidate generation. Do not move to a new matcher to hide a failed front end.

Work package 3 — test the local-structure matcher

Only after a segmented baseline wins, port DetailSemNet as a challenger behind a matcher interface:

signature-verify-service/app/matchers/
  c5.py
  detailsemnet.py

The official DetailSemNet project uses an older Python/PyTorch environment. Reuse the MIT-licensed architecture and permitted weights with attribution, but port the minimal inference/training code into the service's supported environment instead of deploying a second legacy runtime.

Fine-tune on real AHU positive pairs and hard negatives. Keep the split writer/officer-disjoint. Compare:

Multiple references

Score each VERIFIED reference independently and persist the full ranked list. Do not use an unconstrained maximum across UNCERTAIN files. The aggregator should consider:

• reference quality;
• agreement among references;
• tenure/registration relevance;
• score margin between rank 1 and rank 2;
• crop quality on both sides.

If references disagree, abstain and show the ranked alternatives. The first useful release can simply preselect the most plausible verified reference for human confirmation.

Work package 3 completion gate

• C beats segmented C5 on top-1/top-k ranking and selective precision, not only AUC.
• Gains hold across clean, stamped, low-quality, and multi-reference cohorts.
• Local correspondence visualization highlights signature strokes rather than printed text/stamps.
• Latency and GPU memory fit the existing service envelope.
• If C does not produce a meaningful real-data gain, keep segmented C5; the front-end improvement still ships value.

Work package 4 — calibrate and roll out the quick-review signal

Shadow first

Run the winning pipeline without changing verifier wording or workflow. For every comparison, record:

• raw matcher output;
• crop/reference quality;
• candidate rank and margin;
• proposed signal/abstention;
• verifier outcome and elapsed review time.

Fit thresholds only on real shadow results. Do not reuse the current synthetic likelihood-ratio knots.

Product metrics

The primary metric is precision among cases receiving STRONG_MATCH_SIGNAL, with an officer-clustered confidence interval. Secondary metrics are:

• coverage: fraction receiving the signal;
• top-1/top-k specimen ranking;
• CANNOT_CONFIRM and bad-input rates;
• redraw/reference-change rate;
• median reviewer seconds;
• verifier override rate;
• throughput per reviewer hour.

Practical pilot targets:

• observed strong-signal precision at least 99%;
• lower 95% confidence bound at least 98%;
• at least 20% useful coverage, treated as secondary to precision;
• at least 25% median review-time reduction for the highlighted cohort;
• no automatic approval and no change to the paid request lane.

These are proposed engineering gates, not claims about current performance. Publish the precision-versus-coverage curve so the threshold can be chosen transparently.

Rollout ladder

1. Data-only: capture comparison history and labels.
2. Shadow: candidate pipeline runs but is invisible.
3. Rank assist: show ranked verified references and quality warnings.
4. Quick-review signal: show a strong-match band on the validated subset.
5. Ongoing monitoring: drift dashboards and regular adjudicated samples.

There is no automatic-approval rung in this plan.

First three implementation PRs

PR 1 — comparison history and label quality

• Prisma migration for append-only comparison/review records.
• Persist model version, raw cosine, detections, hashes, crop paths, and latency.
• Add CANNOT_ASSESS plus structured reasons to the verdict endpoint/UI.
• Preserve SignerMatch as the latest-state projection.
• Backend tests for rerun history, verdict linkage, and authorization.

PR 2 — corpus exporter and annotation seed

• Deterministic JSONL exporter with file/hash validation.
• Stable identity split key and observed-reference quality state.
• Export manual boxes as initial tight-box annotations.
• Freeze the first adjudicated end-to-end test cohort.
• Produce an annotation guide and 50-image agreement pilot.

PR 3 — segmentation shadow baseline

• segmentation.py interface and off|shadow|on flag.
• U-Net/SegFormer training and real-pair evaluation harness.
• Persist masks, quality, abstention reason, and candidate result.
• A/B report for current path versus learned front end using the same C5 matcher.

Suggested first ten working days

This schedule assumes one backend/full-stack engineer plus one CV engineer and access to a verifier or domain reviewer for the annotation/adjudication blocks. If staffing is smaller, preserve the order rather than running the stages concurrently.

Definition of success

The project succeeds when the system saves verifier time on a demonstrably high-precision subset while honestly abstaining elsewhere. It does not need to classify every signature, and it does not need to detect forgery.

The first meaningful proof is not a better public-dataset EER. It is:

On an untouched, officer-disjoint AHU test set, the learned front end reliably isolates handwriting, ranks the verifier-selected reference first more often than C5, rejects unusable crops, and produces a strong-match cohort whose measured precision and review-time benefit satisfy the rollout gates.