Akta mislabel audit — staging, 2026-08-13
Scope: all 200 documents in staging with OCR text → 33 distinct texts by content hash → 15 distinct akta deeds (the rest are KTP/bukti setor/apostille/etc.). Read-only. Deliberately not reusing the engine's own classifier — checking labels with the code that produced them would be circular. Independent title extraction (anchor-relative, before Pada hari ini / Berhadapan dengan saya) plus body action-verb counts and filename cross-check.
Headline
| Distinct akta deeds audited | 15 |
| Confirmed mislabels | 2 |
| Probable mislabel | 1 |
| Unproven suspect | 1 |
| Mislabel rate (confirmed) | 13% of distinct deeds — vs the 2.34% field correction rate |
| Every mislabel's stored label | perubahan |
The mechanism: perubahan is a sink
classifyAktaTransactionType falls back to opts?.gpuPrior ?? "perubahan". Everything the classifier cannot decide lands on perubahan, so that label accumulates the failures. All four suspects are labelled perubahan; no other label has a single suspect.
This also explains why the field-level correction rate (2.34%) misses it: the document label is wrong, not the extracted field values, so field-by-field review never contradicts it.
Findings
| hash | stored | evidence | verdict |
|---|---|---|---|
b940a54b |
perubahan | title PEMBUBARAN PERSEROAN TERBATAS "PT. FAMILY LOMBARD BALI", body membubarkan×1, filename …PEMBUBARAN-PT-LOMBARD |
mislabel — pembubaran |
112c9474 |
perubahan | body PENGGABUNGAN×41 (every other deed: 0), filename MERGER_…PKRUPSLB-DAN-PENEGASAN |
mislabel — penggabungan |
1e64150a |
perubahan | title …PT VISI TEKNOLOGI GLOBALTAMA (DALAM LIKUIDASI), body likuidasi×7, filename …(pembubaran) |
probable — liquidation, not perubahan |
d90b6736 |
perubahan | filename PEMBUBARAN_…PKRUPS-PT-JASA-CAHAYA only; body has no action verb |
unproven — filename may be wrong, or OCR dropped the verb |
112c9474's 41 hits are not the anggaran-dasar boilerplate ("Penggabungan, Peleburan dan Pengambilalihan" appears once per deed): every correctly-labelled deed in the corpus counts 0.
Correction to an earlier claim
I previously said a human had reviewed and confirmed the PT LOMBARD mislabel. That is wrong. All four suspects are VERIFICATION_READY with 0 confirmed and 0 edited fields — nobody has reviewed them at all. They are sitting in the queue mislabelled, not rubber-stamped through it.
This weakens the "review rubber-stamps errors" story for these documents, but it does not rescue confidence: the labels carry 0.913–0.988 overallConfidence while being wrong. The click-through finding (73% of confirms under one second) still stands on its own evidence from the 728 reviewed fields.
What the audit could NOT check
10 of 15 deeds have non-determinative titles — PERNYATAAN KEPUTUSAN … / BERITA ACARA RAPAT …. Those titles say the shareholders resolved something, not what. For them the label rests entirely on the LLM and cannot be verified from the title. The filename cross-check is what surfaced 3 of the 4 suspects; without it the title audit alone found only 1.
So "2 confirmed mislabels" is a floor, established over the 5 deeds whose titles are determinative plus whatever the filename and verb heuristics happened to catch. The true rate over the 10 deferred deeds is unmeasured.
Recommended follow-ups (engine work — needs a go-ahead)
- Stop defaulting to
perubahan. A fallback that silently adopts the most common label converts "unknown" into a confident wrong answer. Emit an explicitunknown/low-confidence state the UI can surface instead. - Use the filename as a cross-check signal. It is free, it is operator-supplied, and it caught 3 of 4 suspects here. Disagreement between filename and label should flag for review, never silently lose.
- Re-label the 2 confirmed (and adjudicate the other 2) before any KLH demo touches this corpus.
- Verify the 10 deferred deeds by hand — that is the only way to convert the floor into a rate. 10 documents is an afternoon.
Document ids are in the audit output; b940a54b = 31cce4e0-ee37-4cf2-b5fd-c486eae6db1d.