Object storage for engine documents — infra answer (2026-08-13)

Question routed from the OCR session: should we have used MinIO for document storage?

Verdict: adopt object storage, but not now, and not on either of the two options that were proposed. Sequence it behind the production migration and the reconciler. Meanwhile, the survey done to answer this question turned up something more urgent than the question itself — see The finding that outranks the question.

Three corrections to the premise

The OCR session framed this as "extend Artesca vs stand up a separate MinIO — and AHU already has MinIO live." All three parts need adjusting.

The multi-replica claim, checked

This was the strongest argument in the OCR session's list, so it deserved verification rather than agreement. Shared storage alone does not make the OCR backend safe to run at two replicas. Two unguarded in-process schedulers would misbehave the moment a second replica boots:

backend/src/persistence/outbox-drain.ts:131
  const timer = setInterval(async () => {
    if (running) return;        // <- per-PROCESS guard only

The comment calls this "overlap-safe", and within one process it is. There is no advisory lock, no SELECT … FOR UPDATE SKIP LOCKED, no leader election — so two replicas hold two independent running flags and claim the same outbox rows twice. Duplicate outbound side effects.

backend/src/security/uploads-retention.ts:80
  setInterval(run, SWEEP_INTERVAL_MS);

Every replica runs its own retention sweeper. Two concurrent sweepers racing on deletes — which is precisely the class of bug that caused today's incident, now multiplied by replica count.

So "the OCR backend cannot run more than one replica today without shared storage, full stop" is true but incomplete. Storage is blocker #1 of ≥3, and moving to MinIO would fix the one that loses no data while leaving the two that do.

The finding that outranks the question

ahu-backup-01 was delivered, MinIO was deployed, a bucket was created — and nothing has ever been written to it.

minio_cluster_bucket_total          1
minio_cluster_usage_object_total    0          <- zero objects, ever
minio_cluster_capacity_usable_free  1.99 TB / 2.00 TB
minio_cluster_drive_offline_total   0

There is no backup client anywhere in platform-infra/ — no restic, no pgbackrest, no wal-g, no mc config. The repo contains exactly two files for this host: the MinIO server compose and the credential renderer. The server is up; the thing it exists to receive was never built.

The design intent (2026-07-09-devops-server-request-2.md) was PITR/WAL streaming from ahu-gov-01's TimescaleDB — the audit-integrity store — plus pg_dump from ahu-int-01's Postgres/pgvector. TimescaleDB is classed "crown jewels" in the architecture doc. Per the PDP assessment, ai_call_bodies in that store holds NIK.

Net position: the audit store containing citizen NIK has a 2 TB backup target sitting empty, and no job pointed at it. That should be confirmed on-host (.121 is not SSH-reachable from where I can currently probe, so I can only prove the destination is empty, not that no backup exists by some other route). But nothing in the infra repo suggests another route exists.

This is a bigger exposure than the document-storage question that prompted the survey, and it is cheap to close because the destination is already running.

Two more gaps found in the same sweep

1. The MinIO is plaintext HTTP. No TLS config, no cert volume, http://…:9000 answers 200. It is intended to carry backups of a store holding NIK, across hosts. That needs TLS before it carries its first byte — which, conveniently, is right now.

2. Firewall drift on the backup zone. The stated posture is push-only from .121/.122, DWH-grade isolation, explicitly denied from DMZ/internet. From the GPU host (192.168.83.20, a different subnet entirely):

SSH-via-bastion-only holds. But service ports are broadly reachable from 83.x, including the backup zone that is supposed to be the most isolated thing we have. Either that is an intentional allow I should document, or it is drift I should close. Given the host is meant to hold audit/PII one hop removed, I lean close it.

(Note the double edge: this same open path is what would make MinIO adoption cheap for OCR — no new network work needed. Convenience and posture are pointing in opposite directions here, and posture should win.)

The actual three-way decision

A — Artesca: keep it strictly as a read-only upstream for specimen sheets. That is what it is good for and it already works. Do not put PoC upload churn into AHU production — Efran's instinct here was right. The cert expiry is a concrete date worth tracking regardless, because the specimen fetch path depends on it: the custom SigV4 client in lib/s3-signed-fetch.ts exists precisely because that chain is incomplete, and a CA rotation on Oct 3 is a live break risk for a path we already depend on.

B — the existing MinIO: do not repurpose it. Putting live application traffic on the backup target collapses two failure domains into one; a runaway uploads bucket would then threaten the PITR destination. It is also single-drive mode — zero parity. That is acceptable for backup copies of data that exists elsewhere. It is not acceptable as the only copy of citizen PDFs.

C — a second MinIO, in the internal-engines zone. Cheap, because the pattern is already written and proven: copy compose.minio.yaml, same OpenBao AppRole cred rendering, same Prometheus scrape, different bucket layout, TLS on, lifecycle rules from day one. The marginal cost is low precisely because of the work already done for the backup host.

Why later, not now

1. Nothing is blocked today. OCR runs one replica and is not close to needing two — and as shown above, storage is not the only thing standing between it and a second one.
2. The disk pressure is not real pressure. klasifikasi-temp is 903 MB on a host with 87 GB free — about 1% of headroom. A lifecycle rule is genuinely the nicer solution, but a sweeper solves it this week, and a sweeper has to be written anyway (point 4). The marginal saving from going declarative is small.
3. The prerequisite is the production migration, not the object store. OCR still runs on the GPU host (ahu-ai-ocr-web on 192.168.83.20), not on ahu-int-01 where the network map puts it. Migrating storage before migrating the app means doing the storage work twice and pushing upload traffic across 83→82 in plaintext. Move the app first; the storage decision gets simpler once the app sits in the zone that owns its data.
4. The reconciler should be funded first, and it is storage-independent. The OCR session's own closing point is the right one: 20% orphan rate moves into a bucket where ls and du no longer work. Their fitness test is the real asset here — it should be generalized to a reconciliation job before the backend changes, not after.

Full agreement with the OCR session on the core technical point: S3 has no foreign keys. Object storage relocates the orphan, it does not eliminate it. And their diagnosis is right that this would have been harder to find in a bucket — that argues for building the reconciler while the data is still on a filesystem you can du.

Proposed split of work

Infra (me) — do now, independent of the MinIO question:

1. Wire the actual backup jobs to ahu-minio. The destination has been running empty since 2026-07-13. Highest value, lowest effort, already-paid-for hardware.
2. TLS on ahu-minio — before it carries its first byte of NIK-bearing audit data.
3. Resolve the 83.x → 82.126:9000 path: close it, or document it as intentional.
4. Track the Artesca cert expiry (2026-10-03) as a dependency of the live specimen-fetch path.

Infra (me) — do when the migration lands:

5. Stand up MinIO #2 for engine documents in the internal-engines zone, with TLS and lifecycle rules on temp/derived prefixes from day one.
6. Publish the bucket + prefix convention across engines (ocr / apostille / classifier) and a reusable client that handles the Artesca chain — the constraint that ruled out stock Bun.S3Client.

OCR session — do now, storage-independent:

7. Generalize the fitness test into a standing reconciliation job. This is the one that must not wait for a storage decision.
8. If klasifikasi-temp growth becomes real before MinIO #2 exists, a sweeper is fine — write it so its policy is portable to an S3 lifecycle rule later.
9. Before any multi-replica work: put a cross-process lock on outbox-drain and single-owner the retention sweeper. Storage is not the blocker people think it is.

Evidence appendix

All probes run 2026-08-13 from efran@192.168.83.20.