name: obscura-valbox-disk-composition
description: "valbox disk: the 263GB consumer is the SHARED dind sidecar, not Obscura (whose bucket is 414MB). Plus: a rejected MinIO write logs in the APP container, never in minio's own log."
metadata:
node_type: memory
type: project
originSessionId: 835ba1b1-cd57-4e65-8338-0ccb578ef46d
modified: 2026-08-24T14:18:03.871Z

valbox hit 100% disk on 2026-08-24 (~11:40Z) and self-resolved by ~14:00Z when someone
pruned ~190 GB. Reconstructed jointly with Main; both our readings were correct, 2.5h apart.

🔴 A rejected MinIO write logs in the CLIENT, not in MinIO

MinIO refuses a PUT below a reserved free-space threshold and returns the refusal as an API
response. It does not necessarily log its own refusal. So:

• docker logs deploy-minio-1 | grep -i "no space|disk full" → 0 matches, and that proves nothing
• docker logs deploy-obscura-1 | grep "minimum free drive threshold" → the actual evidence

ERROR unhandled error surfaced as 500
  err="blob: put <sha256>: Storage backend has reached its minimum free drive threshold.
       Please delete a few objects to proceed."

⚠️ It is not "disk 100% full" in MinIO's terms — it reserves headroom, so it started refusing
while 3.8 GB of 785 GB was still free.

🔴 My grep vocabulary is a recurring failure mode. I searched blob.*fail and got 0; the real
string is blob: put <hash>: Storage backend… — no "fail" anywhere. I then told Main their report
did not reproduce. Same class of error as grepping for "door" and missing "not the tenant gate"
(see [[obscura-per-tenant-auth]]). When a grep contradicts another agent's direct observation,
suspect the pattern before the claim.

✅ The app failed CLOSED — the outage caused zero data damage

Verified afterwards on x056: 0 document_versions rows created 11:00–13:00Z, and no row
anywhere referencing either failed hash. The blob PUT failed → the whole operation 500'd → no
version row was committed pointing at bytes that do not exist. Three uploads were refused; nothing
needed repair.

Blob-integrity sweep method (x056, 2026-08-24)

MinIO single-drive mode stores each object as a directory named by its content hash, so the
bucket listing diffs directly against the DB. ⚠️ find /data -type f returns 0 — count
directories, not files. Column is document_versions.content_hash (not blob_hash).

ls /data/obscura | grep -E '^[0-9a-f]{64}$' | sort  >  minio.txt
psql -tAc "SELECT DISTINCT content_hash FROM document_versions" | sort  >  db.txt
comm -13 minio.txt db.txt   # DB refs with no bytes  = broken documents
comm -23 minio.txt db.txt   # orphan blobs, no DB ref = garbage

Baseline: 580 DB refs, 1320 objects, 784 orphans (garbage from deleted docs), 44 missing —
42 on deleted docs, 2 on live docs, both SF-PV-* 160-byte Secure Folder test artifacts from
08-07. ⚠️ Those 2 are the only live vault-class versions, so there is no control sample —
cannot distinguish "bytes gone" from "vault blobs keyed differently". Test junk either way.

🔴 Obscura is NOT the disk consumer — the shared dind sidecar is

Obscura's whole MinIO bucket is 414 MB. Of ~348 GB in volumes:

dind-storage is the Docker-in-Docker daemon every agent on this box builds in. It is the
thing to watch when / creeps up. Pruning build cache buys headroom once and does not repeat.

Build cache: pruned, but CI self-heals

The vanished 170.7 GB was the host default builder's cache. Obscura builds use a separate
named obscura-builder (docker-container driver) — which is also now gone, leaving its state
volume orphaned. This does not break CI: .github/workflows/images.yml:303 recreates it when
absent (buildx inspect || buildx rm || buildx create --bootstrap). Next build is slow (cold
cache), not failed. The "hard-fails" note in [[obscura-deploy-build-cache]] is about buildx being
installed at all (release.sh:305), not about the named builder.

No scheduled pruner exists on valbox (no cron, no systemd timer) — the prune was manual or by
an agent. It will not recur on a timer; it can recur from a co-agent.

Related: [[obscura-deploy-build-cache]] · [[obscura-dev-gotchas]] · [[obscura-secure-folders]]