think
16px
820px
← all files

Document content extraction — Design

Date: 2026-07-01
Status: Approved (design); pending implementation plan.

Goal

Populate documents.content_text from uploaded files (PDF/Office/text) via a small air-gapped
extraction sidecar, so document search ranks on content, not just titles. Extracted text flows
through the existing SetContentText seam, which updates the keyword search_tsv index (core) and
re-fires the embed hook (semantic re-embed). Today all 258 demo docs have empty content_text, so
both keyword and semantic search are effectively title-only; this fixes that at the source.

Decisions (locked during brainstorming)

  1. Core infrastructure, not a license module. Config-driven (EXTRACT_PROVIDER: sidecar | none,
    mirroring the embed sidecar's provider selection). It improves the keyword full-text index
    for everyone AND feeds the semantic embeddings (when licensed). EXTRACT_PROVIDER=none is a
    graceful no-op (title-only — today's behaviour).
  2. v1 formats = PDF + Office + text (SeedDMS parity, no OCR). PDF (pypdfium2), docx/xlsx/pptx
    (python-docx/openpyxl/python-pptx), txt/csv/md/html (decode + strip). Unsupported (images, legacy
    .doc/.xls/.ppt, ODF, rtf) → empty text (best-effort). OCR (Tesseract) is a documented future
    differentiator. (SeedDMS itself delegates to CLI converters — pdftotext/catdoc/docx2txt — with no
    default OCR; our sidecar is the modern packaged equivalent.)
  3. Mirror the embed pipeline exactly — sidecar + config + Go context + an on-upload hook +
    a backfill sweep — so the shape is already proven.

Reused / mirrored (existing) systems — do NOT rebuild

  • SetContentText write seam: dms/app/service.go SetContentText(docID, text) bounds to
    maxContentTextBytes (800 KiB) and fires the embedHook. The extractor calls this Service
    method directly (in-process) — no HTTP round-trip. (The PUT /content-text route stays for the
    external/legacy path.)
  • Version bytes: dms/app/service.go OpenVersionContent(docID, version) (io.ReadCloser, err)
    reads the blob by the version's ContentHash; the version's MIME comes from GetVersion. The
    extract context reaches these through a narrow dms port (below).
  • Embed-pipeline templates: deploy/embed-sidecar/ (app.py/Dockerfile/requirements),
    deploy/docker-compose.yml (the embed-sidecar service + EMBED_* env + soft depends_on),
    platform/config/config.go EmbedConfig, ai/adapters/embed_sidecar.go (HTTP client),
    cmd/obscura-server/wire.go (provider build + SetEmbedHook + the semantic.embed_backfill
    scheduler job + boot trigger), cmd/obscura-server/jobs.go runEmbedBackfill,
    semantic/app/service.go OnContentText (async/detached/non-fatal hook), and the dms
    embedHook field + SetEmbedHook setter. Each has a direct analogue below.
  • No in-repo extractor exists (confirmed); the office render path (Gotenberg) is bytes→PDF, the
    opposite direction.

Component 1 — the extract-sidecar

deploy/extract-sidecar/ (FastAPI, mirroring embed-sidecar):
- POST /extract — multipart: a file part (the raw bytes) + a mime form field → {"text": "…"}.
Dispatches by MIME/extension: PDF → pypdfium2 (page text); .docx → python-docx; .xlsx
→ openpyxl (cell text); .pptx → python-pptx (shape text); text/*, .csv, .md →
decode; .html/.htm → strip tags. Anything else → {"text": ""} (best-effort, never errors).
- GET /healthz → {status, formats}.
- Dependencies baked into the image (air-gapped: no network at runtime). Own build context
(deploy/extract-sidecar/, like embed-sidecar, because the repo-root .dockerignore excludes
deploy/). Compose service extract-sidecar with a python-urllib healthcheck + host port mapping.

Component 2 — ExtractConfig

platform/config/config.go, mirroring EmbedConfig:

EXTRACT_PROVIDER     (default "none"; compose sets "sidecar")   // none disables extraction
EXTRACT_SIDECAR_URL  (default "http://localhost:38001")         // compose: http://extract-sidecar:8000
                                                                // (host port 38001, next to embed-sidecar's 38000)

A Validate switch rejects an unknown provider (want none|sidecar). Go default none keeps
go run/tests dependency-free; compose overrides to sidecar so the demo extracts.

Component 3 — the internal/extract context

A new, focused context (single responsibility: turn a version's bytes into content_text),
structured like internal/semantic:
- Service holds an Extractor (the sidecar client) + a DocSource port (the narrow dms slice
it needs) + the provider mode + a logger.
- OnVersionAdded(docID string, version int) — the AddVersion hook. Async/detached/non-fatal;
a no-op when provider=none. Fetches the version's bytes + MIME via the port, POSTs to the
sidecar, clips to maxContentTextBytes, and calls SetContentText(docID, text) (which updates
search_tsv and fires the embed hook). Extraction failures are logged, never fatal.
- RunBackfill(ctx, limit) (int, error) — extracts documents that have a file version but no
content_text yet (the existing-docs catch-up). Page-capped + logged.
- Extractor adapter (internal/extract/adapters): an HTTP client POSTing multipart to the
sidecar (mirrors SidecarEmbedder), returning the text.
- DocSource port (declared in extract/app, satisfied by the dms Service): a method to read a
version's bytes + MIME (backed by OpenVersionContent + GetVersion), and a method listing docs
needing extraction (backed by a dms query current_version > 0 AND content_text = ''). SetContentText
is reached the same way (a port method or the dms Service passed in).

Component 4 — the dms extractHook

Add to dms/app/service.go (mirroring embedHook):
- field extractHook func(docID string, version int) + setter SetExtractHook(fn …).
- fired in AddVersion after the transaction commits (best-effort, non-fatal) with the new
version number. This is the one new dms hook.
- A new dms query DocIDsNeedingExtraction(limit) (current_version > 0 AND content_text = '',
newest first) + a Service wrapper, for the backfill.

Component 5 — wiring + backfill

cmd/obscura-server/wire.go (mirroring the embed wiring):
- Build the Extractor from cfg.Extract (sidecar client, or a nil/no-op when provider=none),
construct extractSvc := extractapp.NewService(...) with the dms port.
- dmsSvc.SetExtractHook(extractSvc.OnVersionAdded).
- Register schedulerSvc.Register("dms.extract_backfill", 1*time.Hour, runExtractBackfill(...)) +
EnsureRegistered + TriggerNow on boot (catch up the existing corpus).
- cmd/obscura-server/jobs.go: runExtractBackfill (mirrors runEmbedBackfill).

Data flow

Upload → AddVersion (bytes → blob) → extractHook(docID, version) → extract.OnVersionAdded
(async) → fetch bytes+MIME → sidecar /extract → SetContentText(docID, text) → (a) search_tsv
updated (keyword search now matches content) and (b) embedHook → semantic re-embed with real
content. The backfill sweep replays this for the existing 258 docs that have a file version, so
after one deploy both keyword and semantic search jump in quality. Documents with no uploaded file
(stubs, in-app text drafts) have nothing to extract and are unaffected.

Error handling & performance

  • Best-effort throughout: sidecar/parse failures are logged and non-fatal — the document simply
    stays title-only. Unsupported MIME → empty text (not an error).
  • The sidecar client has a timeout; the hook is detached (never blocks the upload response), exactly
    like the embed hook.
  • content_text is clipped to 800 KiB (the tsvector bound) before write.
  • Backfill is page-capped and idempotent (re-running only touches docs still missing content_text).

Gating

None — this is core infrastructure. No requireModule. EXTRACT_PROVIDER=none disables it
gracefully (a deployment that doesn't want the sidecar just runs without content extraction). The
downstream embed step remains semantic-gated as before, so content extraction on a non-semantic
deployment still benefits keyword search.

Out of scope (documented futures)

  • OCR (Tesseract) for scanned PDFs + images.
  • Legacy .doc/.xls/.ppt, ODF (.odt/.ods/.odp), and rtf (best-effort empty in v1; would need
    LibreOffice/unoconv or textract).
  • Admin-configurable per-MIME converters (SeedDMS-style command templates) — we ship a fixed sidecar.
  • Structured/per-page extraction, language detection, table-aware extraction.

Testing / verification

Per repo discipline: never go test (writes the live demo Postgres). Verify via
go build ./... && go vet ./...; build the extract-sidecar image + smoke /healthz and /extract;
and a deployed e2e:
- Bring up extract-sidecar; upload a text-rich PDF (or docx); assert documents.content_text is
populated within a few seconds; assert keyword search now matches a word from the body (not the
title); assert semantic search surfaces it for a content paraphrase (with the embed re-fired).
- Run the backfill and confirm existing file-backed docs gain content_text.
- Confirm EXTRACT_PROVIDER=none is a clean no-op (no extraction, no errors).
- After every deploy assert /me enabled_modules unchanged (the 5 modules) + demo intact; clean up
test docs.