Smart Upload Assistant — Design

Date: 2026-07-03
Status: Approved (brainstorm with user)
Modules: ai (license-gated, end-to-end) · core dms upload flow

Goal

When a user drops a file into the Upload Document modal, Obscura analyzes it with AI and
proposes a summary, a classification, and tags — which the user reviews, edits,
and confirms before the document is saved. Nothing is persisted until Save; cancelling
leaves zero rows. The already-extracted text is reused on Save so the file is never
OCR'd/extracted twice.

Title and folder suggestions already exist (filename autofill + FolderSuggestBox) and are
unchanged. Key-entity extraction stays out of the modal (the background enrichment job still
produces entities for the doc-detail AI Insights card).

Decisions (made during brainstorm)

1. Trigger: automatic on file drop, admin-toggleable. Analysis auto-runs when the file
   lands in the dropzone. A new admin AI setting (upload_analyze_enabled, default true)
   can switch it off org-wide.
2. Suggestions in scope: classification, tags, summary. Entities (parties/dates/amounts)
   are explicitly out of scope for the modal.
3. Architecture: stateless analyze (Option A). A new endpoint analyzes the raw bytes and
   caches the result server-side under a short-lived analysis_id. The document is created
   only on Save, with the confirmed values. No orphan documents, no staging table.

Architecture & data flow

Upload modal (file dropped)
  │  auto-run iff: `ai` module enabled (from /me) AND admin toggle on
  ▼
POST /api/v1/ai/analyze-upload          multipart: file (+ optional title form field)
  │  server, synchronous, nothing persisted:
  │    1. extract-sidecar: SidecarExtractor.Extract(bytes, mime, filename) → text
  │    2. enrich.Analyze(title, text):
  │         ai.Summarize(text)                → summary        (metered: "summarize")
  │         ai.ClassifyDocument(title, text)  → {sensitivity, tags}  (metered: "classify")
  │    3. analysisCache.Put(analysisID, {ownerID, text, summary, classification, tags, model})
  │       TTL 30 min, owner-scoped
  ▼
{ analysis_id, summary, classification, tags, text_chars }
  │
  │  user edits summary / accepts or overrides classification / picks tags, folder, title
  ▼
Save (client sequence, mostly existing):
  1. POST /api/v1/documents                    ← classification = confirmed value
  2. POST /api/v1/documents/{id}/versions      ← multipart "file"
       + form fields: analysis_id, summary (the possibly-edited text)
       server, when analysis_id resolves in cache AND cache owner == caller:
         a. seed documents.content_text from cached text (clip 800 KiB)
            + mark content_extracted_version = current  → async extract skipped, no double OCR
         b. upsert document_enrichment {summary, suggested_classification, suggested_tags,
            model} + set its version marker → hourly enrich sweep skips this version
       cache entry consumed (deleted) on use; cache miss/expiry → normal async pipeline,
       upload still succeeds (201)
  3. existing tags mutation applies the confirmed tags

The cache serves two purposes: it carries suggestions between analyze and Save, and it lets
Save reuse the extracted text. Losing it (restart, TTL) degrades gracefully to today's async
pipeline — never an error.

Components

Backend — new

• POST /api/v1/ai/analyze-upload (go/internal/httpapi/handlers_ai_analyze.go)
• Gates: session auth → requireModule("ai") → admin toggle (see Settings) → upload
  validation (same extension allowlist + size limit as AddVersion).
• Reads the multipart file (+ optional title form field), runs extract → analyze,
  stores the cache entry, returns:
  {"analysis_id": "...", "summary": "...", "classification": "internal", "tags": ["..."], "text_chars": 12345}.
• Empty extraction (scan with OCR off, unsupported format) → 200 with
  {"analysis_id": "", "no_text": true} — the UI shows "couldn't read text".
• Toggle off → 409 ai.upload_analyze_disabled. Budget exhausted → the existing
  429 ai.budget_exhausted from meter(). Provider/sidecar failure → 502-mapped
  kernel error ai.analyze_failed. All are soft-failures in the UI.

• Endpoint timeout bounded (extract 60s + two model calls); handler uses the request
  context so an abandoned modal aborts the work.

• enrich.Service.Analyze(ctx, title, text) (Analysis, error)
  (go/internal/enrich/app/service.go) — same Summarize + ClassifyDocument calls the
  backfill uses (same clipping constants), but returns
  Analysis{Summary, Classification, Tags []string, Model} without touching the store.
  Skips ExtractFields (entities out of scope; saves a model call on the hot path).
  Respects the existing enabled() live-license check.

• analysisCache (go/internal/httpapi/analysis_cache.go) — in-process, mutex-guarded
  map keyed by random analysis_id (UUID). Entry: {ownerID, filename, text (≤800 KiB), summary, classification, tags, model, expiresAt}. TTL 30 min; max 64 entries (evict
  oldest); entries removed on use. Owner check on read: a caller can only consume an entry
  it created. Lives in httpapi because both the analyze and AddVersion handlers (delivery
  layer) are its only users. Single-instance by design (Obscura is a single-tenant
  monolith).

Backend — modified

• AddVersion handler (go/internal/httpapi/handlers_dms.go): accept optional
  analysis_id + summary multipart form fields. After the version commit succeeds,
  resolve the cache entry (owner-scoped); on hit: call dms.SetContentText with the cached
  text (which also fires the embed hook → immediate semantic indexing), mark
  content_extracted_version, and call enrich.Service.RecordConfirmed(ctx, docID, version, analysis, summaryOverride) — a new service method that upserts the
  document_enrichment row (user-confirmed summary wins over cached AI summary) with its
  version marker, keeping the enrich store private to its context. All best-effort:
  failures log and fall back to the async pipeline; the upload response is unchanged.
• AI settings (go/internal/ai/app/store.go, adapters/pg.go,
  httpapi/handlers_ai_admin.go): new upload_analyze_enabled bool on the Settings
  struct, GET/PUT wiring, no extra validation beyond bool.
• Migration 00078_ai_upload_analyze.sql:
  ALTER TABLE ai_settings ADD COLUMN upload_analyze_enabled BOOLEAN NOT NULL DEFAULT TRUE;
• wire.go: hand the enrich service (for Analyze + RecordConfirmed) to httpapi
  Deps; construct the analysisCache.

Frontend

• web/src/api/ai.ts: analyzeUpload(file, title?) — raw fetch multipart (same
  pattern as uploadVersion; openapi-fetch would JSON-serialise), returns the typed
  response; treats 409 ai.upload_analyze_disabled as {disabled: true} and memoizes that
  for the session so later uploads don't re-attempt.
• web/src/api/files.ts: uploadVersion(docID, file, extra?) gains optional
  {analysisId, summary} appended as FormData fields.
• UploadDocumentModal.tsx: on file drop, when moduleEnabled('ai'), call
  analyzeUpload and render an AI suggestions panel under the existing fields:
• Analyzing… inline skeleton/spinner state (upload remains fully usable; Save never
  waits on analysis).
• On result: editable Summary textarea; the modal's existing Classification
  select pre-set to the suggestion with an "AI" tag — but only when the suggested value
  matches an entry in the (admin-managed, dynamic) classification registry; a suggestion
  outside the registry is silently dropped. User can override; DLP declassify rules
  unchanged — the server still enforces on create. Tags as clickable chips (all
  pre-selected, click to exclude).
• Soft-fail states: no_text → "Couldn't read any text from this file"; 429/502 →
  "AI suggestions unavailable"; disabled/unlicensed → panel not rendered at all.
• Save passes the confirmed classification into the create body (existing field), then
  uploadVersion(..., {analysisId, summary}), then applies selected tags via the
  existing tags mutation.
• AiTab.tsx (Admin → AI → Settings): a toggle "Analyze uploads with AI" bound to
  upload_analyze_enabled.
• i18n: en/id strings co-located per the features/*/i18n.ts pattern.

Error handling

Analysis is always optional decoration on the upload flow:

Security

• analyze-upload enforces the same extension allowlist and size cap as AddVersion — it
  must not become a laxer file-ingestion path.
• Cache entries are owner-scoped: only the creating principal can consume an
  analysis_id; a guessed/foreign id is ignored (normal async path, no error leak).
• The user-supplied summary form field is stored as the enrichment summary (plain text,
  rendered safely by the existing AI Insights card) — equivalent trust to any other
  user-authored metadata.
• Server-side classification rules (declassify gate) still apply on create; an AI
  suggestion never bypasses them.

Testing (repo discipline: never go test against the live demo DB)

• Build/vet: cd go && go build ./... && go vet ./....
• Curl e2e on the deployed stack: analyze a small PDF → assert suggestions + usage rows
  (ai_usage day/feature summarize+classify increment); Save with analysis_id → assert
  content_text seeded, content_extracted_version marked, document_enrichment row
  matches the edited summary, doc-detail AI Insights shows it; cancel path → assert no
  rows anywhere; toggle off via PUT settings → 409; foreign analysis_id → normal upload.
• Frontend: npx tsc --noEmit && npx vite build; manual click-through of analyzing /
  done / no_text / failed / disabled states.
• Post-deploy invariant: /me enabled_modules == [ai, correspondence, esign, semantic, watermarking], demo intact.

Out of scope (deliberate)

• Key entities in the modal (background enrichment still extracts them for doc detail).
• The Write-document (editor) flow — upload only.
• Multi-instance/distributed cache — single-process map is correct for this deployment
  model.
• A manual "re-analyze" button — replacing the file re-triggers analysis.
• Auto-applying anything without user confirmation.