Stego Engine — Tech Stack & Architecture Spec

Companion to STEGO_ENGINE_FEASIBILITY.md. That doc said whether/what; this one says
how: the locked stack, the internal architecture, and — per explicit requirement —
how the engine is structured so any part of it can be re-coded from scratch without
losing anything that matters. Priorities: security → consistency → compatibility →
speed. Written 2026-07-10.

0. The one design principle everything follows

Code is disposable; artifacts are durable. A watermark engine has a property most
software doesn't: documents encoded today must still decode in ten years, after any number
of rewrites. So the design splits the world in two:

If the implementation is ever "messed up," the recovery procedure is mechanical: re-implement
against WIRE_SPEC.md until the golden corpus passes. Nothing issued in production is ever
invalidated, because what production documents depend on is the wire spec, not the code.

1. Tech stack (locked)

1.1 Sidecar (the engine)

Banned (AGPL/GPL): PyMuPDF/fitz, Ghostscript, poppler/pdftoppm, unidoc/unipdf. CI enforces
via a dependency-allowlist check (§7). Compliance chore: ship PDFium's license files, pin the
pypdfium2 build.

1.2 Quality tooling (dev cost is not a constraint — spend it here)

• pytest + hypothesis (property tests: random PDFs/fonts/ε → encode must produce
  byte-valid PDF; encode→decode must round-trip; PDF with no watermark must decode to nothing).
• mypy --strict + ruff. Full type annotations at every package boundary.
• Golden-corpus harness as a pytest plugin (§6) — the real spec.
• Determinism check in CI: same input + same key ⇒ byte-identical output (no hidden
  randomness/time — required for reproducible forensics and for corpus stability).

1.3 Go side (DMS)

Nothing new. One adapter: internal/protection/adapters/grpc.go implementing the existing
app.ProtectionEngine interface as a gRPC client (mTLS, deadline, traceparent +
issuance-id metadata per the proto's convention). Selected by PROTECTION_ENGINE=grpc;
noop remains the default and the permanent fallback. Generated client in
go/pkg/stegoclient/stegov1. Verify/forensic HTTP routes mount under
requireModule("watermarking") — the gate already exists.

1.4 Deployment

Follows the proven sidecar pattern exactly: own build context deploy/stego-sidecar/,
python:3.12-slim, everything baked at build time (tesseract engine+models, pinned
PDFium, calibration data) — air-gapped-first, zero network at runtime. Stateless ⇒ scale
horizontally behind the compose network / k8s Service. gRPC on :9000 with health service;
compose healthcheck gates DMS readiness only when the watermarking module is licensed
(soft dependency otherwise, like embed-sidecar).

2. Architecture

Go DMS (unchanged)                         stego-sidecar (Python, stateless)
┌─────────────────────────┐   gRPC/mTLS   ┌──────────────────────────────────────────────┐
│ Deliverer ─► Protection │ ─────────────►│ server.py          (edge: proto ⇄ domain,     │
│ orchestrator            │               │                     deadlines, metrics, logs) │
│  PROTECTION_ENGINE=     │               │        │                                      │
│   noop | grpc           │               │  orchestrator.py   (mode → pipeline wiring;   │
└─────────────────────────┘               │        │            fail-closed rules)        │
                                          │  ┌─────┴──────────────────────────────────┐   │
      DURABLE ARTIFACTS                   │  │ pure-function core packages             │   │
      (survive any rewrite)               │  │                                         │   │
  ┌──────────────────────┐                │  │  payload/    bits ⇄ (id, MAC, RS)       │   │
  │ WIRE_SPEC.md         │◄─implements──  │  │  surgery/    PDF ⇄ text-run edit plan   │   │
  │ golden corpus        │◄─must pass───  │  │  channel_v/  bits ⇄ baseline deltas     │   │
  │ calibration/*.toml   │◄─loads──────   │  │  channel_k/  bits ⇄ kern deltas+starmap │   │
  │ proto/stego/v1       │◄─serves──────  │  │  normalize/  artifact → rectified image │   │
  └──────────────────────┘                │  │  verify/     channel results → verdict  │   │
                                          │  └─────────────────────────────────────────┘   │
                                          └──────────────────────────────────────────────┘

2.1 Layer rules (what makes a box independently rewritable)

1. Edge (server.py) — the only file that imports grpc. Converts proto ⇄ plain typed
   domain objects. No logic.
2. Orchestrator — the only place that knows what a mode is: wires
   Traceability→V, Authenticity→K, Hybrid→V+K; owns fail-closed decisions
   (surgery says "text-layer lost" + policy says invisible ⇒ refuse). ~200 lines, mostly
   declarative.
3. Core packages — each is a pure function over values: bytes/arrays in, values out.
   No I/O, no network, no globals, no clocks, no randomness (keys and calibration passed in
   as arguments). Each package has:
   - a one-page CONTRACT.md (its interface semantics in prose + types),
   - its own fixture set (inputs + expected outputs, committed),
   - no imports from sibling core packages — composition happens only in the
   orchestrator. surgery/ is the single shared dependency of channel_v/ and
   channel_k/, and they consume it through one narrow type: an EditPlan
   (list of {page, byte_offset, op, delta}) — channels plan edits, surgery applies
   them. Channels never touch PDF bytes; surgery never knows what a bit is.

This is the "easy re-coding" property in concrete terms: any box can be deleted and
rewritten against its CONTRACT.md + fixtures alone, in isolation, without reading the
rest of the engine. The two riskiest boxes (surgery/, channel_k/) are exactly the ones
most likely to need a rewrite, and they have the narrowest contracts.

2.2 Decode path (same discipline)

normalize/ turns any artifact (PDF/PNG/JPEG/photo/scan) into rectified grayscale images +
a capture-type tag, once; both channel decoders consume the same rectified input.
verify/ is a pure function (v_result, k_result, tier) → verdict implementing the
cross-channel agreement table — trivially rewritable, exhaustively table-tested.

3. Wire-format versioning (how rewrites never strand issued documents)

• The in-band payload starts with 4 version bits (per blueprint §7.3). v1 is what we
  ship. Any future change to geometry semantics, ECC, or MAC truncation bumps the version.
• Encoders emit exactly one version (the current). Decoders must decode every version
  ever shipped — enforced by keeping each version's golden corpus in CI forever. A rewrite
  that passes corpus/v1/ provably still attributes every document issued by the old code.
• The starmap blob is self-describing: {format_version, channel, geometry…} in a
  pydantic-validated envelope, encrypted by the DMS (KMS DEK) — the sidecar stays stateless.
• Calibration thresholds live in calibration/vN.toml (data, not constants), produced by a
  committed calibration harness from the real-capture corpus. Re-running calibration is a
  data release, not a code change.

4. WIRE_SPEC.md (to be authored with the engine — the recovery document)

One file, versioned, that pins with zero ambiguity:
payload layout (version 4b | issuance_id 32b | MAC tag T=32b | RS(n,k) parameters, byte order, tiling/majority-vote rule per channel); Channel V geometry (what "bit=1" means in pt,
sign convention, line-selection rule); Channel K geometry (one-sided +ε encoding — the
Starfield collapse fix — gap-selection rule, ratio measurement definition); MAC definition
(HMAC-SHA256(master_key[epoch], issuance_id) truncated to T bits); starmap schema.
Test: a competent engineer with only WIRE_SPEC.md + the golden corpus must be able to
write a from-scratch decoder that passes. That test is literally how a catastrophic rewrite
would be executed, so we rehearse it (§6.3).

5. Security architecture (unchanged from feasibility doc, made concrete)

• Keys: master secret per install in KMS/HSM, addressed by key_epoch; the sidecar receives
  the derived per-epoch HMAC key per-request over mTLS — it never holds long-term secrets
  (stateless = nothing to steal at rest).
• mTLS Go⇄sidecar (SPIFFE-style SANs); sidecar refuses plaintext.
• Fail-closed in the orchestrator + Go policy, not scattered: encode error/timeout ⇒
  Deliverer blocks the download (per-classification override).
• Blind tier (Tier=BLIND) runs V only, redacts payload; K's vault path is structurally
  unreachable from the public endpoint (separate RPC context, enforced in Go and sidecar).
• Post-encode self-check: every Encode is immediately Decoded (PDF path) in-process
  before returning; a failed self-check is an encode failure ⇒ fail-closed. This converts
  silent corruption into loud rejection — the cheapest consistency guarantee available.

6. The golden corpus — the real specification

Three tiers, committed (LFS for the heavy tier):

1. Synthetic round-trip (fast, every CI run): fixture PDFs spanning the font matrix
   (Type1/TrueType/WinAnsi, Type0/CID, subsetted, embedded CMaps, rotated CTMs, XObjects) ×
   3 modes × edge payloads. Assert: byte-valid PDF, visually-clean render diff, exact decode.
2. Capture corpus (nightly): real phone photos, WhatsApp/Telegram recompression,
   print→scan at multiple DPIs, crops. Assert: per-channel trace rates ≥ published SLA
   floors; regressions fail the build.
3. Cold-decoder rehearsal (per release): decode the entire corpus using only
   WIRE_SPEC.md-pinned parameters loaded from data files — proving the spec, not tribal
   knowledge in code, is sufficient to attribute. This is the "we messed up, rebuild it"
   drill, run routinely so it works when needed.

Every corpus entry stores {input, key material (test keys), expected channel results, expected verdict} as data. A rewrite in any language passes or fails objectively.

7. Repo layout + CI

stego/                          # new top-level (blueprint §4)
├── WIRE_SPEC.md                # durable — the recovery document
├── obscura_stego/
│   ├── server.py               # grpc edge (only file importing grpcio)
│   ├── orchestrator.py         # mode wiring + fail-closed
│   ├── payload/    CONTRACT.md + code + fixtures/
│   ├── surgery/    CONTRACT.md + code + fixtures/     # EditPlan applier
│   ├── channel_v/  CONTRACT.md + code + fixtures/
│   ├── channel_k/  CONTRACT.md + code + fixtures/
│   ├── normalize/  CONTRACT.md + code + fixtures/
│   └── verify/     CONTRACT.md + code + fixtures/
├── calibration/v1.toml         # durable data
├── corpus/                     # durable — golden corpus (LFS)
└── tests/                      # pytest + hypothesis + corpus harness
deploy/stego-sidecar/Dockerfile # follows embed/extract pattern; all deps baked
go/internal/protection/adapters/grpc.go   # the only new Go file of substance

CI gates (in order): ruff+mypy‑strict → unit+property tests → synthetic corpus →
license-allowlist scan (fails on any GPL/AGPL in the resolved dependency tree) →
determinism check → image build. Nightly: capture corpus + trace-rate report.

8. Build order (matches feasibility DR plan)

1. DR-0 — fix the starmap/mode inversion in proto + ports.go (starmap belongs to
   Authenticity/Hybrid, not Traceability).
2. Author WIRE_SPEC.md v1 + payload/ (pure bits — no PDF involved; hypothesis-test it
   to death first, it's the security keystone).
3. surgery/ against the font matrix (the compatibility keystone) + channel_v/.
4. Orthogonality spike (DR-1) with a minimal channel_k/; then the one-sided K fix +
   normalize/ + joint calibration (DR-2/DR-4) on the capture corpus.
5. verify/ + server.py + Go grpc.go adapter + verify routes under
   requireModule("watermarking") + self-check + compose wiring.
6. Hardening: pen-test the tier boundary, DPIA, publish measured trace-rate SLA.

9. Why this stack and not alternatives (recorded so it isn't relitigated)

• Rust or Go for the whole engine? Decode needs OpenCV+OCR+PDFium; encode needs
  token-level PDF surgery. In Go that's CGo-heavy (gocv, gosseract) and still wants
  PDFium; in Rust the PDF-surgery ecosystem (lopdf) is lower-level than pikepdf/qpdf with
  more sharp edges. Python is where the verified AGPL-free toolchain already exists, and the
  two existing sidecars establish the ops pattern. Crucially, §0/§4/§6 make this choice
  reversible: the wire spec + corpus are language-neutral, so a hot path can be ported to
  Rust later behind the same gRPC seam with zero DMS impact.
• In-process Go library instead of a sidecar? Rejected: couples the DMS release cycle to
  CV/PDF dependency churn, breaks the fail-closed isolation story, and the seam + deploy
  pattern for sidecars already exists.
• One service or two (encode/decode split)? One binary, two clearly separated paths.
  They share normalize-independent code minimally; splitting later is trivial because the
  proto already separates the RPCs. Don't pre-shard.