Obscura Stego — WIRE SPEC v1

This is the durable recovery document. Code is disposable; this file is not.
A competent engineer with only this file, the calibration data, and the golden corpus
must be able to write a from-scratch decoder that attributes every document ever
issued under wire version 1. Any change to the in-band message (§1) bumps the in-band
version and adds a new spec section — v1 message semantics are frozen forever once the
first production document is issued.

Geometry variants vs. message version. The traceability mark has three embedding
geometries: Channel V vertical baseline (§2.1: v1 reference-line, v2 QIM) and Channel W
word-gap (§2.1a — the only geometry a screenshot of the document can decode). All carry the
same in-band message (version 0001); the encoder chooses one per document and decoders
try all geometries and let the MAC select the true one, so no issued document is ever
stranded. Adding a geometry therefore does NOT bump the in-band version — only changing the
message fields does. (Channels W and K additionally wrap the 64-bit message in an RS(15, 8)
codeword for error correction; the message semantics in §1 are unchanged.)

1. In-band message (identical across all channels)

message = version(4 bits) | issuance_seq(32 bits) | mac(24 bits) | pad(4 bits) = 64 bits (8 bytes)

Bit order everywhere: MSB-first within each byte, bytes in order. "Bit i" of an
n-byte block = byte i//8, bit 7-(i%8).

MAC width rationale (recorded, do not relitigate): T=24 chosen over 32 because
Channel V is line-count-limited (~1 bit per inter-line gap; a dense page ≈ 30–50 gaps)
and both channels must carry the same message. Random-decode false-accept = 2⁻²⁴ ≈
6·10⁻⁸ per phase test; with V's 64-phase search the effective bound is 64·2⁻²⁴ ≈ 4·10⁻⁶
per document, and cross-channel agreement (Hybrid) squares it. This clears the forensic
bar that CRC-16 (2⁻¹⁶, unkeyed, forgeable) did not.

Key derivation: K_epoch = HMAC-SHA256(master_key, "obscura-stego-epoch" ‖ epoch_be32).
The master key lives in the DMS/KMS side; the sidecar receives K_epoch per-request and
holds no long-term secret. Old epochs remain verifiable forever.

2. Channel framing

2.1 Channel V — "traceability" (vertical baseline)

Channel V has two geometries. Both carry the same 64-bit message, no RS — protection
is repetition: the codeword is tiled (bit slot_index mod 64) across carrier lines in
reading order and the decoder majority-votes each bit position, with a 64-phase search
for crops (unique MAC-passing phase wins; ≥2 ⇒ INCONCLUSIVE). The encoder picks the geometry
per document (see Scheme selection); decoders try both and let the MAC disambiguate.

v1 — reference-line (differential, blind-decodable, needs regular spacing). Lines group
into blocks — maximal runs joined by usable inter-line gaps (§3.1). Within a block, odd
positions (1,3,5,…) are carriers, even positions are unshifted references; a carrier
needs a reference on both sides. Carrier L shifts ±δ along its baseline (bit 1 = down
+δ, bit 0 = up −δ), δ = 0.6 pt (clamped to ≤ 0.15 × median pitch). Decode measures one
ink-centroid baseline per line and reads residual = carrier − ½(left_ref + right_ref);
residual > 0 ⇒ bit 1, |residual| ≪ δ ⇒ erasure. Because it is differential (each bit
is a sign relative to local neighbours), v1 is scale/offset-invariant and is the only V
geometry a blind pixel decoder can read (screenshots/photos of a regular-layout page). It
fails on irregular spacing (headings, varied leading, proportional/justified text): the
two references no longer bracket the carrier, so the interpolation residual swamps δ.

v2 — QIM / dither modulation (absolute, layout-independent, exact-decode only). Every
axis-aligned line with ≥3 glyphs is a carrier (no reference/carrier split, no regular-spacing
requirement — this is what makes v2 work on arbitrary business layouts). Bit b is embedded
by snapping the line's absolute baseline onto a lattice: bit-0 points at multiples of the
step, bit-1 points offset by half a step; max displacement step/2. Decode reads
bit = round(baseline / (step/2)) mod 2 — a per-line absolute parity, no neighbours needed.
- The step is a FIXED constant (1.2 pt ⇒ 0.6 pt max shift, 0.3 pt decode margin), not
a fraction of pitch. Because the read is absolute over large baseline coordinates (~700 pt),
the encode and decode lattices must be byte-identical; a pitch-derived step is perturbed by
the encoding itself and a mere ~0.5% mismatch flips high-y parities. This absolute nature is
also why v2 is not blind-decodable — pixel captures have unknown scale/offset that the
same sensitivity punishes (a blind ±half-step is absorbed as a global parity flip, but the
scale is not). v2 is the reliable exact/PDF path (self-check, forwarded-PDF forensics).
- Collision avoidance: if a snap would bring a baseline within the line-clustering
tolerance of the line above (which would merge them and frame-shift the decode), the
encoder drops to the highest same-parity lattice point that clears the separation
(tol + 0.35 pt). Parity/bit is preserved, so the encoded PDF keeps the identical line
set and the decoder re-derives the same carrier order. Carrier selection uses only
shift-invariant properties (glyph count, axis-aligned), never gaps, so encode and decode
agree exactly.

• Scheme selection (encoder): try v1; keep it only if its own exact self-check passes
  (i.e. the document is regular enough to be blind-decodable). Otherwise emit v2. Every text
  document with ≥64 carrier lines thus gets a mark whose decode_pdf self-check passes;
  applied=False (too little text) is the only non-marking outcome.
• Minimum capacity: ≥ 64 carrier lines. v1 uses half the lines (≈128+ body lines); v2 uses
  every line (≈64+). Decoders MUST NOT report CONFIRMED without a full MAC-checked codeword.
• Calibration: v1 δ = 0.6 pt decodes blind at 100% across 120/150/200/300 DPI on the
  clean-render regular corpus (centroid noise ≈ 0.3 px). v2 decodes exactly at 100% on the
  irregular/proportional/justified/real-document corpus for all payloads, any layout.
• Known limit: blind pixel decode of an irregular layout is not achievable with a
  vertical blind mark (v2 is absolute; v1 needs regularity). The robust path for screenshot
  forensics is the horizontal word-gap carrier, Channel W (§2.1a), which the encoder prefers
  for the traceability mark whenever the document can carry it; Channel V remains the fallback
  (and the sole carrier for the forwarded-PDF leak vector, fully covered by v2 exact decode).

2.1a Channel W — "traceability" (word-gap; the screenshot-blind carrier)

Channel W carries the same 64-bit traceability message as Channel V, but along the
horizontal axis (inter-word spacing) and with a signal that is multi-pixel, so it
survives a screenshot of a few lines — the one thing the sub-pixel vertical geometries cannot do.
It is the encoder's preferred traceability geometry; Channel V is the fallback for documents W
cannot carry (§2.3).

• Codeword = RS(15, 8) over GF(2⁸) = 120 bits (the same build_codeword_k codeword as
  Channel K — 8 message bytes + 7 parity bytes), then whitened (XOR a fixed SHA-256 keystream)
  so short zero-runs cannot masquerade as an unmarked line. Corrects ≤3 byte errors / ≤7 erasures.
• Carrier lines & words are detected from PIXELS (a 150-DPI render), by both encode and decode,
  with a morphological-close word segmenter (bridge inter-glyph gaps below 0.10 × font_size,
  never the larger inter-word spaces). This makes the word segmentation stable under the very
  gap edits the channel applies — the property that makes blind decode work on real documents; a
  hard-threshold segmenter flips word boundaries when a word shifts sub-pixel and the stride tiling
  cascades. A body-height line with ≥ 4 words is a carrier.
• Geometry (differential, justification-preserving): for a carrier line with n gaps, take the
  n codeword bits assigned to it, map each bit to a sign s_j ∈ {+1,−1}, and let m = mean(s_j).
  Gap j is nudged by ε_j = δ · (s_j − m), δ = 1.4 pt. The per-line nudges sum to ≈ 0, so the
  line stays justified / keeps its width; the mark reads as a differential against the line's
  own mean gap, hence scale- and offset-invariant under a screenshot. The nudge is realised by
  surgery.AdjustGap — a signed TJ kern inserted at the space sitting in that gap (works even
  when spacing is done with positioning numbers rather than space glyphs, as in real embedded-font
  PDFs). δ = 1.4 pt is mildly visible (looks like imperfect justification); smaller δ trades
  robustness, larger δ risks merging adjacent words past the segmenter threshold.
• Codeword→line mapping: line i starts at codeword position (STRIDE · i) mod 120,
  STRIDE = 7 (coprime with 120 ⇒ even coverage; a non-coprime stride starves positions). A
  mis-read line therefore corrupts a spread of positions, not a contiguous run RS cannot fix.
• Decode (decode_blind from a screenshot, decode_pdf via an internal render for the
  self-check / a forwarded PDF): per carrier line, measure the gaps, threshold each against the
  line mean (|dev| < 0.30 × spread ⇒ erasure, else sign ⇒ bit), then a line-offset (A)
  search over the 120 phases, majority-vote each position, RS-erasure decode, and accept on
  MAC. A unique MAC-passing offset wins; none ⇒ INCONCLUSIVE.
• Global-then-per-image passes. The decoder first runs one offset search over all carrier
  lines of the capture concatenated, then — if that misses — over each image/page on its own.
  A page of non-body structure (tables, forms, cover/signature blocks) injects spurious carrier
  lines that desync the global line→position mapping for every line after them (the offset search
  corrects a single global shift, not a mid-stream one). Any one clean body page holds a full
  codeword, so per-image recovery makes a real 20-page document (and its encode self-check)
  decode even when only some pages are clean prose. The MAC gate makes multiple windows safe.
• Multi-scale retry (decode_blind). On a miss the decoder re-runs at a couple of resamples
  (×0.8, ×1.25) — a capture arrives at an unknown zoom, and a slight resample shifts antialiasing
  enough to stabilise borderline word segmentation. decode_pdf gets the same effect from its
  multi-DPI render list. MAC-gated; only the first (native-scale) pass runs on the common success.
• Minimum capacity: ≥ 120 carrier gaps (one RS codeword). In practice ≈ 16–24 lines of
  justified/spaced body text (≈ 2 dense pages); below that, encode returns applied=False and the
  orchestrator falls back to Channel V. A single short paragraph is below the information floor
  for any in-text carrier.
• Supported document class: documents with clean, well-separated inter-word spacing and stable
  segmentation — real embedded-font business/government PDFs and monospaced/left-aligned renders
  (test corpus). Validated on a real 20-page government PDF (subset NimbusSans fonts): a single-page
  clean screenshot recovers the full 64-bit payload ≈ 95–100%, a two-page shot 100%; ≈ 70%
  through an aggressive rescale+JPEG channel. Not supported: reportlab-platypus tight
  justification and mixed-layout proportional docs (fragile pixel↔geometry space mapping) — these
  fall back to Channel V. The encoder is self-verifying: W is kept only if its own render
  self-check decodes the seq back, so an unsupported document can never ship a bad W mark.
• Space-glyph detection (why it works on real PDFs). Word gaps are located at the space glyph,
  identified from the font's is_space flag. Real PDFs almost always embed subsetted fonts
  (BAAAAA+Foo) that renumber glyphs from 1, so the space is rarely at 0x20 — its true code
  (e.g. 1/4/7/12) is read from the font's ToUnicode CMap (codes mapping to U+0020), for simple
  and composite fonts. Missing this makes is_space wrong and starves Channel K of carriers.
• Calibration: δ = 1.4 pt, STRIDE = 7, deadband 0.30, close-fraction 0.10, encode/detect DPI
  150, decode_blind retry scales ×0.8/×1.25. Blind decode of the monospaced clean-render corpus =
  100% at the pristine/clean operating points; heavier operating points (rescale + JPEG) are
  measured in stego/research/.

2.2 Channel K — "authenticity" (horizontal kerning, non-blind, starmap-bound)

• Codeword = RS(15, 8) over GF(2⁸): the 8 message bytes + 7 parity bytes
  (generator/primitive polynomial: reedsolo defaults — prim 0x11d, generator 2,
  fcr=0), = 120 bits. Corrects ≤3 unknown byte errors or ≤7 erasures with known
  positions. The codeword is written with 2× block repetition (codeword A then
  codeword B = same bits) when capacity allows; the decoder may combine both blocks.
• Geometry (one-sided — the Starfield collapse fix): for carrier gap j
  (an inter-word gap selected per §3.2): bit 1 ⇒ widen by +ε, bit 0 ⇒ unchanged.
  ε default 1.5 pt, never negative, overridable per-encode. No gap is ever narrowed,
  so no gap can collapse and break segmentation.
• Decode measurement (non-blind): requires the starmap. For each carrier gap the
  starmap records its line, ordinal gap index within the line, and the nominal
  (pre-encode) gap-to-line-height ratio. bit = 1 if measured ratio exceeds nominal by
  more than the calibration threshold, else 0. Erasures (unmeasurable gaps) are passed
  to RS with positions.
• Sync: provided by the starmap (it names every carrier gap) — no phase search.
• Decode paths: (a) exact — when the artifact is a PDF, re-analyze it and read
  each carrier gap's advance directly; this is the reliable K verification path. (b)
  raster — for camera captures, locate each line band and measure the word-gap width
  by column projection. The raster path measures the widening signal (~3% BER on a clean
  render) but reliable camera/print-scan classification is a calibration item against
  the real-capture corpus (feasibility §3.3, DR-2), not a v1 guarantee.
• 1:N identification (Starfield "score all starmaps"). Channel K is non-blind, but the
  investigator does not need to know which issuance a leak came from. When the candidate
  issuance is unknown, the DMS supplies a batch of recent vault starmaps (candidate_starmaps,
  most-recent first, bounded) alongside any explicit candidate; the decoder parses the artifact
  geometry once and tries each starmap, and the per-issuance MAC identifies the true copy
  (a wrong starmap can never false-attribute — it either fails RS or fails the MAC). An
  authenticity-only document therefore self-identifies with no seq hint. Two starmaps of the same
  document layout share carrier positions, so the recovered seq comes from the decoded payload
  (MAC-bound), not from which starmap was used — attribution is cryptographic, not positional.
  This mirrors Starfield's decoder (build a starmap index, recall/score candidates, verify by RS +
  MAC); a per-artifact fingerprint recall step to shortlist before scoring is the scale-out
  follow-up (not needed at demo issuance counts). An explicit candidate_seq is only an
  optimisation/tie-break — it points the search at one copy first, never changes a correct verdict.

2.3 Modes

Every traceability encode is verified inline (the chosen channel must decode the seq back from
its own output) before the document is accepted, and Channel K is likewise verified when it was
applied. The self-check asserts what was actually embedded: traceability must round-trip, and K
must round-trip only if K found carriers. A stale pre-existing mark can therefore never shadow the
freshly embedded one; a failed self-check is an encode failure (the DMS fails closed for that
channel). Rationale (do not relitigate): requiring K to succeed even when it has no carriers made
the DMS's fail-open download policy serve K-incapable documents unmarked (untraceable) — the
opposite of the forensic goal. Traceability is the primary mark; K is additive.

3. Carrier selection (must be reproducible from the PDF alone)

3.1 V: usable inter-line gaps

Within each page, text lines are the surgery layer's line clusters (text runs grouped by
baseline Y within a tolerance of 20% of median pitch, ordered top→bottom). A gap is
usable iff both lines belong to the same text block (gap ≤ 1.8× median pitch), both
lines have ≥ 3 glyphs, and neither line is rotated (CTM axis-aligned, uniform scale).
Gaps are ordered top→bottom, pages in order.

3.2 K: carrier inter-word gaps

Within each line with ≥ 4 inter-word gaps (a "gap" = a U+0020-coded advance between two
non-space runs in the same BT block and same font size), carriers are every other
gap starting from the second (indices 1, 3, 5… zero-based), capped at 8 carriers per
line. Lines are consumed top→bottom, pages in order, one bit per carrier, until the
codeword (×2 blocks) is exhausted. All selections are recorded in the starmap — the
decoder trusts the starmap, so §3.2 constrains only the encoder.

4. Starmap blob (vault format, versioned)

Canonical JSON (UTF-8, sorted keys), then encrypted by the DMS (KMS DEK) — the
sidecar emits/consumes plaintext JSON and never stores it.

{
  "format_version": 1,
  "channel": "k",
  "epsilon_pt": 1.5,
  "codeword_bits": 120,
  "blocks": 2,
  "carriers": [
    {"page": 0, "line": 4, "gap": 1, "bit": 0, "nominal_ratio": 0.31, "line_h_pt": 11.2}
  ]
}

nominal_ratio = pre-encode gap width ÷ line height (both in text-space points).
bit = absolute bit index into the concatenated (block-repeated) codeword stream.
Unknown extra keys MUST be ignored by decoders (forward compatibility); format_version
governs breaking changes.

5. Verdict semantics (decode output contract)

• CONFIRMED — a full codeword passed MAC+version+pad on ≥1 channel with confidence
  ≥ τ_confirm (calibration data), or both channels decoded the same issuance_seq with
  MAC ok (cross-channel agreement).
• LIKELY — MAC-passing decode below τ_confirm, or partial-capacity V match against a
  provided candidate list.
• INCONCLUSIVE — everything else, including multi-phase MAC ambiguity.
• Blind tier returns only verdict + boolean "traced"; issuance_seq, channel detail
  and confidence are redacted. K never runs in the blind tier.

6. Calibration data (versioned, shipped beside the code)

calibration/v1.toml pins: V decode thresholds (min pitch samples, τ_confirm/τ_likely),
K ratio threshold vs ε and DPI, normalization parameters. Produced by the calibration
harness from the capture corpus; a data release, not a code change. Decoders load
thresholds ONLY from this file.

7. Reference test vector (frozen)

master_key = 32×0x0b, epoch = 1
K_epoch    = HMAC-SHA256(master, "obscura-stego-epoch" || 00000001)
           = a44ecd6f1574e6ef2ca26289ba08b47ca295945a83dd8904dc0417beab39fab1
seq        = 305419896 (0x12345678)
mac24      = 0x7ed0f9
message    = 1123456787ed0f90
K codeword = 1123456787ed0f90f409098c6b48be   (RS(15,8) parity appended)

The authoritative copy is corpus/v1/test_vectors.json, generated once by the
reference implementation and frozen. CI asserts the implementation reproduces it
byte-for-byte (tests/test_wire_vectors.py).