Task 8 report — fan-out job groups (children + max_parallel + merge)

Commit: 438cdba — feat(p2): fan-out job groups with max_parallel scheduling + merge
Branch: feat/gateway-p2-jobs-ocr (on top of f767251)

Fan-out request encoding

POST /jobs with a JSON body carrying a top-level fanout object:

{"fanout":{"max_parallel":2,"children":[
  {"model":"paddleocr","doc_hash":"d0","pages":3,"payload":"<base64>","content_type":"application/json"},
  ...
]}}

A group is distinguished from a single job purely by the presence of the
fanout key (parseFanout probes the body with json.Unmarshal). A raw
document, or JSON without fanout, falls through to the unchanged single-job
path (TestFanoutSingleJobUnaffected + all existing api tests stay green).
Child payloads are inline base64 (documents are binary); the whole group is
validated + decoded before anything is created, so an invalid group fails
atomically (no orphan parent).

How max_parallel is enforced (coordinator design)

Children are created as StatusHeld (not on any ready list, never claimed by a
worker). A per-Manager coordinator goroutine (coordinateLoop, default 100 ms
cadence) sweeps active parents each tick and:

1. counts in-flight children (queued-for-slot + processing),
2. releases held children to queued (Store.ReleaseChild) until
   max_parallel are in-flight — releasing is done ONLY by this single
   goroutine, and nothing else raises the in-flight count between the snapshot
   and the release, so concurrent runs can never exceed the cap (a completing
   child only lowers the count),
3. persists the parent's derived status/progress for pollers, and
4. finalizes the parent exactly once when all children are terminal.

The cap is enforced entirely in the coordinator, NOT by shrinking the pool: the
concurrency-cap test gives the upstream 4 pool slots but max_parallel=2, and
the released children still go through the same pool admission + adapters as
single jobs. max_parallel <= 0 means "no explicit cap" (bounded only by the
pool).

Parent-status derivation (deriveParentStatus, shared by coordinator + Poll)

• queued until a child actually starts (processing or terminal),
• processing with progress = terminal_children / total while running,
• completed when ALL children succeed,
• failed (collect-all) once all children are terminal if ANY failed — every
  child's result is retained, no sibling is cancelled.

GET /jobs/{parent} derives status live from the children (fresh regardless of
the coordinator tick) and returns {status, progress, children:[{id,status, doc_hash, stage/progress|result|error_code}]}. GET /jobs/{child} uses the
normal single-job poll.

Children idempotency-key derivation

childIdemKey(parentKey, i) = parentKey + "#" + i. The parent is idempotent on
its own Idempotency-Key: a resubmit returns the existing parent and skips child
creation; even if it did not, each deterministic child key dedups in the store.
TestFanoutIdempotentResubmit asserts a stable child count across resubmits.

Store / audit changes

• job.go: StatusHeld, KindParent, Job.Kind, Job.MaxParallel.
• Store interface + Local + Redis: ListActiveParents, ReleaseChild,
  FinalizeParent. Create now preserves a caller-set status (held); Redis
  only LPUSHes runnable queued jobs and indexes parents in a jobs:parents set;
  ClaimNext skips parent records. FinalizeParent/ReleaseChild are atomic
  Lua on Redis so parent finalize + audit fire once even across HA replicas.
• manager.go: audit events now populate job_id and parent_job_id; parent
  event (operation=job) emitted by the coordinator at parent-terminal.

On-prem enforcement, idempotency, drain, slot-release and panic-recover all
apply per-child (children inherit the single-job path); the coordinator loop
stops on drain/cancel like the reclaim loop.

Verification

• go build ./... — OK
• go vet ./... — clean
• go test -race ./internal/jobs/ ./internal/server/ — OK (jobs: 66 test cases
  pass incl. Redis-gated ones which ran against a reachable Redis; server: 5)
• go test ./... — all packages OK

Concurrency-cap test: TestFanoutMaxParallelNeverExceedsCap (4 children,
max_parallel=2, 4 pool slots free, asserts peak concurrent Run <= 2, parent
completed, 4-child summary, progress=1, 1 job + 4 job_child audit events).
Also: TestFanoutOneChildFailsCollectAll, TestFanoutIdempotentResubmit,
TestFanoutSingleJobUnaffected.

Concerns

• Fan-out detection parses any JSON body to probe for fanout; a single-job
  JSON payload that itself contained a top-level fanout object would be
  misread as a group. Not a concern for OCR/document payloads; documented.
• The coordinator persists non-terminal parent status each tick via
  UpdateStatus (Redis WATCH/MULTI) — small per-parent overhead; Poll derives
  live so this is observability-only, not correctness-critical.
• Parent finalize is once-only across replicas via FinalizeParent; the
  coordinator itself runs per replica (each releases/merges), which is safe
  because release + finalize are atomic store ops, but multiple replicas do
  redundant sweep work. Acceptable for the current HA pair.