Cutover plan: coordination: local → redis

Status: EXECUTED 2026-08-26. Both this cutover and the §3a gpu_ids fix are live and verified in production. Kept as the record of what was done and how to roll back.
Scope: the gateway (ahu-gpu-manager). No engine repo was touched.

Execution record (2026-08-26)

§3a gpu_ids — done first. Investigation refined the finding: doc-classifier-svc: gpu_ids:[1] is CORRECT (the classifier genuinely runs on H100 GPU 1 — DeviceIDs ["1"], sole process, 1541 MiB). The other four were wrong, and three of them (cleanup-3b, tei-embeddings, gpu-server) pointed at index [1] — the classifier's own GPU — so a classifier fault would have dropped three unrelated B200-backed upstreams from rotation. gpu_ids removed from those four; doc-classifier-svc keeps its own. Verified after: qwen-35b 200 in 50 ms, cleanup-3b 200 in 34 ms.

Cutover — all 7 verification steps passed:

Rollback was rehearsed, not just documented — flipping back to local and forward to redis both worked in ~20 s, and the job record survived the round trip intact.

One cosmetic artefact: pausing Redis for V7 briefly marked ahu-platform-redis unhealthy on the board. It self-recovered (failing streak 0) within one healthcheck interval; fleet back to 49/49 healthy.

0. Verdict up front

The change itself is two lines of selection and a config flag. The code it switches on is the strongest part of the codebase, and it now passes against a real Redis, not just an emulator.

But be clear about what it buys, because the honest answer is "insurance, not throughput":

Exposure being insured: ~20 async jobs/day (gpu-server 12.9/day, azure-di-onprem 7.3/day). Longest observed job 300 s. So a restart at a bad moment costs at most a handful of jobs, each already recoverable by client resubmit — which the OCR engine implements (gpu-job-client.ts:303).

Do it because it removes a class of failure and unlocks HA. Don't do it expecting a performance change.

1. What actually changes

cmd/gateway/main.go, two selections:

// line 45-49
if cfg.Coordination == "redis" { pools[u.ID] = pool.NewRedisPool(rdb, u.ID, u.Slots) }
else                           { pools[u.ID] = pool.NewLocal(u.Slots) }

// line 95-100
if cfg.Coordination == "redis" { store = jobs.NewRedis(rdb) }
else                           { store = jobs.NewLocal() }

Everything else is already Redis-backed today, under coordination: local: idempotency (idem.NewRedis), the audit stream (audit.NewEmitter), the response cache (jobs.RedisCache), and refusal memory (jobs.RedisNegativeCache). Redis is already a hard dependency — this cutover does not add one, it extends an existing one.

No new config, no migration, no instance ID. Lease identity is a random per-lease ID in a ZSET keyed by expiry (gwslots:<upstream>), so a second instance joins with zero configuration.

Design worth knowing before you operate it

redisPool is two-stage. Stage 1 is an ordinary local pool (preserving class priority + FIFO); stage 2 acquires a global lease. That is why fail-open is cheap: if Redis is unreachable, stage 2 is skipped and stage 1 alone admits. Job records carry a 24 h TTL (jobs and idempotency keys alike) — more predictable than LocalStore, which keeps them until the process dies.

2. The main risk is already retired

Both Redis implementations were covered only by miniredis — an in-process Go reimplementation. The store leans on EVAL/Lua, RPOPLPUSH and WATCH/MULTI; the pool leans on Lua-scripted lease acquire/release and key expiry. An emulator agreeing with itself does not prove the real server agrees.

So I added TEST_REDIS_ADDR to both suites and ran them against real Redis 7 (appendonly):

internal/jobs   19/19 PASS   (dedup, atomic claim, stale reclaim, restart durability, fan-out)
internal/pool   10/10 PASS   (incl. LeaseExpiryFreesSlotAfterCrash,
                              TwoInstancesShareCap, FailOpenWhenRedisDown)

TestFailOpenWhenRedisDown needed real work rather than a skip — it is the property the whole decision rests on. Real-Redis mode now points the client at a closed port, which is how a dead server actually presents (connection refused), a more faithful simulation than miniredis's injected errors.

Committed on master (test/redis-path-against-real-redis). Default is still miniredis, so go test ./... needs no services.

3. Do these two FIRST — they are cheaper and one is a live landmine

3a. gpu_ids still name H100 GPUs — fix before enabling shedding

Five upstreams declare GPU indices from before the B200 migration:

qwen-35b           gpu_ids: [0]     # actually runs on 6 endpoints across os-b200-02
cleanup-3b         gpu_ids: [1]
tei-embeddings     gpu_ids: [1]
gpu-server         gpu_ids: [1]
doc-classifier-svc gpu_ids: [1]

gpu_health.dcgm_url points at 192.168.83.20:9400 — the H100. Monitor.state is map[int]*gpuState keyed by bare GPU index with no host dimension, so gpu_ids: [0] means "GPU 0 on whichever DCGM is configured".

couple_faults: true is live, and Guard() is wired (main.go → reg.SetGPUGuard). So if H100 GPU 0 ever faults, the guard drops all six B200 qwen-35b endpoints from rotation — a total outage of the main model caused by an unrelated, near-idle GPU.

Probability today is low: couple_xid: false, so isFault reduces to TempC >= 90, and an idle H100 will not hit that. But the failure mode is total and the fix is a config edit. Decide the intent — either point dcgm_url at the B200 node and renumber gpu_ids, or drop gpu_ids from upstreams whose GPUs that exporter cannot see.

3b. The audit stream is unbounded

ahu.ai.audit holds 122,293 entries / ~1.15 GB, and there is no XTRIM/MAXLEN anywhere in internal/audit. Redis runs with maxmemory 0 and maxmemory-policy noeviction, in a container with no memory limit. 102 GB is free on the host, so this is a slow burn, not an emergency — but noeviction means that when it does fill, writes start failing rather than old data being dropped. Worth a MAXLEN ~ trim before leaning on Redis harder.

4. Cutover

Low-risk, reversible in under a minute. ~20 jobs/day means almost any window is quiet; still, avoid running it while an OCR batch is in flight.

Pre-flight

1. docker exec ahu-platform-redis redis-cli PING → PONG; confirm appendonly yes (it is).
2. Confirm no jobs in flight — the audit shows no gpu-server/azure-di-onprem calls in the last 5 minutes.
3. cp deploy/gateway.yaml deploy/gateway.yaml.bak-pre-redis-coordination
4. Note the current image tag for rollback.

Change — one line in deploy/gateway.yaml:

coordination: redis        # was: local

redis_url is already correct (redis://ahu-platform-redis:6379/0).

Apply

cd /home/efran/ahu-gpu-manager/deploy && docker compose up -d gateway

Confirm it took — the startup line names the mode:

gpu-server compat façade enabled (coordination=redis)

5. Verification (in order; stop and roll back on any failure)

1. Health — curl -s localhost:8200/healthz → 200.
2. Sync path unaffected — one cleanup-3b chat call → 200.
3. Admission still queues correctly — rerun the saturation test: 24 concurrent batch on cleanup-3b (batch_max: 8) must yield 8 admitted / 16 queued, with interactive fired into the backlog staying < 150 ms and never queued. This is the same test that proved the local pool on 2026-08-26 (avg_queue 257 ms, max 500 ms, 0 errors); the numbers should be close, plus a small Redis round-trip.
4. Leases appear — redis-cli ZCARD gwslots:cleanup-3b should be > 0 during load and drain to 0 after.
5. Job path end-to-end — run one real OCR extraction; it must complete normally.
6. The actual point — submit a job, restart the gateway mid-flight, confirm the job still resolves instead of returning JOB_UNKNOWN. This is the only step that proves the cutover did anything; skipping it means shipping untested insurance.
7. Fail-open — optional but valuable: docker pause ahu-platform-redis, confirm sync calls still succeed (degraded to local admission), then docker unpause. Verified in test; worth seeing once in production.

6. Rollback

cp deploy/gateway.yaml.bak-pre-redis-coordination deploy/gateway.yaml
docker compose up -d gateway     # log must read coordination=local

Cost: jobs created in Redis are orphaned and expire on their own 24 h TTL; anything in flight at the moment of rollback is lost — the same exposure as today's normal restart. No cleanup required. Leftover gwslots:* keys are harmless and expire by score.

7. What this does NOT give you

It is not HA yet. There is still one gateway container. coordination: redis makes shared state possible; actual HA is a separate phase:

• Phase A (this plan): durable jobs + shared slot accounting. Single instance. Removes restart data-loss.
• Phase B (later): a second gateway instance behind a load balancer. Needs Phase A first, but no further gateway config — leases are self-identifying, and TestTwoInstancesShareCap covers exactly this. The open work in Phase B is the LB and the edge nginx, not the gateway.

Also unchanged: throughput, latency, slot counts, and every shedding decision — shed_batch stays false regardless.

8. Recommended sequence

1. Now / before the demo: nothing. The queue is not the demo risk (7 days: 5 of 92,652 interactive calls ever queued; zero QUEUE_TIMEOUT in 30 days).
2. After the demo, cheapest first: fix gpu_ids/dcgm_url (§3a) — a live landmine and a config-only fix.
3. Then: trim the audit stream (§3b).
4. Then: this cutover, with §5.6 actually performed.
5. Later, if HA is wanted: Phase B.