Gateway GPU-Health Coupling — implementation report

Branch: feat/gateway-gpu-health · Date: 2026-07-08
Spec: docs/superpowers/specs/2026-07-08-gpu-health-coupling-design.md

Status

Complete. Build, go vet, and go test ./internal/... all green; -race green
on every changed package (gpuhealth, jobs, registry, config, proxy, server, and
pool in isolation).

Commits (logical steps)

SHA	Scope
`540d821`	`feat(gpuhealth)`: DCGM parser + Monitor + `gpu_health` config surface + gpu_ids (dormant)
`b822bd3`	`feat(registry)`: injectable GPU guard drops faulted upstreams from rotation
`b80173f`	`feat(gateway)`: batch-shed admission gate at the 3 sites + server/main wiring

Each commit builds and tests independently (gpuhealth references nothing from
later commits; registry guard is standalone; the integration commit carries the
server.New/NewManager signature changes together with every updated call
site so no intermediate state is broken).

Files

New
- internal/gpuhealth/parser.go — minimal, quote-aware DCGM Prometheus parser.
- internal/gpuhealth/monitor.go — Monitor (poller, per-GPU state, snapshot
accessors, Guard, ShouldShed, Collectors).
- internal/gpuhealth/parser_test.go, internal/gpuhealth/monitor_test.go
- internal/jobs/shed_test.go — manager + façade admission shed tests.

Modified
- internal/config/config.go — GPUHealth block type on Config; GpuIDs []int on Upstream; defaults (active-only) + gpu_ids validation in Load().
- internal/registry/registry.go — gpuGuard field, SetGPUGuard, guard check
in PickEndpoint.
- internal/proxy/handler.go — Shedder interface, Deps.Shedder, batch shed
gate before Enqueue.
- internal/jobs/manager.go — Shedder interface (shared with façade), shed
field + NewManager param, shed gate in process.
- internal/jobs/facade.go — Facade.Shedder, shed gate in SyncForward.
- internal/server/server.go — shed field, New param, threaded into
handler/manager/façades; RegisterCollectors.
- cmd/gateway/main.go — construct Monitor iff dcgm_url set, register
collectors, set guard, go mon.Run(ctx); nil everywhere otherwise.
- Test call-site updates (nil defaults): internal/server/*_test.go (5 files),
internal/jobs/manager_test.go, internal/jobs/fanout_test.go.
- go.mod — added github.com/kylelemons/godebug (indirect; pulled by
prometheus/.../testutil used in a gpuhealth test). go.sum already carried
the hash, so it was untouched; readonly build passes.

Test counts (new)

30 new test functions:
- gpuhealth: 16 (parser 4, monitor 12 incl. the -race concurrent-Run test)
- config: 4 · registry: 1 · proxy: 4 · jobs (shed): 5

Design / judgment calls

Metrics registration via Collectors() + server.RegisterCollectors,
not gpuhealth.New(cfg, registry). The spec sketched gpuhealth.New(cfg, promRegFromServer), but the Monitor must exist before server.New (it is
the Shedder passed in), while the server's promReg only exists after
New — a construction-order cycle. Resolved by having New(cfg) build the
gauges unregistered and exposing Collectors(); main registers them on
the server registry post-construction (srv.RegisterCollectors(...)). Keeps
gpuhealth free of any server import and registry free of gpuhealth.
gpuhealth.New(cfg *config.Config) (dropped the registry param). Reads
cfg.GPUHealth and builds the upstream→gpu_ids map from cfg.Upstreams, so
Guard(id) (which only has the id) resolves without a config lookup.
Two structurally-identical Shedder interfaces (proxy + jobs), not one
shared type. proxy and jobs don't import each other and neither should
import gpuhealth for a type. server.New takes proxy.Shedder and assigns
it to the jobs.Shedder sites — legal interface-to-interface assignment
(identical method sets), and a nil interface stays nil across the assignment.
Typed-nil avoidance in main. var shed proxy.Shedder is assigned the
*gpuhealth.Monitor only inside the dcgm_url != "" branch, so when the
feature is dormant the interface is a true nil (never a typed-nil *Monitor),
and every Shedder != nil guard behaves correctly. Monitor methods are
also defensively nil-receiver-safe as belt-and-braces.
Manager shed action = fail the job with GPU_SATURATED. The spec text
("return 429 … emit audit error") is written for the two synchronous HTTP
sites. The manager is async and cannot return 429, so its equivalent of
"reject the batch admission + emit the audit error event + count the metric"
is finishWithPayload(StatusFailed, "GPU_SATURATED"), which sets the terminal
status and emits the audit event. Job audit events use the status vocabulary
completed/failed (not ok/error), so the shed event is
status=failed, error_code=GPU_SATURATED — consistent with the manager's
other refusals (e.g. EXTERNAL_UPSTREAM_FORBIDDEN). See Concerns.
ShouldShed increments gateway_gpu_shed_total internally. A true
verdict is always followed by an actual shed at all three call sites (there
is no path where it returns true and the request is still admitted), so the
counter is the honest count of shed admissions and the three sites stay a
single line each. The interface stays minimal (no separate RecordShed).
Class gate at both the call site and inside ShouldShed. Each site only
calls the shedder for class == ClassBatch ("skip the check entirely" for
interactive/system, and it keeps the counter honest), and ShouldShed also
returns false for non-batch — defense in depth.
Config defaults applied only when dcgm_url is set. A dormant config is
left all-zero (byte-identical), so a build/deploy without gpu_health is
provably unchanged. gpuhealth.New re-defaults defensively so a
directly-constructed Monitor is always sane regardless.
Saturation freshness / ring semantics. Saturated requires the latest
sample to be fresh (ts within staleness_s) AND the util ring to be full
(len == sustained_samples) AND every entry ≥ shed_util_pct. Interpreted
"the last N fresh utils" as: staleness is checked on the most-recent sample
(a long polling gap makes the GPU stale → fail open); the ring holds the last
N poll utils. A single missed poll between two polls does not reset the ring
(only staleness does) — acceptable for a minimal monitor given
staleness_s (60) ≫ poll_s (15).
Cache hits are served before the shed check (façade) and idempotent
replays before admission (proxy) — both consume no GPU, so serving them
under saturation is correct and matches the existing pre-admission ordering.

Concerns

Async shed + idempotency resubmit. When the manager sheds a batch job
(shed_batch on, sustained saturation), it becomes terminally failed with
GPU_SATURATED. A client resubmitting with the same deterministic
Idempotency-Key hits Store.Create dedup and gets the same failed job
back rather than a fresh attempt — so a shed job does not auto-retry the way a
synchronous 429 does. This only bites when shed_batch is explicitly enabled
(it ships off), and the client does see a clear GPU_SATURATED to back off
on. If job-path shedding is ever turned on, consider leaving the job queued for
ReclaimStale re-drive instead of failing it (would trade the audit-error
event for silent retry).
AzureDIFacade is intentionally not gated. The spec names exactly three
admission sites; the Azure DI passthrough's submit-side admission is out of
scope and left untouched. Flagged in case a batch shed there is later wanted.
Deploy stance unchanged from spec: ship couple_faults: true, shed_batch: false with the verified gpu_ids mapping (GPU 0 → qwen-35b; GPU 1
→ cleanup-3b / tei-embeddings / gpu-server / doc-classifier-svc). Fault
coupling fires only on real XID/thermal faults (none present); shed stays off
until saturation behavior is observed.

Verification done

GOFLAGS=-mod=mod go build ./... · go vet ./... · go test ./internal/... -count=1 (all ok) · -race on gpuhealth, jobs, registry, config, proxy,
server, and pool (isolation) — all ok. Dormancy proven by
TestLoadGPUHealthDormantByDefault + the unchanged existing suites (nil
Shedder/guard in every fixture).