Predictive harness — implementation report

Date: 2026-07-08 · Branch: feat/predictive-harness · Design: docs/PREDICTIVE-HARNESS.md

Folds time-series forecasting, anomaly detection, and an early-warning system into
the observatory (Go, querying TimescaleDB directly — no new service). Moves the
roadmap's three 0% items to working software on live AI-platform metrics.

Files

File	Purpose
internal/predict/forecast.go	Pure Forecast(values, horizon, season) Result — Holt-Winters → Holt → drift → naive → insufficient_data
internal/predict/anomaly.go	Pure ZScore(values, window, k) + IQR(values, k) → []Anomaly
internal/predict/warnings.go	Pure evidence-citing rules QueueSaturationWarning / ErrorRateAnomalyWarning / GPUSaturationWarning + Collect
internal/predict/{forecast,anomaly,warnings}_test.go	23 test functions (34 incl. subtests)
internal/store/metricseries.go	QueryMetricSeries(...) + pure reducers (reduceCallsSum/QueueWeighted/GPUUtilMax/One)
internal/store/metricseries_test.go	8 test functions (6 pure reducer + 2 DB integration)
internal/api/predict.go	Handlers for /api/forecast, /api/anomalies, /api/early-warnings; metric allowlist; errorRateByBucket
internal/api/predict_test.go	9 test functions
internal/api/server.go	Route mounting (3 lines, rolesSummarySeries gate)

Test totals: predict 23 funcs / 34 with subtests; store +8; api +9. 40 new test functions.

Method-selection thresholds (chosen)

Forecast(values, horizon, season), n = len(values), checked in order:

1. n == 0 → insufficient_data ("series is empty").
2. any non-finite (NaN/±Inf) → insufficient_data.
3. n < minPoints (2) → insufficient_data ("need at least 2 observations"). This is the honest floor — a single dot supports no line, trend, or variance.
4. constant series (all values equal) → naive: flat at the last value, zero band. Checked before seasonal so a flat line never gets a spurious cycle.
5. season >= 2 && n >= 2*season → holt_winters (additive trend + seasonal).
6. n >= minHolt (3) → holt (double-exponential, additive trend).
7. else (n == 2) → drift (line through the two endpoints).

Smoothing constants are fixed (not fitted) for determinism: Holt α=0.5, β=0.1;
Holt-Winters α=0.5, β=0.1, γ=0.3. On exact-linear data Holt reproduces the line
(zero residual → zero band); on a pure repeating pattern with no trend HW recovers
the cycle exactly (first-period mean equals the global mean, so the additive seeds
are already the true seasonal deviations).

Band: yhat ± 1.96·popStd(one-step residuals), constant across the horizon (per
the design's ±1.96·residStd; band-widening with horizon is noted below as a known
simplification). Population std (÷N) is deliberate: a single residual honestly yields
0 (no observed spread), which is exactly the zero-band behaviour on minimal series.
Holt-Winters excludes the seeded first-period warm-up steps from the residual set so
an arbitrary seed cannot inflate/deflate the band.

Anomaly detectors

• ZScore: rolling baseline of the window points strictly preceding index i;
  a point is scored only once a full trailing window exists, so the first window
  points are never flagged and a series no longer than the window yields nothing
  (short series → none). window < 2 or k <= 0 → none. A zero-spread baseline is
  skipped (no scale) rather than producing an infinite score. Score is the
  signed z (positive = upward spike) so downstream rules can filter by direction.
• IQR: global Tukey fences [Q1 − k·IQR, Q3 + k·IQR]; quartiles via linear
  interpolation matching Postgres percentile_cont. n < 4 or IQR == 0 → none.
  Score is signed distance past the nearest fence in IQR units.

Early-warning rules (each cites Evidence; no evidence ⇒ nil)

1. queue_saturation — queue-wait forecast crosses saturationMs within horizon.
   critical if already saturated at step 1, else warning. Cites eta_buckets + crossing yhat.
2. error_rate_spike — most recent upward error-rate anomaly within the last
   recentBuckets buckets, whose rate ≥ minRate (ignores blips on trivial volume).
3. gpu_saturation — GPU-util forecast reaching saturationPct while climbing
   (last forecast > first), so a steady-but-high plateau does not re-alarm.

Insufficient_data forecasts never produce a warning (no fabrication).

API

Route	Notes
GET /api/forecast?metric=&from=&to=&bucket=&horizon=&season=	{status,method,reason?,history,forecast[{ts,yhat,lo,hi}]}
GET /api/anomalies?metric=&...&method=zscore\|iqr&window=&k=	{metric,method,anomalies[{ts,value,score,kind}]}
GET /api/early-warnings?bucket=	{warnings,lookback_from,lookback_to,bucket} over a fixed 6h lookback

All three share rolesSummarySeries (executive/operator/auditor). Bucket parsed and
validated like handleSeries; store errors map to 500 QUERY_FAILED, bad input to
400 BAD_REQUEST.

Metric allowlist: {calls, avg_queue_ms, gpu_util, gpu_util:<id>}. Anything else —
especially percentiles (p95_total_ms, …) and non-recombinable aggregates
(avg_total_ms) — is a 400. A fleet percentile can never be recombined from group
percentiles, so it is never forecast.

QueryMetricSeries reductions (honesty-preserving)

• calls → sum of calls across engines per bucket (counts add).
• avg_queue_ms → calls-weighted mean of per-engine avg_queue_ms per bucket.
• gpu_util → fleet = max avg_util_pct across GPUs per bucket.
• gpu_util:<id> → a single GPU's avg_util_pct.

Reduction logic is factored into pure helpers unit-tested without a DB; a DB
integration test covers the SQL path end-to-end (weighted-mean = 1000/3, sum, gpu max).

Honesty / edge decisions

• insufficient_data over fabrication: empty/single/non-finite series refuse with a
  reason and nil points/forecast; the /api/forecast empty-window test asserts empty
  history and empty forecast (no fabricated zeros).
• No gap-filling: QueryMetricSeries returns only the buckets the DB produced;
  empty windows → empty slices, never fake zero buckets where a gauge is meant. The
  forecaster treats the returned series as evenly spaced (see limitations).
• clamp ≥ 0: every supported metric is non-negative, so Yhat/Lo/Hi are all
  clamped to max(0, ·) (the design specifies Lo≥0; clamping all three is monotonic so
  Lo ≤ Yhat ≤ Hi is preserved and a declining projection cannot go negative).
• error-rate status classification (interpreted — not spelled out in the design):
  isErrorStatus(s) = s != "" && s != "ok" && s != "success". Empty/unknown status is
  deliberately not counted as an error, to avoid inflating the rate on incomplete data.
• constant → naive, not Holt, to give naive a clear semantic home and avoid a
  spurious tiny trend from smoothing warm-up on a flat line.

Known limitations (documented, in scope to note)

• Constant-width prediction band (does not widen with horizon).
• Series assumed contiguous: absent (zero-activity) buckets are omitted, not
  zero-filled, so the forecaster sees a compacted grid — acceptable for the MVP's
  days-of-data / short-lookback use, called out in QueryMetricSeries docs.
• Early-warning saturation thresholds (DefaultQueueSaturationMs=5000,
  DefaultGPUSaturationPct=90, error minRate=0.05, recentBuckets=3) are
  illustrative operating heuristics passed in by the API layer, not hard SLOs.

Verification

GOFLAGS=-mod=mod go build ./..., go vet ./..., go test ./..., and
go test -race ./internal/predict/ all pass. Test DB reached via the throwaway
TimescaleDB container.

Pre-existing flake (not introduced here): running go test ./... with default
package parallelism intermittently hits SQLSTATE 40P01 deadlocks in
TestInsertEventDedups / TestMigrateConcurrentSafe — pre-existing store tests whose
paths this change does not touch — because multiple packages migrate/insert against the
one shared test DB at once. Serialized go test -p 1 ./... is reliably green; each of
the new predict/store/api test functions is stable in isolation.