Watchman rollouts and deployment

A rollout gate should be reproducible from the saved evidence. Watchman offers offline comparison for saturation ladders and canaries as well as report comparison for hardware and telemetry regressions.

Compare like with like

Benchmark comparison requires compatible evidence. Saved saturation reports must be complete, ordered, reconstructable, and compatible in workload identity, model, route, stream mode, full schedule, and source policy before quantitative gates can produce a pass or regression outcome.

12345678watchman benchmark compare baseline-benchmark.json candidate-benchmark.json \
  --max-p95-e2e-regression-percent 10 \
  --min-successful-rps-ratio 0.95 \
  --fail-on-regression

watchman rollout baseline-canary.json candidate-canary.json \
  --max-p95-ttft-regression-percent 10 \
  --min-output-tps-ratio 0.9 --fail-on-regression

Missing or incompatible evidence is not a pass. It stays not_evaluable, which keeps a weak comparison from becoming a confident deployment decision.

Run it as an observer

watchman serve exposes Prometheus, health, report, and history surfaces. It can run on a short interval, listen on loopback, probe a runtime, and write NDJSON history.

12345watchman serve --interval 5s \
  --listen 127.0.0.1:9400 \
  --api-token-file /run/secrets/watchman-api-token \
  --probe http://vllm:8000 --allow-insecure-http \
  --history /var/lib/gpu-watchman/history.ndjson

The service exposes /livez, /metrics, /healthz, and /api/v1/report. Authentication is required by default, even on loopback; disabling it is an explicit loopback-only debugging exception, while a non-loopback listener needs an explicit remote-listen opt-in.

Configuration and history

Profiles are explicit and versioned. Pass --config PATH or GPU_WATCHMAN_CONFIG, optionally select a profile, and expect precedence to be built-in defaults, profile, environment, then CLI.

123watchman config init gpu-watchman.toml
watchman --config gpu-watchman.toml config validate
watchman --config gpu-watchman.toml --profile local serve --no-api-auth

History summarizes runtime/source availability, peaks, non-OK sources, and recurring findings. Rotate it like an operational artifact, and use the project's deployment documentation before placing it behind a remote listener or Kubernetes service.

Troubleshoot evidence first

Start with doctor, then inspect the typed source evidence. Missing nvidia-smi, absent container GPU access, unavailable topology/MIG/Xid fields, generic endpoints, cleartext transport rejection, stale health, and 401 responses all have distinct causes and should not receive the same retry.

For endpoint-related confusion, return to health and collection. For active-generation and memory-model limits, return to inference and capacity.