A GPU snapshot should say what it saw and what it could not see. Watchman collects stable NVIDIA inventory first, then joins process accounting, optional GPU details, topology, Xid logs, and inference endpoint telemetry into a schema-versioned report.

Hardware collection is read-only. Watchman does not change clocks, power limits, compute modes, MIG state, or workloads. The required NVIDIA inventory query decides whether a collection cycle can succeed; optional sources preserve any valid evidence instead of discarding the entire report.

An optional source is recorded as ok, partial, unavailable, or skipped. This distinction matters: a passively cooled GPU with an unavailable fan sensor is not automatically unhealthy, while a missing required kernel-log source can be made fail-closed by policy.

Use a full snapshot for node state and ps for ownership triage. The CLI keeps the focused process view separate from the full hardware report.

12345watchman snapshot --all --details
watchman ps
watchman top --watch 5s \
  --probe http://vllm:8000,http://triton:8002 \
  --allow-insecure-http

The underlying report can include NVIDIA GPU, driver, clocks, power, temperature, VRAM, utilization, ECC, retired-page, PCIe, MIG, throttle, and topology evidence alongside VRAM ownership by PID, user, container, and Kubernetes attribution.

Runtime probes run concurrently with collection. Watchman normalizes supported Prometheus sources from vLLM, TGI, Triton, TensorRT-LLM, SGLang, and Ollama into requests, queue/KV pressure, throughput, errors, preemption, and bounded latency estimates.

Remote probes are HTTPS by default, do not follow redirects, and do not inherit ambient proxy settings. Explicit cleartext is a deliberate exception through --allow-insecure-http; it should not become a production default.

Human output and machine output are different views of the same evidence. Use text for a terminal, JSON for one report, and NDJSON for continuous history. Watchman bounds command output and errors, reduces diagnostics to operational classifications, and avoids serializing arbitrary wrapper output into support artifacts.

Read inference and capacity for active workflows and rollouts and deployment for exporter, history, and admission decisions.