Watchman inference and capacity

Active inference work has a different contract from passive observation. Watchman keeps canaries, saturation experiments, runtime fingerprints, artifact inspection, and capacity planning as distinct workflows so their evidence is not silently mixed.

Validate one real generation path

A canary is a correctness and SLO probe, not a load test. It sends bounded synthetic OpenAI-compatible requests, tracks streaming TTFT, end-to-end latency, expected output, usage-backed output throughput, and explicit success gates.

123456watchman canary \
  --base-url https://candidate.example/v1 \
  --model served-model \
  --api-key-file /run/secrets/inference-api-key \
  --count 5 --concurrency 2 \
  --max-ttft 2s --max-e2e 10s --min-success-percent 100

Use secret files rather than command-line values. Prompts, credentials, and generated content are excluded from output; direct CLI values can still leak to process listings or shell history.

Characterize explicit concurrency

Saturation Benchmark v1 is a bounded closed-loop ladder. It uses explicit, strictly increasing concurrency stages, excluded warmup, a simultaneous release, fixed measured attempts, and independently reported goodput and latency evidence.

1234567watchman benchmark saturation \
  --base-url https://candidate.example/v1 \
  --model served-model \
  --api-key-file /run/secrets/inference-api-key \
  --concurrency-stages 1,2,4,8 \
  --verify-concurrency 8 \
  --max-error-percent 1 --max-p95-ttft 2s --max-p95-e2e 10s

The highest accepted stage is not production capacity. A single-process closed-loop test can be biased by its own network, coordinated omission, prefix caching, unrelated traffic, and a distinction between client concurrency, server batching, and GPU occupancy.

Inspect without loading payloads

Runtime Fingerprint v1 is bounded local evidence for explicit PIDs. On Linux, it reads selected /proc evidence without starting nvidia-smi, an engine, Python, a package manager, or any child process. It does not claim a compatibility verdict and excludes paths, hostnames, raw argv, environment values, and model identities from the report.

Artifact Report v1 inspects safetensors metadata, not tensor contents. It validates headers, index membership, dtype geometry, and offsets while retaining aggregate facts rather than paths, shard names, tensor names, metadata values, or payload bytes.

12watchman runtime inspect --pid 4242 --format json
watchman artifact inspect /models/served-model/model.safetensors.index.json --format json

Model capacity conservatively

Capacity Report v3 models an explicit topology and charges the worst rank. It accepts tensor, pipeline, data, and expert parallelism, then derives worst-rank weights, KV cache, headroom, and a fit decision from stated bounds.

12345watchman capacity \
  --params 70 --weight-bits 4 --tp 2 --gpu-vram 80 \
  --max-shared-rank-weight-percent 50 \
  --layers 80 --kv-heads 8 --head-dim 128 \
  --max-kv-heads-per-rank 4 --context 32768 --concurrency 8

Do not mistake serialized checkpoint bytes for resident GPU allocation. Capacity planning names its assumptions and caveats instead of assuming tensor placement, allocator behavior, activations, CUDA graphs, or communication workspaces away.