docs: add local model capability profile plan

README.md

@@ -18,3 +18,7 @@ diff explorer.
 Latest report:
 
 - `gepa-evalstate-qwen-overlay-c4-full-20260616T172947Z/final-report.html`
+
+Plans:
+
+- `experiments/2026-06-18-local-model-capability-profile-plan.md`

experiments/2026-06-18-local-model-capability-profile-plan.md

@@ -0,0 +1,169 @@
+# Local Model Capability Profile Plan
+
+Date: 2026-06-18
+
+## Problem
+
+LocalPager, LocalPi, and vLLM currently have separate partial truths about the
+loaded model.
+
+- vLLM is started with runtime flags such as model id, context length, tool
+  parser, reasoning parser, max sequences, and batching limits.
+- LocalPager worker config separately stores model id, endpoint, context window,
+  output token cap, timeout, and concurrency.
+- Pi receives a generated `pi-config-runtime/models.json` snapshot that tells it
+  whether the selected model supports reasoning and which thinking request
+  format to use.
+
+When these drift, the client can silently do the wrong thing. The concrete
+failure found on 2026-06-18 was:
+
+- vLLM served `nvidia/Gemma-4-26B-A4B-NVFP4` with `--reasoning-parser gemma4`.
+- A direct vLLM request with `chat_template_kwargs.enable_thinking=true` returned
+  a real `reasoning` block.
+- The generated Pi config still said `"reasoning": false`.
+- LocalPi warned `thinking: off (clamped from medium)`.
+
+vLLM did not advertise that thinking was off. Its `/v1/models` response exposed
+the model id and `max_model_len`, but no reasoning capability field.
+
+## Goal
+
+Create one source of truth for local model serving capabilities, then generate
+the vLLM server command, LocalPager worker config, and Pi model config from that
+source.
+
+The desired invariant is:
+
+> If port 8000 serves Gemma with reasoning enabled, LocalPager and Pi must know
+> that Gemma is loaded, reasoning is supported, and the correct request shape is
+> `chat_template_kwargs.enable_thinking`.
+
+## Proposed Design
+
+Add a local model profile file. It should be explicit, small, and checked into
+LocalPager examples for known setups. Local machine copies can live under
+`~/.config/localpager/`.
+
+Example:
+
+```json
+{
+  "id": "gemma4-26b-a4b-nvfp4",
+  "model": "nvidia/Gemma-4-26B-A4B-NVFP4",
+  "base_url": "http://127.0.0.1:8000/v1",
+  "server": {
+    "runtime": "vllm",
+    "host": "127.0.0.1",
+    "port": 8000,
+    "max_model_len": 32768,
+    "max_num_seqs": 16,
+    "max_num_batched_tokens": 32768,
+    "tool_call_parser": "gemma4",
+    "reasoning_parser": "gemma4",
+    "moe_backend": "cutlass",
+    "kv_cache_dtype": "fp8"
+  },
+  "client": {
+    "context_window": 32768,
+    "max_tokens": 4096,
+    "timeout_ms": 300000,
+    "concurrency": 16
+  },
+  "capabilities": {
+    "tools": true,
+    "reasoning": true,
+    "thinking_format": "qwen-chat-template"
+  }
+}
+```
+
+`thinking_format: "qwen-chat-template"` is the current Pi-compatible way to send:
+
+```json
+{
+  "chat_template_kwargs": {
+    "enable_thinking": true,
+    "preserve_thinking": true
+  }
+}
+```
+
+That request shape was verified against the local Gemma NVFP4 vLLM server.
+
+## Implementation Steps
+
+1. Add a typed model profile schema in LocalPager.
+2. Add example profiles for:
+   - `nvidia/Qwen3.6-35B-A3B-NVFP4`
+   - `nvidia/Gemma-4-26B-A4B-NVFP4`
+3. Change the vLLM launcher to read the profile and emit the server command.
+4. Change LocalPager Agent to read the same profile when generating
+   `pi-config-runtime/models.json`.
+5. Change LocalPager worker config examples to be generated from, or validated
+   against, the same profile.
+6. Keep model-name heuristics only as fallback behavior.
+7. Add explicit override fields:
+   - `LOCALPAGER_AGENT_REASONING`
+   - `LOCALPAGER_AGENT_THINKING_FORMAT`
+   - `LOCALPAGER_AGENT_PROFILE`
+8. Add preflight checks before benchmarks.
+
+## Preflight Checks
+
+Before a benchmark or classifier run:
+
+1. Query `/v1/models`.
+2. Verify exactly one expected model is served.
+3. Verify served model id matches the profile and LocalPager worker config.
+4. Verify server-reported context length does not exceed the profile in an
+   unexpected way.
+5. Generate Pi config from the profile.
+6. Launch a tiny Pi/localpager-agent request with `--thinking medium`.
+7. Fail if Pi reports `thinking: off (clamped from medium)`.
+8. Send a direct OpenAI-compatible request with the profile's thinking format.
+9. Fail if no `reasoning` field is returned for a reasoning-enabled profile.
+10. Fail if `worker.max_concurrency` exceeds `server.max_num_seqs`.
+
+## Why This Is Better Than Name Guessing
+
+Name guessing is brittle. It failed for `nvidia/Gemma-4-26B-A4B-NVFP4` because
+the existing mapping recognized older Gemma 12B names but not the new NVIDIA
+Gemma 4 NVFP4 id.
+
+Server discovery is also insufficient. vLLM's OpenAI-compatible `/v1/models`
+endpoint does not advertise thinking support or the required thinking request
+format.
+
+The stable solution is explicit capability declaration plus a runtime preflight.
+
+## Acceptance Criteria
+
+- Gemma NVFP4 profile generates Pi config with:
+
+```json
+{
+  "reasoning": true,
+  "compat": {
+    "thinkingFormat": "qwen-chat-template"
+  }
+}
+```
+
+- Qwen NVFP4 profile keeps its existing thinking behavior.
+- Running LocalPi/LocalPager Agent with Gemma and `--thinking medium` no longer
+  clamps to off.
+- Direct vLLM probe for Gemma returns a non-empty `reasoning` field.
+- A mismatch between served model and LocalPager worker config fails preflight
+  before any benchmark starts.
+- Benchmarks record the profile file used for the run.
+
+## Open Questions
+
+- Should profiles live in LocalPager only, or should LocalPi also understand the
+  same file format directly?
+- Should the profile schema allow multiple served models, or keep one model per
+  vLLM server process?
+- Should preflight be a standalone command, a runner option, or both?
+- Should vLLM server launch and LocalPager worker config be generated files or
+  validated hand-written files?