vLLM NVFP4 fused-scale warning and 5090 quality result
Thanks for publishing this. I tested this exact checkpoint on a single RTX 5090 (Blackwell/SM120) with vLLM 0.25.1, no MTP/speculative decoding, BF16 activations, Marlin MoE, FlashInfer attention, NVFP4 KV cache, 131,072 context, and 8 concurrent sequences.
vLLM emitted this at load:
In NVFP4 linear, the weight global scale is different for parallel layers (e.g. q_proj, k_proj, v_proj). This will likely result in reduced accuracy. Please verify the model accuracy. Consider using a checkpoint with a shared global NVFP4 scale for fused layers.
The message is from vLLM’s compressed-tensors NVFP4 linear loader, before MoE kernel selection. It appears to see different global scales across fused parallel linear shards and reduce them to a shared scale.
As a quality signal—not proof that this warning is the sole cause—we ran one complete local 8-pack / 150-scenario agent+coding suite with thinking forced on (16,384-token budget): 109/150 (72.7%). Per pack: ToolCall 11/15, InstructFollow 13/15, StructOutput 14/15, DataExtract 11/15, ReasonMath 9/15, BugFind 10/15, HermesAgent 13/20, CLI 28/40.
For comparison only, our same-machine NVIDIA Qwen3.6-35B-A3B NVFP4 no-MTP serving profile scored 127/150 (84.7%) on the same suite; it is not a like-for-like model comparison. The actionable issue is whether this KAT checkpoint’s fused Q/K/V (and other parallel linear) global scales are expected for vLLM’s compressed-tensors NVFP4 path.
If useful I can provide the exact vLLM command/config and benchmark JSON details.