Qwen3-4B-int4-sym-g128-ov

Qwen/Qwen3-4B (revision 1cfa9a7) as an OpenVINO IR with int4 symmetric weights (group size 128, ratio 1.0), a form the OpenVINO GenAI NPU guide gives for LLMs on the Intel NPU.

This export answers on the Intel NPU. The official OpenVINO/Qwen3-4B-int4-ov (AWQ) gave no usable answer on the same NPU; the details are in the last section.

Export

optimum-cli export openvino --model <Qwen/Qwen3-4B folder> --task text-generation-with-past --weight-format int4 --sym --ratio 1.0 --group-size 128 <out folder>

The --model folder held the files of Qwen/Qwen3-4B at revision 1cfa9a7208912126459214e8b04321603b3df60c, each checked against the Hub's sha256 (git blob sha1 for small files) before the export. Versions: optimum-intel 2.2.0, optimum 2.3.0, nncf 3.4.0, openvino 2026.4.1, openvino-tokenizers 2026.4.1.0, transformers 5.5.4, torch 2.14.0, Python 3.12.10, on Windows 11 Home. The IR's rt_info: mode int4_sym, group_size 128, ratio 1.0, awq False, scale_estimation False, backup_mode int8_asym.

Measured on one laptop's NPU

OpenVINO GenAI 2026.4.1 with LLMPipeline(<this folder>, "NPU", CACHE_DIR=<dir>), no other NPU property. One user turn (edge-llm-bench's short-chat prompt) in the folder's chat template, 19 tokens; greedy, at most 128 new tokens. Each run is a fresh process that answers twice: cold = its first generation, warm = its second. The numbers below come from one measuring session on the laptop (three runs). Run 1 is not counted (the rule of windows-llm-zoo for NPU and GPU runs); runs 2 and 3 are. CACHE_DIR was the same folder in every run of these records.

laptop Intel Core Ultra 9 386H, Intel NPU driver 32.0.100.5540, Windows 11 Home build 26200.9711, 31.5 GB RAM (as Windows reports it), MSI Center's Balanced mode; 2026-11-11
decode, cold 22.0 tokens/s (median of 2 runs, spread 0.2 %)
decode, warm 22.5 tokens/s
prefill, cold 18.6 tokens/s, 1022 ms to the first token
peak working set 5,562 MiB (the process)
load 3.6 s in run 1, 3.3, 4.1 s in runs 2 and 3 (all three after the first load below, with the same CACHE_DIR)
output check 6/6 PASS (edge-llm-bench's text check); new tokens per answer: 128, 128

On this driver the first load of this folder, with CACHE_DIR empty, took 38.4 s. It was in the answer-agreement run below (2026-11-11; load_s in run1.json).

The answer begins after its <think> line: "Okay, the user wants me to explain what on-device AI is in simple terms. Let me start by recalling what I know about AI โ€ฆ". At 128 new tokens every answer here ends inside the think block (no </think> in any of them).

Records: the three runs and the laptop's environment in records/2026-11-11/; the script: pc/run_ovgenai.py.

Answer agreement with the original model

On 20 short questions (thinking off), the first line of this export's answer matched Qwen/Qwen3-4B's (fp32, transformers, CPU) on 19. The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. One run on the same NPU, 2026-11-11, driver 32.0.100.5540: record.

Facts in long prompts: this export got the fact right in all 160 prompts on the NPU and on the CPU in f32, with the same answer tokens on both devices; the original model also got all 160 right. The 160 prompts (8 short facts ร— 5 places ร— 4 lengths: 256, 512, 768 and 1,000 tokens) put one fact into a passage (as its first sentence, about a quarter, half or three quarters in, or near the end) and ask about it at the end (thinking off); decoding is greedy. The NPU ran LLMPipeline(<this folder>, "NPU", CACHE_DIR=<dir>) as above (a second run, from the cache, gave the same answer tokens); the CPU ran the same IR in f32, to tell the export's own misses from the NPU's (its properties are in the record); the original model is Qwen/Qwen3-4B in transformers on the CPU, computing in fp32 with most of its weights kept in bf16 to save memory (run this way, Qwen/Qwen3-1.7B gave the same answer tokens as with its weights in fp32, on every prompt of this check). On the same laptop, 2026-11-13, NPU driver 32.0.100.5540: record (the answers, the prompts and the scoring).

The official OpenVINO IR on this NPU

OpenVINO/Qwen3-4B-int4-ov (int4_sym, group size 128, ratio 1.0, AWQ: its rt_info, the quantization metadata in the IR's .xml) gave no usable answer on this laptop's NPU with OpenVINO GenAI 2026.4.1, NPU driver 32.0.100.4841: the prompt in its chat template, as the runner sends it, produced a stop token (<|endoftext|>) as its first token, so the answer was empty; the pipeline's own generate(text) wrote one word over and over, with the template and without it (raw/diag/q4npu.log). On driver 32.0.100.5540 (2026-11-11) the runner's answer was empty again: one new token and no text in each of the run's two answers (records/2026-11-11/try_OpenVINO-Qwen3-4B-int4-ov.json). The same IR answered on the CPU (raw/diag/q4cpu.log). This export, without AWQ, answers on the NPU. The cause was not found (the export tool versions differ too).

Downloads last month
144
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mlboydaisuke/Qwen3-4B-int4-sym-g128-ov

Finetuned
Qwen/Qwen3-4B
Quantized
(397)
this model