Qwen3-4B-int4-sym-g128-ov
Qwen/Qwen3-4B (revision 1cfa9a7) as an OpenVINO IR with int4 symmetric weights (group size 128, ratio 1.0), a form the OpenVINO GenAI NPU guide gives for LLMs on the Intel NPU.
This export answers on the Intel NPU. The official OpenVINO/Qwen3-4B-int4-ov (AWQ) gave no usable answer on the same NPU; the details are in the last section.
Export
optimum-cli export openvino --model <Qwen/Qwen3-4B folder> --task text-generation-with-past --weight-format int4 --sym --ratio 1.0 --group-size 128 <out folder>
The --model folder held the files of Qwen/Qwen3-4B at revision 1cfa9a7208912126459214e8b04321603b3df60c, each checked against the Hub's sha256 (git blob sha1 for small files) before the export. Versions: optimum-intel 2.2.0, optimum 2.3.0, nncf 3.4.0, openvino 2026.4.1, openvino-tokenizers 2026.4.1.0, transformers 5.5.4, torch 2.14.0, Python 3.12.10, on Windows 11 Home. The IR's rt_info: mode int4_sym, group_size 128, ratio 1.0, awq False, scale_estimation False, backup_mode int8_asym.
Measured on one laptop's NPU
OpenVINO GenAI 2026.4.1 with LLMPipeline(<this folder>, "NPU", CACHE_DIR=<dir>), no other NPU property. One user turn (edge-llm-bench's short-chat prompt) in the folder's chat template, 19 tokens; greedy, at most 128 new tokens. Each run is a fresh process that answers twice: cold = its first generation, warm = its second. The numbers below come from one measuring session on the laptop (three runs). Run 1 is not counted (the rule of windows-llm-zoo for NPU and GPU runs); runs 2 and 3 are. CACHE_DIR was the same folder in every run of these records.
| laptop | Intel Core Ultra 9 386H, Intel NPU driver 32.0.100.5540, Windows 11 Home build 26200.9711, 31.5 GB RAM (as Windows reports it), MSI Center's Balanced mode; 2026-11-11 |
| decode, cold | 22.0 tokens/s (median of 2 runs, spread 0.2 %) |
| decode, warm | 22.5 tokens/s |
| prefill, cold | 18.6 tokens/s, 1022 ms to the first token |
| peak working set | 5,562 MiB (the process) |
| load | 3.6 s in run 1, 3.3, 4.1 s in runs 2 and 3 (all three after the first load below, with the same CACHE_DIR) |
| output check | 6/6 PASS (edge-llm-bench's text check); new tokens per answer: 128, 128 |
On this driver the first load of this folder, with CACHE_DIR empty, took 38.4 s. It was in the answer-agreement run below (2026-11-11; load_s in run1.json).
The answer begins after its <think> line: "Okay, the user wants me to explain what on-device AI is in simple terms. Let me start by recalling what I know about AI โฆ". At 128 new tokens every answer here ends inside the think block (no </think> in any of them).
Records: the three runs and the laptop's environment in records/2026-11-11/; the script: pc/run_ovgenai.py.
Answer agreement with the original model
On 20 short questions (thinking off), the first line of this export's answer matched Qwen/Qwen3-4B's (fp32, transformers, CPU) on 19. The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. One run on the same NPU, 2026-11-11, driver 32.0.100.5540: record.
Facts in long prompts: this export got the fact right in all 160 prompts on the NPU and on the CPU in f32, with the same answer tokens on both devices; the original model also got all 160 right. The 160 prompts (8 short facts ร 5 places ร 4 lengths: 256, 512, 768 and 1,000 tokens) put one fact into a passage (as its first sentence, about a quarter, half or three quarters in, or near the end) and ask about it at the end (thinking off); decoding is greedy. The NPU ran LLMPipeline(<this folder>, "NPU", CACHE_DIR=<dir>) as above (a second run, from the cache, gave the same answer tokens); the CPU ran the same IR in f32, to tell the export's own misses from the NPU's (its properties are in the record); the original model is Qwen/Qwen3-4B in transformers on the CPU, computing in fp32 with most of its weights kept in bf16 to save memory (run this way, Qwen/Qwen3-1.7B gave the same answer tokens as with its weights in fp32, on every prompt of this check). On the same laptop, 2026-11-13, NPU driver 32.0.100.5540: record (the answers, the prompts and the scoring).
The official OpenVINO IR on this NPU
OpenVINO/Qwen3-4B-int4-ov (int4_sym, group size 128, ratio 1.0, AWQ: its rt_info, the quantization metadata in the IR's .xml) gave no usable answer on this laptop's NPU with OpenVINO GenAI 2026.4.1, NPU driver 32.0.100.4841: the prompt in its chat template, as the runner sends it, produced a stop token (<|endoftext|>) as its first token, so the answer was empty; the pipeline's own generate(text) wrote one word over and over, with the template and without it (raw/diag/q4npu.log). On driver 32.0.100.5540 (2026-11-11) the runner's answer was empty again: one new token and no text in each of the run's two answers (records/2026-11-11/try_OpenVINO-Qwen3-4B-int4-ov.json). The same IR answered on the CPU (raw/diag/q4cpu.log). This export, without AWQ, answers on the NPU. The cause was not found (the export tool versions differ too).
- Downloads last month
- 144