gemma-4-E4B-it-int4-sym-cw-ov

google/gemma-4-E4B-it (revision 4ecc2ba) as an OpenVINO IR with int4 symmetric weights (channel-wise, ratio 1.0), a form the OpenVINO GenAI NPU guide gives for LLMs on the Intel NPU.

Run it

On a PC with an Intel NPU (Core Ultra) and its driver:

pip install openvino-genai==2026.4.1 huggingface_hub
hf download mlboydaisuke/gemma-4-E4B-it-int4-sym-cw-ov --local-dir gemma-4-E4B-it-int4-sym-cw-ov
import openvino_genai as ov_genai

pipe = ov_genai.VLMPipeline("gemma-4-E4B-it-int4-sym-cw-ov", "NPU", CACHE_DIR="npu_cache")
print(pipe.generate("What is on-device AI? Answer in two sentences.", max_new_tokens=64).texts[0])

The first run fills npu_cache. On the laptop below, the first load took 89.0 s with it empty, and later loads took 5.3 to 14.2 s with it filled. On 2026-11-12 these lines printed this answer on that laptop (Intel NPU driver 32.0.100.5540, openvino-genai 2026.4.1.0, Python 3.12.16, Windows 11 build 26200.9711):

On-device AI refers to the integration of artificial intelligence directly into devices like smartphones or computers, allowing them to process data and execute tasks locally. This enables features like enhanced voice recognition, personalized alerts, and improved performance without needing a constant internet connection.

Export

The OpenVINO GenAI NPU guide says LLMs must be exported with --sym, --weight-format int4 or nf4, --group-size -1 or 128 and --ratio 1.0; OpenVINO/gemma-4-E4B-it-int4-ov is int4_asym, group size 64, ratio 1.0, AWQ (its rt_info, the quantization metadata in the IR's .xml), so this folder was exported in that form (int4). This model's group-size-128 export (--group-size 128) ended its NPU run with exit code 0xC0000005 (access violation) before the first answer here, on driver 32.0.100.4841 (2026-11-08) (raw/walk-npuexp3/), so this folder is the channel-wise one (--group-size -1).

optimum-cli export openvino --model <google/gemma-4-E4B-it folder> --task image-text-to-text --weight-format int4 --sym --ratio 1.0 --group-size -1 <out folder>

The --model folder held the files of google/gemma-4-E4B-it at revision 4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, each checked against the Hub's sha256 (git blob sha1 for small files) before the export. Versions: optimum-intel 2.2.0, optimum 2.3.0, nncf 3.4.0, openvino 2026.4.1, openvino-tokenizers 2026.4.1.0, transformers 5.5.4, torch 2.14.0, Python 3.12.10, on Windows 11 Home. The IR's rt_info: mode int4_sym, group_size -1, ratio 1.0, awq False, scale_estimation False, backup_mode int8_asym.

Measured on one laptop's NPU

OpenVINO GenAI 2026.4.1 with VLMPipeline(<this folder>, "NPU", CACHE_DIR=<dir>), no other NPU property. One user turn (edge-llm-bench's short-chat prompt) in the folder's chat template, 20 tokens; greedy, at most 128 new tokens. Each run is a fresh process that answers twice: cold = its first generation, warm = its second. The numbers below come from one measuring session on the laptop (three runs). Run 1 is not counted (the rule of windows-llm-zoo for NPU and GPU runs); runs 2 and 3 are. CACHE_DIR was the same folder in every run of these records.

laptop Intel Core Ultra 9 386H, Intel NPU driver 32.0.100.5540, Windows 11 Home build 26200.9711, 31.5 GB RAM (as Windows reports it), MSI Center's Balanced mode; 2026-11-11
decode, cold 13.2 tokens/s (median of 2 runs, spread 0.5 %)
decode, warm 13.3 tokens/s
prefill, cold 15.8 tokens/s, 1264 ms to the first token
peak working set 7,037 MiB (the process)
load 14.2 s in run 1, 5.5, 5.3 s in runs 2 and 3 (all three after the first load below, with the same CACHE_DIR)
output check 6/6 PASS (edge-llm-bench's text check); new tokens per answer: 128, 128

On this driver the first load of this folder, with CACHE_DIR empty, took 89.0 s. It was in the answer-agreement run below (2026-11-11; load_s in run1.json).

The answer begins: "Imagine you have a small, smart assistant that lives inside your device (like your phone or laptop), rather than …"

The numbers are text only: the image path of this folder was not run on the NPU here.

Records: the three runs and the laptop's environment in records/2026-11-11/; the script: pc/run_ovgenai.py.

Answer agreement with the original model

On 20 short questions (thinking off), the first line of this export's answer matched google/gemma-4-E4B-it's (bf16, transformers, CPU) on 19. The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. One run on the same NPU, 2026-11-11, driver 32.0.100.5540: record.

Downloads last month
121
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for mlboydaisuke/gemma-4-E4B-it-int4-sym-cw-ov

Quantized
(464)
this model