Qwen3-0.6B — INT4 · LiteRT-LM

On-device INT4 builds of Qwen/Qwen3-0.6B in the .litertlm format for Google's LiteRT-LM runtime (Android / iOS / desktop / embedded).

At ~332 MB each (≈7× smaller than the fp32 source), these load fast and keep a small memory footprint on phones.

Two flavours — pick your thinking behaviour

Qwen3 is a hybrid reasoning model. Because LiteRT-LM bakes the prompt format into the file, the thinking mode is fixed per file, so this repo ships both:

File	Thinking	Best for
`qwen3_0.6b_q4_block32_ekv1280.litertlm`	ON (default Qwen3)	quality / reasoning; emits `<think>…</think>` then the answer
`qwen3_0.6b_nothink_q4_block32_ekv1280.litertlm`	OFF	fast, short, direct answers — ~2× quicker on a CPU backend; no chain-of-thought to filter out

The standard file embeds Qwen3's Jinja chat template (faithful ChatML; if your runtime can pass enable_thinking, it stays controllable).
The no-think file prefills a closed empty <think></think> into the assistant turn, so the model answers directly. This was built for a wearable health-assistant where a phone streams short replies to a watch.

Build details

Converted with litert-torch (the generative API, formerly ai-edge-torch) from the original PyTorch checkpoint.

Setting	Value
Source checkpoint	`Qwen/Qwen3-0.6B` (bf16)
Quantization	`dynamic_int4_block32` (INT4 weights, block size 32, fp32 activations)
Size	~2.2 GiB → 329 MiB (≈6.9× smaller)
Prefill signatures	`8, 64, 128, 256, 512, 1024` (runtime picks the smallest that fits)
KV cache max length	`1280`
KV layout	transposed · `mask_as_input=true`
Stop tokens	`151645` (`<
Tokenizer	bundled from `tokenizer.json`

Multiple prefill signatures matter for latency: with a single large signature every prompt is padded to it. The exponential set above lets a short prompt run a tiny prefill instead of always paying the 1024-token cost.

Prompt format

Standard Qwen3 ChatML. The no-think file pre-closes thinking on the assistant side:

<|im_start|>user
{user message}<|im_end|>
<|im_start|>assistant
<think>

</think>

{answer}<|im_end|>

Usage (LiteRT-LM, Android / Kotlin)

val engine = Engine(
    EngineConfig(
        modelPath = "/path/to/qwen3_0.6b_nothink_q4_block32_ekv1280.litertlm",
        backend = Backend.CPU(),
        cacheDir = context.cacheDir.absolutePath,
    )
).apply { initialize() }

val conversation = engine.createConversation(
    ConversationConfig(
        systemInstruction = Contents.of("You are a helpful, concise assistant."),
        samplerConfig = SamplerConfig(temperature = 0.8, topK = 40, topP = 0.9),
    )
)

conversation.sendMessageAsync("How did I sleep last night?")
    .collect { msg -> /* stream text */ }

Stop tokens and the prompt format are embedded in each file — no extra runtime configuration required. With the standard (thinking-on) file you may want to strip the leading <think>…</think> block before display.

Performance (on device, measured)

Community measurement on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16): litert_lm_advanced_main from the litert-lm v0.16.0 release, GPU backend OpenCL (LITERT_CL) against CPU (XNNPACK) on the same device, one fixed 205-token prompt text (202 tokens under the standard file's tokenizer, 206 under the nothink file's), --benchmark. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, every file/backend pair was generation-checked with a real prompt (for the nothink file a simple arithmetic prompt — its GPU and CPU outputs were identical, so the GPU runs the model faithfully); the GPU rows are full delegation (both files: decode 1437/1437 ops and every prefill signature fully on LITERT_CL).

File	Backend	Prefill	Decode	Time-to-first-token	Peak RSS
`qwen3_0.6b_q4_block32_ekv1280.litertlm`	GPU (OpenCL)	1383–1396 tok/s	39.5–45.4 tok/s	0.17 s	730–731 MB
`qwen3_0.6b_q4_block32_ekv1280.litertlm`	CPU (XNNPACK)	221–281 tok/s	37.9–42.1 tok/s	0.74–0.94 s	981–1037 MB
`qwen3_0.6b_nothink_q4_block32_ekv1280.litertlm`	GPU (OpenCL)	1404–1416 tok/s	44.2–50.0 tok/s	0.17 s	761–770 MB
`qwen3_0.6b_nothink_q4_block32_ekv1280.litertlm`	CPU (XNNPACK)	311–313 tok/s	47.2–47.3 tok/s	0.68 s	1032–1036 MB

What the table says:

The GPU's win is the prompt side: prefill 4.5–6.3× faster than the same phone's CPU, and time-to-first-token 0.17 s against 0.7–0.9 s. Decode is a wash (44–50 against 47 tok/s on the nothink file) — the usual shape for sub-1B models, where the CPU keeps up with generation but not with prompt processing.
Peak RSS is ~25% lower on the GPU (730–770 against 980–1040 MB). GPU engine init is the one cost the table hides: 2.7–5.6 s per process against 0.9–1.3 s on CPU.

Reproduce

import json
from litert_torch.generative.examples.qwen import qwen3
from litert_torch.generative.utilities import converter, export_config as ec_lib
from litert_torch.generative.layers import kv_cache as kv_utils

CKPT = "Qwen3-0.6B"  # local HF snapshot
ec = ec_lib.ExportConfig()
ec.kvcache_layout = kv_utils.KV_LAYOUT_TRANSPOSED
ec.mask_as_input = True

common = dict(
    output_path=".",
    prefill_seq_len=[8, 64, 128, 256, 512, 1024],
    kv_cache_max_len=1280,
    quantize="dynamic_int4_block32",
    export_config=ec,
    output_format="litertlm",
    hf_tokenizer_model_path=f"{CKPT}/tokenizer.json",
    stop_token_ids=[151645, 151643],
)

# Standard (thinking on): bake the Qwen3 Jinja template
tpl = json.load(open(f"{CKPT}/tokenizer_config.json"))["chat_template"]
converter.convert_to_litert(
    qwen3.build_0_6b_model(CKPT), output_name_prefix="qwen3_0.6b",
    jinja_prompt_template=tpl, **common)

# No-think: prefill a closed empty think block in the assistant turn
converter.convert_to_litert(
    qwen3.build_0_6b_model(CKPT), output_name_prefix="qwen3_0.6b_nothink",
    user_prompt_prefix="<|im_start|>user\n", user_prompt_suffix="<|im_end|>\n",
    model_prompt_prefix="<|im_start|>assistant\n<think>\n\n</think>\n\n",
    model_prompt_suffix="<|im_end|>\n", **common)

Training data, provenance & privacy

This is a format/precision conversion only — no fine-tuning or additional training was performed. Weights and behaviour are inherited directly from Qwen/Qwen3-0.6B; see the Qwen3 technical report for that model's pretraining data.

No new training data was introduced by this contribution, so there is no additional dataset to summarise. Because no data was collected or used here, there is no PII involved in this submission — the conversion operates purely on the public base-model weights and tokenizer.

Intended use & limitations

Use: offline assistants on phones / edge devices — chat, summaries, simple Q&A over provided context.
No-think file: not for complex multi-step reasoning or math (thinking is off). Use the standard file or a larger Qwen3 for that.
Quantization: INT4 introduces some quality loss vs the fp32/INT8 source.
Inherits the capabilities, biases, and limitations of Qwen/Qwen3-0.6B.