Ovis2.5-2B — LiteRT-LM (on-device Vision-Language Model)

AIDC-AI/Ovis2.5-2B converted to the LiteRT-LM (.litertlm) format for on-device image+text inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models, and the same runtime that runs litert-community/FastVLM-0.5B).

Ovis2.5 is a SOTA-for-size vision-language model (OpenCompass ~73.9 for the 2B) with a distinctive structural-embedding vision path: a Siglip2 NaViT encoder feeds a visual tokenizer that turns each image patch-group into a probability distribution over a 65 536-word visual vocabulary, then embeds it — giving the language model image tokens that live in the same structured space as text. The language decoder is Qwen3-1.7B. This bundle runs the whole thing through LiteRT-LM's fast_vlm multimodal path — give it an image and a question, get a grounded answer, fully on-device.

File	Ovis2.5-2B.litertlm (~2.15 GB)
Vision	Siglip2-NaViT encoder + visual-tokenizer (head → softmax → visual-vocab embedding), int8 weights — single 512×512 image → 256 image tokens
Decoder	Qwen3-1.7B, int4 weights (symmetric, blockwise-32 + OCTAV optimal-clipping); input embedding INT8 (externalized section)
Compute	integer
Context (KV cache)	2048
Image input	resized to 512×512 (Siglip normalization is baked into the vision encoder)
Base model	AIDC-AI/Ovis2.5-2B (Apache-2.0)

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the bundle, encodes your image and answers:

litert-lm run --from-huggingface-repo litert-community/Ovis2.5-2B Ovis2.5-2B.litertlm \
  --attachment photo.jpg \
  --prompt "Describe this image in one sentence."

On the COCO sample image huggingface/documentation-images/coco_sample.png (two tabby cats on a pink blanket, remote controls beside them) this prints:

Three tabby kittens lie on a pink blanket, with two remote controls beside them.

Drop --prompt for an interactive chat, and pass --attachment more than once for several images. litert-lm serve exposes the same bundle as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Device	Backend	Prefill (256)	Decode	TTFT
Apple M4 Max (macOS)	CPU	141 tok/s	34.9 tok/s	2.00 s
Apple M4 Max (macOS)	GPU (Metal)	1854 tok/s	142.6 tok/s	0.15 s

Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.

Accuracy note

The vision tower converts bit-faithfully to the reference — float CPU-parity end-to-end corr ≈ 1.0 (max abs diff 3e-6), with no FLEX/CUSTOM fallback ops; int8 vision weights keep end-to-end corr **0.99**. The Qwen3-1.7B decoder uses the same blockwise-32 + OCTAV int4 recipe that scores 90.7% GSM8K on the sibling Ministral-3-3B-Reasoning build and shipped the InternVL3.5-2B VLM. On a reference deployed-path eager run (fixed-512 vision → 256 tokens → Qwen3-1.7B) the model describes real photos accurately and in detail (e.g. a black-and-white Ansel-Adams-style landscape → "snow-capped sharp mountain peaks … a river winding through the valley … cloud layers … black-and-white contrast with depth of field").

Reasoning VLM. Ovis2.5 is a thinking model: it may emit a <think>…</think> block before its final answer (this matches the base model). Allow enough max-tokens (≥1024) for the answer to follow.

Thinking channel. Since the 2026-10-14 re-pack (metadata only: LlmMetadata.channels declares the thought channel with <think>…</think>; every other section of the file is byte-identical), litert-lm returns the reasoning on the thought channel (channels["thought"]) and the message text carries only the answer (litert-lm 0.18.0, measured 2026-10-14); the earlier file returned the <think> block inside the message text. enable_thinking=False does not switch the reasoning off (the chat template does not read it). If you downloaded before 2026-10-14, re-download.

On-device performance: decode/load are expected to be in line with the InternVL3.5-2B build on the same runtime (~20 tok/s CPU, ~45 tok/s GPU on iPhone 17 Pro for single-image VQA). Independent on-device measurement for this specific build is recommended before quoting exact numbers.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend and generates.

file	GPU backend	delegation	peak
Ovis2.5-2B.litertlm	runs	3840 / 3840 ops across 3 subgraphs on LiteRT GPU	820 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-25.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in main; the runtime accepts that split.

The gate prompt carries no image, so this covers engine creation and the text path. The vision path on the GPU is not measured here.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Run on iPhone / macOS

Use the LiteRT-LM Swift runtime (swift-litert-lm / the LiteRTDemo sample). Load Ovis2.5-2B.litertlm with the image (vision) tower enabled (modalities [.vision] / Modality.textImage), attach a photo, and ask a question.

Note for app integrators: this is a vision-only bundle (no audio tower). Bring up the engine with the vision modality only — requesting the audio tower (.all) on a bundle with no audio section fails at session creation.

Run on Android — Google AI Edge Gallery

Install a recent Google AI Edge Gallery (1.0.16+ can import .litertlm directly from Hugging Face), download Ovis2.5-2B.litertlm, import it (tap +), attach an image and ask. The bundle already carries the tokenizer and prompt template.

Conversion notes

• LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,512,512,3]→[1,256,4608]) + VISION_ADAPTER ([1,256,4608]→[1,256,2048], matched to the Qwen3-1.7B hidden size) + single-token EMBEDDER + PREFILL_DECODE (embeddings-input).
• The NaViT static rewrite is the enabling trick. Ovis's Siglip2-NaViT vision tower is dynamic resolution (.item()/.tolist()/grid-loops/argsort) and does not torch.export. Because the config's fullatt_block_indexes=None makes every layer use full attention, the window-reorder is a mathematical no-op — so it can be dropped and replaced with a precomputed position embedding + rotary and a single full attention over the fixed 512×512 grid (1024 patches). Static-vs-original feature corr 0.99999964.
• The encoder bakes Siglip normalization ((x-0.5)/0.5, the runtime feeds a [0,1] NHWC image) and does patchify GPU-safe: the patch-embedding Conv2d is applied to the whole image (raster order), then a single gather reorders patches into Ovis's hidden-stride "merge" order — all reshapes ≤4D, no >5D op that GPU delegates reject.
• The adapter is Ovis's visual-tokenizer tail: head (Linear 4608→65532 + LayerNorm) → softmax → visual-vocabulary embedding (vte, 65536×2048). The 256-token bundle carries the visual atoms; Ovis's two learned image-boundary indicator embeddings are omitted (the fast_vlm path splices only the atom embeddings) — verified to stay coherent in eager.
• Decoder extracted from the Ovis2_5 wrapper as a standalone Qwen3ForCausalLM and exported with cache ≤ base max so base RoPE is exact.

2026-08-28 — start_token fix (weights unchanged)

The bundle's LlmMetadata start_token held the literal string "None". This tokenizer has no BOS, and the LiteRT-LM engine resolved that string to a real vocabulary token — so every prompt began with the word None, which the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

2026-08-30 — tokenizer section replaced (weights unchanged)

The tokenizer in Ovis2.5-2B.litertlm was a SentencePiece conversion of the model's BPE tokenizer, and the conversion lost the byte-level semantics: a standalone accented letter or symbol (é, ñ, ü, °, ·, …) was encoded to the id of a single-byte token instead of the token the upstream tokenizer uses, and any character without a whole-character vocabulary entry (emoji, most of Latin Extended-A) became the token the conversion had reused as UNK — the end-of-text token for this vocabulary. Ovis2.5-2B.litertlm now embeds the upstream tokenizer.json (the same HF tokenizer path most bundles in this collection use).

Tokenizer-only change: every section of the bundle except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token — only the file's own sha256 differs. Verified on the LiteRT-LM runtime: the default turn, 7 probe strings, the 223 standalone characters U+00A1–U+017F and every special token now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions answer byte-identically to the previous file (same ids in, same tokens out). Prompts containing accented letters, symbols or emoji reach the model differently from before, so individual answers to such prompts can change. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file, and any on-device rows were measured on the previous file too — the on-device gate has not been re-run on this one (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30, re-download.

2026-08-31 — chat template fix: history turns with role 'assistant' (metadata-only, weights unchanged)

The chat template in Ovis2.5-2B.litertlm matched history turns only on role 'model', so a conversation created through the Conversation API with the standard role 'assistant' (for example create_conversation(messages=[...]) restoring a chat) silently dropped every assistant turn from the rendered prompt — the model saw consecutive user turns with its own replies gone. The template's role conditions now also accept 'assistant'. The engine's own send_message flow was never affected.

Template-only change: every section of the bundle except the metadata is byte-identical to the previous file (verified by per-section sha256), and single-turn renders, token ids, prefill counts and greedy answers are unchanged (verified on the LiteRT-LM runtime), so the numbers on this card still describe this file — only the file's own sha256 differs. Re-download only matters if you inject conversation history with role 'assistant'. Details: google-ai-edge/LiteRT-LM#3417 (https://github.com/google-ai-edge/LiteRT-LM/issues/3417).

Known runtime issues

After an answer without a thought (no <think> … </think> block), from the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because this file holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). After an answer with a thought, the runtime rebuilds the cache from the re-rendered history at the next turn, so the second turn is free of it when the conversation opens with a user message; from the third turn on a zero embedding is written either way, at the previous answer's stop token or, after a rebuild, at another token (read in the source, not measured). The bundle's chat template leaves it to the model whether an answer has a thought. One question per conversation avoids both, as does a new conversation that starts with the earlier messages and the new question.