Voxtral-Mini-3B-2507 → onnx-asr speech-llm: export report
Source model: mistralai/Voxtral-Mini-3B-2507 (Mistral AI, Apache-2.0, not gated). Offline Voxtral only — the realtime/streaming variant is out of scope.
Runtime: TigreGotico/onnx-asr, branch
feat/speech-llm-voxtral (off feat/speech-llm-qwen3-asr).
Graph contract
| File | Inputs | Outputs |
|---|---|---|
encoder.onnx |
input_features (1, 128, 3000 * N) |
audio_embeds (1, 375 * N, 3072) |
embed_tokens.onnx |
input_ids (1, S) |
inputs_embeds (1, S, 3072) |
decoder.onnx |
inputs_embeds (1, S, 3072), attn_bias (1, 1, S, P + S), position_ids (1, S), past_key_values.{0..29}.{key,value} (1, 8, P, 128) |
logits (1, S, 131072), present.{0..29}.{key,value} (1, 8, P + S, 128) |
The audio tower is a Whisper-large-v3 encoder: full attention over a fixed 30 s (3000 frame)
mel window. It therefore declares only input_features, and the runtime sends the features of
the whisper128 preprocessor unchanged — no packing indices and no block-diagonal bias, unlike
Qwen3-ASR. Inside the graph the flat window is split into 30 s chunks, encoded to 1500 frames
per chunk, grouped four frames at a time (the projector input width is 4 * 1280 = 5120) and
projected to the 3072-wide language model embedding space, so 375 audio embeddings per chunk.
The feature extractor is a stock Whisper 128-mel extractor, identical to the whisper128
preprocessor already in onnx-asr, so no in-graph feature extraction was needed. Voxtral is a
full model with no LoRA or adapter to merge.
Prompt
mistral-common encodes a transcription request as
<s> [INST] <begin_audio> <audio> * 375 <end_audio> [language tokens] [/INST]
so the ids around the audio are prompt_prefix_ids = [1, 3, 25] and
prompt_suffix_ids = [4, 34]. The language marker sits after the audio, where Qwen3-ASR
puts it before, so the export writes language_suffix_ids (both the ISO 639-1 code and the
English name as keys) for the eight languages Voxtral supports.
Tokenizer
Voxtral uses tekken, a byte-level BPE. MistralCommonBackend.get_vocab() decodes every piece
as UTF-8, which is lossy — thousands of byte-level pieces collapse onto the replacement
character and the map loses entries. vocab.json is therefore built straight from
tekken.json: the 1000 control tokens by name, then every regular token written in the GPT-2
byte-to-unicode encoding that the onnx-asr byte-level decoder reverses exactly. 131072 entries,
no collisions.
Runtime change
One commit on feat/speech-llm-voxtral: the optional language_suffix_ids config key, which
lets the language marker sit in the prompt suffix. Everything else in the family runtime is
unchanged, and models without the key behave exactly as before. Covered by three new
parametrized cases in tests/onnx_asr/test_speech_llm.py (11 tests pass).
Parity
fp32 ONNX against native transformers fp32 (VoxtralForConditionalGeneration, greedy), four
FLEURS validation clips:
| Clip | fp32 ONNX vs native |
|---|---|
en_1 (6.5 s) |
identical |
en_2 (16.4 s) |
identical |
pt_1 (11.8 s) |
identical |
pt_2 (14.6 s) |
identical |
4/4 character-for-character.
int8 differs on two clips, in both cases by a word that is at least as good as the fp32 output:
en_2: "Japanese cuisines feature an array" (fp32: "cuisines features an array").pt_1: "que orienta as pesquisas científicas" (fp32: "que orienta e pesquisa científica").pt_2: "Esquiar" (fp32: "Esquiá"), and the quotes around ski bums are dropped.
Speed
Hardware: AMD Ryzen 5 7600 (6 cores / 12 threads), 124 GB RAM, CPU only, onnxruntime 1.28, box shared with other jobs.
| Clip | audio | fp32 RTF | int8 RTF | native torch fp32 RTF |
|---|---|---|---|---|
en_1 |
6.54 s | 8.30 | 1.06 | 3.48 |
en_2 |
16.38 s | 4.19 | 0.77 | 2.39 |
pt_1 |
11.82 s | 2.07 | 2.32 | 2.14 |
pt_2 |
14.58 s | 2.83 | 1.94 | 2.54 |
The first clip of every run carries the warm-up. Session load takes 222 s (fp32) and 122 s (int8) from a cold page cache. int8 is the variant to use on CPU.
Sizes
| Graph | fp32 | int8 |
|---|---|---|
encoder |
2.65 GB | 0.67 GB |
embed_tokens |
1.61 GB | 0.40 GB |
decoder |
14.45 GB | 3.61 GB |
| total | 18.7 GB | 4.7 GB |
encoder.onnx, decoder.onnx and decoder_int8.onnx keep their weights in a side
.onnx_data file.
Limitations
- One 30 s window per call, as for every Whisper-preprocessor model in onnx-asr. Segment longer audio yourself. Voxtral itself handles up to 30 minutes.
- Greedy decoding only, one waveform at a time — both are properties of the family runtime.
- Only the transcription request is exported. Voxtral's audio understanding and chat modes are not reachable through this contract.
Reproduce
export_voxtral.py, verify_native.py and verify_onnx.py are in this repository.
python export_voxtral.py --out <dir>
python verify_native.py
python verify_onnx.py fp32 int8