# Voxtral-Mini-3B-2507 → onnx-asr `speech-llm`: export report

Source model: [mistralai/Voxtral-Mini-3B-2507](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507)
(Mistral AI, Apache-2.0, **not gated**). Offline Voxtral only — the realtime/streaming variant
is out of scope.

Runtime: [TigreGotico/onnx-asr](https://github.com/TigreGotico/onnx-asr), branch
`feat/speech-llm-voxtral` (off `feat/speech-llm-qwen3-asr`).

## Graph contract

| File | Inputs | Outputs |
| --- | --- | --- |
| `encoder.onnx` | `input_features` `(1, 128, 3000 * N)` | `audio_embeds` `(1, 375 * N, 3072)` |
| `embed_tokens.onnx` | `input_ids` `(1, S)` | `inputs_embeds` `(1, S, 3072)` |
| `decoder.onnx` | `inputs_embeds` `(1, S, 3072)`, `attn_bias` `(1, 1, S, P + S)`, `position_ids` `(1, S)`, `past_key_values.{0..29}.{key,value}` `(1, 8, P, 128)` | `logits` `(1, S, 131072)`, `present.{0..29}.{key,value}` `(1, 8, P + S, 128)` |

The audio tower is a Whisper-large-v3 encoder: full attention over a fixed 30 s (3000 frame)
mel window. It therefore declares only `input_features`, and the runtime sends the features of
the `whisper128` preprocessor unchanged — no packing indices and no block-diagonal bias, unlike
Qwen3-ASR. Inside the graph the flat window is split into 30 s chunks, encoded to 1500 frames
per chunk, grouped four frames at a time (the projector input width is `4 * 1280 = 5120`) and
projected to the 3072-wide language model embedding space, so 375 audio embeddings per chunk.

The feature extractor is a stock Whisper 128-mel extractor, identical to the `whisper128`
preprocessor already in onnx-asr, so no in-graph feature extraction was needed. Voxtral is a
full model with no LoRA or adapter to merge.

### Prompt

mistral-common encodes a transcription request as

```
<s> [INST] <begin_audio> <audio> * 375 <end_audio> [language tokens] [/INST]
```

so the ids around the audio are `prompt_prefix_ids = [1, 3, 25]` and
`prompt_suffix_ids = [4, 34]`. The language marker sits **after** the audio, where Qwen3-ASR
puts it before, so the export writes `language_suffix_ids` (both the ISO 639-1 code and the
English name as keys) for the eight languages Voxtral supports.

### Tokenizer

Voxtral uses tekken, a byte-level BPE. `MistralCommonBackend.get_vocab()` decodes every piece
as UTF-8, which is lossy — thousands of byte-level pieces collapse onto the replacement
character and the map loses entries. `vocab.json` is therefore built straight from
`tekken.json`: the 1000 control tokens by name, then every regular token written in the GPT-2
byte-to-unicode encoding that the onnx-asr byte-level decoder reverses exactly. 131072 entries,
no collisions.

## Runtime change

One commit on `feat/speech-llm-voxtral`: the optional `language_suffix_ids` config key, which
lets the language marker sit in the prompt suffix. Everything else in the family runtime is
unchanged, and models without the key behave exactly as before. Covered by three new
parametrized cases in `tests/onnx_asr/test_speech_llm.py` (11 tests pass).

## Parity

fp32 ONNX against native transformers fp32 (`VoxtralForConditionalGeneration`, greedy), four
FLEURS validation clips:

| Clip | fp32 ONNX vs native |
| --- | --- |
| `en_1` (6.5 s) | identical |
| `en_2` (16.4 s) | identical |
| `pt_1` (11.8 s) | identical |
| `pt_2` (14.6 s) | identical |

4/4 character-for-character.

int8 differs on two clips, in both cases by a word that is at least as good as the fp32 output:

* `en_2`: "Japanese cuisines **feature** an array" (fp32: "cuisines features an array").
* `pt_1`: "que orienta **as pesquisas científicas**" (fp32: "que orienta e pesquisa científica").
* `pt_2`: "**Esquiar**" (fp32: "Esquiá"), and the quotes around *ski bums* are dropped.

## Speed

Hardware: AMD Ryzen 5 7600 (6 cores / 12 threads), 124 GB RAM, CPU only, onnxruntime 1.28,
box shared with other jobs.

| Clip | audio | fp32 RTF | int8 RTF | native torch fp32 RTF |
| --- | --- | --- | --- | --- |
| `en_1` | 6.54 s | 8.30 | 1.06 | 3.48 |
| `en_2` | 16.38 s | 4.19 | 0.77 | 2.39 |
| `pt_1` | 11.82 s | 2.07 | 2.32 | 2.14 |
| `pt_2` | 14.58 s | 2.83 | 1.94 | 2.54 |

The first clip of every run carries the warm-up. Session load takes 222 s (fp32) and 122 s
(int8) from a cold page cache. int8 is the variant to use on CPU.

## Sizes

| Graph | fp32 | int8 |
| --- | --- | --- |
| `encoder` | 2.65 GB | 0.67 GB |
| `embed_tokens` | 1.61 GB | 0.40 GB |
| `decoder` | 14.45 GB | 3.61 GB |
| total | 18.7 GB | 4.7 GB |

`encoder.onnx`, `decoder.onnx` and `decoder_int8.onnx` keep their weights in a side
`.onnx_data` file.

## Limitations

* One 30 s window per call, as for every Whisper-preprocessor model in onnx-asr. Segment
  longer audio yourself. Voxtral itself handles up to 30 minutes.
* Greedy decoding only, one waveform at a time — both are properties of the family runtime.
* Only the transcription request is exported. Voxtral's audio understanding and chat modes
  are not reachable through this contract.

## Reproduce

`export_voxtral.py`, `verify_native.py` and `verify_onnx.py` are in this repository.

```sh
python export_voxtral.py --out <dir>
python verify_native.py
python verify_onnx.py fp32 int8
```