Jarbas's picture
Upload REPORT.md with huggingface_hub
c790f15 verified
|
Raw
History Blame Contribute Delete
5.19 kB

Voxtral-Mini-3B-2507 → onnx-asr speech-llm: export report

Source model: mistralai/Voxtral-Mini-3B-2507 (Mistral AI, Apache-2.0, not gated). Offline Voxtral only — the realtime/streaming variant is out of scope.

Runtime: TigreGotico/onnx-asr, branch feat/speech-llm-voxtral (off feat/speech-llm-qwen3-asr).

Graph contract

File Inputs Outputs
encoder.onnx input_features (1, 128, 3000 * N) audio_embeds (1, 375 * N, 3072)
embed_tokens.onnx input_ids (1, S) inputs_embeds (1, S, 3072)
decoder.onnx inputs_embeds (1, S, 3072), attn_bias (1, 1, S, P + S), position_ids (1, S), past_key_values.{0..29}.{key,value} (1, 8, P, 128) logits (1, S, 131072), present.{0..29}.{key,value} (1, 8, P + S, 128)

The audio tower is a Whisper-large-v3 encoder: full attention over a fixed 30 s (3000 frame) mel window. It therefore declares only input_features, and the runtime sends the features of the whisper128 preprocessor unchanged — no packing indices and no block-diagonal bias, unlike Qwen3-ASR. Inside the graph the flat window is split into 30 s chunks, encoded to 1500 frames per chunk, grouped four frames at a time (the projector input width is 4 * 1280 = 5120) and projected to the 3072-wide language model embedding space, so 375 audio embeddings per chunk.

The feature extractor is a stock Whisper 128-mel extractor, identical to the whisper128 preprocessor already in onnx-asr, so no in-graph feature extraction was needed. Voxtral is a full model with no LoRA or adapter to merge.

Prompt

mistral-common encodes a transcription request as

<s> [INST] <begin_audio> <audio> * 375 <end_audio> [language tokens] [/INST]

so the ids around the audio are prompt_prefix_ids = [1, 3, 25] and prompt_suffix_ids = [4, 34]. The language marker sits after the audio, where Qwen3-ASR puts it before, so the export writes language_suffix_ids (both the ISO 639-1 code and the English name as keys) for the eight languages Voxtral supports.

Tokenizer

Voxtral uses tekken, a byte-level BPE. MistralCommonBackend.get_vocab() decodes every piece as UTF-8, which is lossy — thousands of byte-level pieces collapse onto the replacement character and the map loses entries. vocab.json is therefore built straight from tekken.json: the 1000 control tokens by name, then every regular token written in the GPT-2 byte-to-unicode encoding that the onnx-asr byte-level decoder reverses exactly. 131072 entries, no collisions.

Runtime change

One commit on feat/speech-llm-voxtral: the optional language_suffix_ids config key, which lets the language marker sit in the prompt suffix. Everything else in the family runtime is unchanged, and models without the key behave exactly as before. Covered by three new parametrized cases in tests/onnx_asr/test_speech_llm.py (11 tests pass).

Parity

fp32 ONNX against native transformers fp32 (VoxtralForConditionalGeneration, greedy), four FLEURS validation clips:

Clip fp32 ONNX vs native
en_1 (6.5 s) identical
en_2 (16.4 s) identical
pt_1 (11.8 s) identical
pt_2 (14.6 s) identical

4/4 character-for-character.

int8 differs on two clips, in both cases by a word that is at least as good as the fp32 output:

  • en_2: "Japanese cuisines feature an array" (fp32: "cuisines features an array").
  • pt_1: "que orienta as pesquisas científicas" (fp32: "que orienta e pesquisa científica").
  • pt_2: "Esquiar" (fp32: "Esquiá"), and the quotes around ski bums are dropped.

Speed

Hardware: AMD Ryzen 5 7600 (6 cores / 12 threads), 124 GB RAM, CPU only, onnxruntime 1.28, box shared with other jobs.

Clip audio fp32 RTF int8 RTF native torch fp32 RTF
en_1 6.54 s 8.30 1.06 3.48
en_2 16.38 s 4.19 0.77 2.39
pt_1 11.82 s 2.07 2.32 2.14
pt_2 14.58 s 2.83 1.94 2.54

The first clip of every run carries the warm-up. Session load takes 222 s (fp32) and 122 s (int8) from a cold page cache. int8 is the variant to use on CPU.

Sizes

Graph fp32 int8
encoder 2.65 GB 0.67 GB
embed_tokens 1.61 GB 0.40 GB
decoder 14.45 GB 3.61 GB
total 18.7 GB 4.7 GB

encoder.onnx, decoder.onnx and decoder_int8.onnx keep their weights in a side .onnx_data file.

Limitations

  • One 30 s window per call, as for every Whisper-preprocessor model in onnx-asr. Segment longer audio yourself. Voxtral itself handles up to 30 minutes.
  • Greedy decoding only, one waveform at a time — both are properties of the family runtime.
  • Only the transcription request is exported. Voxtral's audio understanding and chat modes are not reachable through this contract.

Reproduce

export_voxtral.py, verify_native.py and verify_onnx.py are in this repository.

python export_voxtral.py --out <dir>
python verify_native.py
python verify_onnx.py fp32 int8