WhisperLv3-FT-EP-CPP

Fine-tuned Whisper large-v3 model for European Portuguese automatic speech recognition using CAMOES (Capitalized, Punctuated and Pós-Acordo).

Installation

pip install -U torch transformers accelerate soundfile

Usage

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "inesc-id/WhisperLv3-FT-EP-CPP"

# Use the base Whisper large-v3 processor.
# This is needed because the fine-tuned model repo may not include processor files.
processor_id = "openai/whisper-large-v3"

device = 0 if torch.cuda.is_available() else -1
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

processor = AutoProcessor.from_pretrained(processor_id)

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch_dtype,
    low_cpu_mem_usage=True,
    use_safetensors=True,
)

if torch.cuda.is_available():
    model.to("cuda")

forced_decoder_ids = processor.get_decoder_prompt_ids(
    language="portuguese",
    task="transcribe",
)

asr = pipeline(
    task="automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    torch_dtype=torch_dtype,
    device=device,
    generate_kwargs={
        "forced_decoder_ids": forced_decoder_ids,
        "num_beams": 1,
    },
)

result = asr("audio.wav")
print(result["text"])

Important Processor Note

Load the model from:

Miamoto/WhisperLv3-FT-EP-CPP

but load the processor from:

openai/whisper-large-v3

This is because the fine-tuned model repository may not include files such as preprocessor_config.json.

Intended Use

This model is intended for:

• European Portuguese speech-to-text
• ASR research
• ASR evaluation
• Speech transcription demos and prototypes

Citation

BibTeX:

@inproceedings{camoes,
    title={{CAMÕES: A Comprehensive Automatic Speech Recognition Benchmark for  European Portuguese}},
    author={Carlos Carvalho, Francisco Teixeira, Catarina Botelho, Anna Pompili, Rubén Solera-Ureña, Sérgio Paulo, Mariana Julião, Thomas Rolland, John Mendonça, Diogo Pereira, Isabel Trancoso, Alberto Abad},
    booktitle={Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},
    year={2025},
}