MoRoBERTa — XLM-RoBERTa-base continued on Moroccan Arabizi

MoRoBERTa is XLM-RoBERTa-base continue-pretrained with masked language modelling on Moroccan Darija written in Latin script (Arabizi, with chat digits such as 3, 7 and 9). It is the Arabizi encoder of the IANLP 2026 paper that introduced the MoriscoNER corpus, and part of the Aljamiado project for Moroccan Latin-script NLP.

Training

• Corpus: 11,555 unique Arabizi sentences — public YouTube comments on Moroccan content plus the MoriscoNER v1 training sentences (labels ignored), deduplicated, with sentences shorter than three words dropped and every sentence of the MoriscoNER gold test set excluded, so the encoder never saw test text. The comments themselves are not redistributed.
• Recipe: 8 epochs of masked language modelling (15 % masking), sequence length 128, batch 8 × 4 gradient-accumulation steps, learning rate 3e-5 with 10 % warm-up, fp16, on one NVIDIA T4 (June 2026).

Use

Load it as an encoder and fine-tune it, for example for named-entity recognition on MoriscoNER. In the IANLP 2026 paper, MoRoBERTa fine-tuned on 1,400 native sentences reached 0.527 exact span-F1 on the MoriscoNER test set, against 0.464 for XLM-RoBERTa-base (three seeds each, same protocol).

Note on this repository

Until 19 October 2026 this repository held a different model: XLM-RoBERTa-base fine-tuned for NER on transliterated data. That model now lives at AchrafSoltani/xlmr-arabizi-darija-ner.

Citation

@inproceedings{soltani2026morisconer,
  title     = {{MoriscoNER}: A Native {Moroccan-Darija} Latin-Script {NER} Corpus and an Encoder-Conditional Native-vs-Transliteration Study},
  author    = {Soltani, Achraf},
  booktitle = {Proceedings of IANLP 2026},
  series    = {Advances in Computer Science Applications and Research},
  publisher = {Springer},
  year      = {2026},
  note      = {in press}
}