mdeberta-v3-wanli-ja-nli

Fine-tuned microsoft/mdeberta-v3-base on a Japanese wanli-ja-nli dataset for natural language inference.

• Task: natural language inference (Zero-Shot Classification)
• Language: Japanese
• License: CC BY 4.0
• Base model: microsoft/mdeberta-v3-base

Usage

from transformers import pipeline

model_id = "takehika/mdeberta-v3-wanli-ja-nli"
classifier = pipeline("zero-shot-classification", model=model_id)

text = "市議会は二酸化炭素排出量を削減し、都市公園を保護するための新しい環境イニシアチブを実施しています。"
labels = ["天気", "環境", "娯楽", "経済", "政治"]

output = classifier(text, labels, multi_label=False)
print(output)

Data

• Dataset: wanli-ja-nli (takehika/wanli-ja-nli, ja_only)
• Train data: 73,942 examples
• Validation data: 3,505 examples (test split)

Training

• Base: microsoft/mdeberta-v3-base
• Epochs: 2
• Learning rate: 1e-5
• Warmup ratio: 0.1
• Batch size: 4 per device, grad accumulation 2
• Best model selection: f1

Evaluation

• Accuracy: 0.7506
• F1: 0.7497

Intended Use & Limitations

• Intended for Japanese NLI.
• Inputs are limited to 512 tokens (longer pairs are truncated).
• Domain shifts or adversarial examples can reduce performance.

Attribution & Licenses

• License: CC BY 4.0
• Base model: microsoft/mdeberta-v3-base by Microsoft - MIT License
  • Model card: https://huggingface.co/microsoft/mdeberta-v3-base
• Dataset: wanli-ja-nli (takehika/wanli-ja-nli) - CC BY 4.0
  • Dataset card: https://huggingface.co/datasets/takehika/wanli-ja-nli

This model modifies the base model by fine-tuning on the above dataset.

Base Model Citation

@misc{he2021debertav3,
      title={DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing}, 
      author={Pengcheng He and Jianfeng Gao and Weizhu Chen},
      year={2021},
      eprint={2111.09543},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}