Datk4/vifeedback-sentiment-phobert

ViFeedback sentiment: PhoBERT-base, robust to missing diacritics and real informal typing

Sentiment (negative / neutral / positive) for Vietnamese student feedback about university courses. PhoBERT-base fine-tuned on UIT-VSFC with diacritic and teencode augmentation and, at every step, trained to give a real informally typed comment the label a frozen teacher gives its normalized form. Real typing changes 12% of its labels, against 18% for the same model without that training. Shipped as the ONNX graph with its weights stored in FP16 and computed in FP32 (ADR-039) the ViFeedback service runs, plus the PyTorch checkpoint it was exported from. Its vocabulary is cut to the 17,500 entries the service's text uses, so the graph is 198.9 MB (S7b, ADR-045).

Code, evaluation protocol and every number below: https://github.com/ThanhDatVN/ViFeedback-NLP-Service

Input contract

The model expects pyvi word-segmented, lowercase text (giảng_viên nhiệt_tình), at most 96 tokens.

import numpy as np, onnxruntime as ort
from pyvi import ViTokenizer
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Datk4/vifeedback-sentiment-phobert")
sess = ort.InferenceSession("model.fp16.onnx")  # downloaded from this repository
text = ViTokenizer.tokenize("thầy dạy rất dễ hiểu".lower())
enc = tok([text], truncation=True, max_length=96, return_tensors="np")
logits = sess.run(None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0]
print(['negative', 'neutral', 'positive'][int(logits.argmax())])

The service divides the logits by T = 1.35 (fitted on validation) before the softmax, so the probabilities it reports are calibrated; do the same with the raw graph.

Optional parts of the release

Each is declared in manifest.json with a SHA-256; the service refuses to start on a mismatch.

• restorer.json: a diacritic restorer (word-bigram Viterbi over UIT-VSFC train text). The service applies it after lowercasing and before segmentation, and only to essentially unaccented sentences (accented share below 0.091). Validation with the diacritics stripped: macro-F1 0.658 → 0.846; clean validation labels changed: 0. Load it with vifeedback.preprocess.diacritics.Restorer.
• scope.npz: a topic-aware scope detector (ADR-034): logistic regression on TF-IDF word unigrams and bigrams of the model input (70,377 terms), trained to tell course-related student text (UIT-VSFC, and in-scope posts of another university's forum, NEU-ESC) from spam, news, job and club posts. Below -0.177 the input is out of scope. On NEU-ESC test it separates in-scope from off-topic posts at AUROC 0.922, flags 7% of in-scope posts and catches 72% of off-topic ones; it flags 0.2% of UIT-VSFC validation. Served by a numpy re-implementation checked equal to scikit-learn. The service reports it as in_scope and scope_score next to the label and never refuses an input.

Training

• Base. vinai/phobert-base at 01daacda68af; 4 epochs, lr 2e-5, batch 32, max length 96, seed 42; the epoch selected on validation macro-F1.
• Augmentation. 30% of the UIT-VSFC training sentences replaced by a perturbed copy (half their diacritics removed, or common teencode substituted) and re-segmented.
• Anchored consistency (ADR-042). At every step, a batch of 6,035 real social-media comments from ViLexNorm, each paired with its human normalization. The loss adds KL(q ‖ p(comment)), where q is the probability a frozen 5-seed ensemble of the model without this training gives the normalized form. The target is fixed and diverse, so the model cannot satisfy it by predicting one class, the failure of an earlier attempt that used its own prediction as the target (ADR-038).

It passed a rule declared before it was trained (configs/experiments/cycle5.yaml v4, five seeds each side, 1,500 held-out comment pairs):

• agreement with the teacher's label on the normalized form +0.061 [+0.048, +0.075];
• label flips between a comment and its normalization -0.061 [-0.073, -0.049];
• label-distribution distance 0.013 (limit 0.10);
• and no loss on UIT-VSFC (+0.0014), its neutral class, stripped text or another university's posts (NEU-ESC, -0.0039).

Evaluation

Macro-F1 is the headline because neutral is 4% of the data and carries a third of the average.

Split	Macro-F1	Negative F1	Neutral F1	Positive F1
Validation (1,583)	0.8617	0.963	0.655	0.966
Test (3,166), evaluated once	0.8208	0.958	0.550	0.954
Test, this graph (evaluated once)	0.8198	0.957	0.548	0.954

Test macro-F1 over five seeds of this recipe: 0.8208, 0.8275, 0.8267, 0.8142, 0.8294. Against five seeds of the recipe without the consistency training, the difference is -0.0059, 95% interval [-0.0193, +0.0074]: not distinguishable. (The seed-42 model without it happened to score 0.8371 on test.)

The graph and the PyTorch model with the same FP16-rounded weights agree on every validation label (max logit difference 1.8e-05); a sentence scores the same alone and inside a padded batch.

Robustness (test, macro-F1, without the restorer): no diacritics 0.623; half the diacritics 0.722; teencode 0.810; 5% character noise 0.781. The service restores diacritics first: validation with every diacritic stripped scores 0.846 through it.

Challenge set (305 constructed sentences, not from UIT-VSFC; accuracy per category, the model alone):

Category	n	Accuracy
code_switch	25	0.960
long_context	20	0.850
mixed_aspect	40	0.925
negation_pair	30	0.867
objective_neutral	30	0.867
suggestion_cue	25	1.000
suggestion_implicit	25	0.840
teencode_typed	40	1.000
unaccented_typed	50	0.580

Through the service's own pipeline (lowercase, diacritic restorer, pyvi) the same sentences score macro-F1 0.916 (neutral F1 0.885), against 0.851 (0.771) for the model without the consistency training.

Size: a vocabulary cut to the text it serves

PhoBERT's vocabulary has 64,001 entries; this release keeps 17,500: the 15,318 pieces that UIT-VSFC train (with its unaccented and teencode forms), NEU-ESC train and the ViLexNorm training pairs use, then the most frequent others. No other weight changes. With the weights stored in FP16 the graph is 198.9 MB instead of 540 MB. A piece outside the vocabulary reads as <unk>. Against the full-vocabulary FP32 graph of the same weights, on text none of the vocabulary came from:

Held-out set	n	Same label	Texts with a removed piece
uit_validation	1,583	100.00%	1.5%
uit_validation_stripped	1,583	100.00%	1.1%
neu_esc_validation	3,305	99.52%	8.3%
vilexnorm_dev_original	837	99.04%	9.7%
vilexnorm_dev_normalized	837	99.64%	9.8%
vilexnorm_confirm_original	1,500	99.53%	7.7%
vilexnorm_confirm_normalized	1,500	99.60%	6.4%
challenge_v1	285	100.00%	2.1%

On UIT-VSFC test, scored once, it gives the same label as the full graph on 99.94% of 3,166 sentences.

Limitations

• Neutral is weak (test F1 0.550). Neutral errors are confident, so thresholds do not fix them.
• Real informal typing still changes labels, less often: about 12% of labels between a social-media comment and its human normalization. Those comments come from social media (ViLexNorm), not from course feedback.
• Real student text from another university is much harder (0.43 macro-F1 on NEU-ESC validation). Forum posts are mostly non-evaluative; the model was trained on course surveys.
• Input is lowercased before scoring, as the training data is.
• Off-topic input still gets a label; in_scope (above) is the signal to act on.
• One training domain. Student feedback from one Vietnamese university.
• The suggestion convention is the corpus's: a request for change (thầy nên…) is negative.
• The vocabulary is cut. A word made only of pieces outside it reads as <unk>; text far from student feedback and social media (other domains, rare names) meets this more often.

Files and checksums (SHA-256)

File	SHA-256
added_tokens.json	c65ec8a1007f65ff52b8143f045864e3f9c4050e6473f48bb4c0ad1b8caa6366
bpe.codes	9dfe53fb8abbddd3387da99641c36e00bee757a3679aad9cf0a7cdc4b30f6ef8
manifest.json	19083ddcaaf7f7a313c1b8c9ecebbf8c5b527fa2f1c56ba56453a26089e48e2c
model.fp16.onnx	27fda01b71a1d2867fe2d1565c2e87022dcd21966e2c0f333f17948a48d1163a
pytorch/added_tokens.json	b2f16ff3cea9d87837d923040a26d439c0abd947e63d9caa18c17ac6d883b5d2
pytorch/bpe.codes	9dfe53fb8abbddd3387da99641c36e00bee757a3679aad9cf0a7cdc4b30f6ef8
pytorch/config.json	54b9c1383b1c6afe34781642fb4295740de9a086b79167118aa1a45518e38040
pytorch/model.safetensors	3b8c508556286ba5a60e24afd61f4567a70d5d9e64861df92a3f4c7d9e608b10
pytorch/tokenizer_config.json	feb4e99754d3fc39608371fa3ee617d9cc2f83355078ce5e4e4a6c6cbae13bfa
pytorch/trim.json	59a3b07d450b83d824346c88d1371764983f4276297300936389abaed7406af8
pytorch/vocab.txt	7095e7dd8728cdafca2b69702fda528dbaf4d70cf18e03e736abb58c046c4ee8
restorer.json	7ed4a90368905eb6d532d795ced2f138c3a03434200e47d7b31cdf4b4f63c018
scope.npz	d3c62672b10fb47cfcf3ff0830dc64199a6a59a7614513f9849169d59f22ca82
tokenizer_config.json	05c5a37124662608e5700a10f514ceb1a9953d4286e2d36f7fbde75a292223b7
training_config.yaml	8b5645e17e02bd03838150f008a66cc4b8ce93ecd8ac06f9a08eaececbfc8b58
vocab.txt	7095e7dd8728cdafca2b69702fda528dbaf4d70cf18e03e736abb58c046c4ee8

Release manifest: manifest.json (acceptance on the full validation set). Training run: p15-sent-phobert-base-seg_pyvi-h10b-anchored_orig-s42-063770c8.

Intended use

Research and experiments on Vietnamese student feedback about courses: aggregate sentiment over many comments, error analysis, robustness studies. It is not meant for decisions about an individual student or lecturer, and its labels follow UIT-VSFC's annotation guide (a request for change is negative), which other institutions' annotators may not share.

Licence

Weights: CC BY-NC-SA 4.0, for research and other non-commercial use, shared alike. The model learned from UIT-VSFC (released by its authors for research) and from ViLexNorm comments (CC BY-NC-SA 4.0), and derives from PhoBERT (MIT). The code that produced it is MIT-licensed in the GitHub repository.

Citation

UIT-VSFC: Nguyen et al., 2018. ViLexNorm: Nguyen et al., 2024 (EACL). PhoBERT: Nguyen and Nguyen, 2020. Consistency training: Xie et al., 2020 (UDA).