license: apache-2.0
base_model: answerdotai/ModernBERT-base
library_name: transformers
pipeline_tag: text-classification
language:
- en
tags:
- log-analysis
- severity-classification
- modernbert
- aiops
- observability
metrics:
- accuracy
- f1
model-index:
- name: modernlogbert-wce
results:
- task:
type: text-classification
name: Log severity classification
metrics:
- type: accuracy
value: 0.8818
name: Accuracy (in-distribution)
- type: f1
value: 0.7813
name: Macro F1 (in-distribution)
ModernLogBERT (WCE)
A ModernBERT encoder
fine-tuned to classify the severity level of a single log line into one of
six levels: TRACE, DEBUG, INFO, WARN, ERROR, FATAL.
This checkpoint is trained with a Weighted Cross-Entropy (WCE) objective. A sibling checkpoint trained with Weighted Generalized Cross-Entropy (a noise-tolerant loss) is at hazemkhaled-94/modernlogbert-gce.
Built with the log-lens project, which also provides the Drain3 preprocessing pipeline these inputs require (see "How to use").
Intended use
- Intended: triage and observability research — predicting or sanity-checking log severity, and flagging entries whose predicted severity disagrees with the emitted level as candidate anomalies.
- Out of scope: a sole source of truth for alerting or incident severity. Aggregate accuracy hides brittle behavior on unfamiliar log formats — keep a human in the loop.
How to use
Inputs must be Drain3-masked the same way as in training (variables
replaced by placeholders such as <NUM>, <IP>, <UUID>); raw text degrades
predictions. The log-lens
repo ships a ready-to-use Drain3 preprocessing pipeline that produces exactly
this masked form.
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
repo = "hazemkhaled-94/modernlogbert-wce"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
text = "Connection refused after <<NUM>> retries" # Drain3-masked input
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
pred = model(**inputs).logits.argmax(-1).item()
print(model.config.id2label[pred])
Training data
- In-distribution (train + held-out eval): a publicly available collection of system log corpora (loghub), preprocessed into a level-balanced, stratified sample.
- Out-of-distribution (evaluation only): a single private industrial Kubernetes log deployment. Not released; used purely as an OOD generalization probe.
Training procedure
| Hyperparameter | Value |
|---|---|
| Backbone | ModernBERT-base |
| Loss | Weighted Cross-Entropy |
| Epochs | 8 |
| Per-device batch size | 32 |
| Gradient accumulation | 4 (effective batch 128) |
| Learning rate | 1e-5 (separate LRs for head vs backbone) |
| Weight decay | 0.01 |
| Warmup ratio | 0.1 |
| Max sequence length | 512 |
| Best-model metric | macro F1 |
Evaluation
In-distribution (held-out stratified slice)
| Metric | Value |
|---|---|
| Accuracy | 88.18% |
| Macro precision | 0.7647 |
| Macro recall | 0.8271 |
| Macro F1 | 0.7813 |
| Weighted F1 | 0.8947 |
On the curated in-distribution slice this WCE checkpoint is the stronger and better-calibrated model (vs the WGCE sibling).
Out-of-distribution
Evaluated on a private industrial Kubernetes domain — a different log distribution than training. Performance degraded modestly but stayed usable, the expected cost of moving to unfamiliar formats. As always for OOD use, validate on your own log distribution before relying on it.
Limitations and biases
- OOD generalization — only modest degradation was observed on a single private industrial domain; other distributions are unverified, so validate on your own logs.
- Confidence ≠ correctness — treat scores as signals, not guarantees.
- Preprocessing coupling — inputs must be Drain3-masked exactly as in training (use the log-lens preprocessing pipeline).