---
license: apache-2.0
base_model: answerdotai/ModernBERT-base
library_name: transformers
pipeline_tag: text-classification
language:
- en
tags:
- log-analysis
- severity-classification
- modernbert
- aiops
- observability
metrics:
- accuracy
- f1
model-index:
- name: modernlogbert-wce
  results:
  - task:
      type: text-classification
      name: Log severity classification
    metrics:
    - type: accuracy
      value: 0.8818
      name: Accuracy (in-distribution)
    - type: f1
      value: 0.7813
      name: Macro F1 (in-distribution)
---

# ModernLogBERT (WCE)

A [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) encoder
fine-tuned to classify the **severity level** of a single log line into one of
six levels: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`.

This checkpoint is trained with a **Weighted Cross-Entropy (WCE)** objective.
A sibling checkpoint trained with Weighted Generalized Cross-Entropy (a
noise-tolerant loss) is at
[hazemkhaled-94/modernlogbert-gce](https://huggingface.co/hazemkhaled-94/modernlogbert-gce).

Built with the [log-lens](https://github.com/hazemkhaled-94/deep-learning)
project, which also provides the **Drain3 preprocessing pipeline** these inputs
require (see "How to use").

## Intended use

- **Intended:** triage and observability research — predicting or
  sanity-checking log severity, and flagging entries whose predicted severity
  disagrees with the emitted level as candidate anomalies.
- **Out of scope:** a sole source of truth for alerting or incident severity.
  Aggregate accuracy hides brittle behavior on unfamiliar log formats — keep a
  human in the loop.

## How to use

Inputs must be **Drain3-masked the same way as in training** (variables
replaced by placeholders such as `<NUM>`, `<IP>`, `<UUID>`); raw text degrades
predictions. The [log-lens](https://github.com/hazemkhaled-94/deep-learning)
repo ships a ready-to-use Drain3 preprocessing pipeline that produces exactly
this masked form.

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "hazemkhaled-94/modernlogbert-wce"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

text = "Connection refused after <<NUM>> retries"  # Drain3-masked input
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    pred = model(**inputs).logits.argmax(-1).item()
print(model.config.id2label[pred])
```

## Training data

- **In-distribution (train + held-out eval):** a publicly available collection
  of system log corpora ([loghub](https://github.com/logpai/loghub)),
  preprocessed into a level-balanced, stratified sample.
- **Out-of-distribution (evaluation only):** a single private industrial
  Kubernetes log deployment. Not released; used purely as an OOD
  generalization probe.

## Training procedure

| Hyperparameter | Value |
|---|---|
| Backbone | ModernBERT-base |
| Loss | Weighted Cross-Entropy |
| Epochs | 8 |
| Per-device batch size | 32 |
| Gradient accumulation | 4 (effective batch 128) |
| Learning rate | 1e-5 (separate LRs for head vs backbone) |
| Weight decay | 0.01 |
| Warmup ratio | 0.1 |
| Max sequence length | 512 |
| Best-model metric | macro F1 |

## Evaluation

### In-distribution (held-out stratified slice)

| Metric | Value |
|---|---|
| Accuracy | 88.18% |
| Macro precision | 0.7647 |
| Macro recall | 0.8271 |
| Macro F1 | 0.7813 |
| Weighted F1 | 0.8947 |

On the curated in-distribution slice this WCE checkpoint is the stronger and
better-calibrated model (vs the WGCE sibling).

### Out-of-distribution

Evaluated on a private industrial Kubernetes domain — a different log
distribution than training. Performance **degraded modestly but stayed
usable**, the expected cost of moving to unfamiliar formats. As always for OOD
use, validate on your own log distribution before relying on it.

## Limitations and biases

- **OOD generalization** — only modest degradation was observed on a single
  private industrial domain; other distributions are unverified, so validate on
  your own logs.
- **Confidence ≠ correctness** — treat scores as signals, not guarantees.
- **Preprocessing coupling** — inputs must be Drain3-masked exactly as in
  training (use the log-lens preprocessing pipeline).