---
language:
  - en
license: apache-2.0
library_name: transformers
base_model: zhihan1996/DNABERT-2-117M
tags:
  - genomics
  - virology
  - dnabert
  - foundation-model
  - hvilm
  - viral-genomics
  - pathogenicity
  - transmissibility
  - host-tropism
  - hvue-v2
datasets:
  - duttaprat/HVUE-v2
pipeline_tag: feature-extraction
widget:
  - text: "ATGCGTACGTTAGCCGATCG"
    example_title: "Virus sequence example"
---

# HViLM-base: A Foundation Model for Viral Genomics

<div align="center">

[![Preprint](https://img.shields.io/badge/bioRxiv-2026-B31B1B)](https://www.biorxiv.org/content/10.64898/2026.03.18.712700v1)
[![Code](https://img.shields.io/badge/Code-GitHub-black)](https://github.com/duttaprat/HViLM)
[![Dataset](https://img.shields.io/badge/Dataset-HVUE--v2-yellow)](https://huggingface.co/datasets/duttaprat/HVUE-v2)
[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](LICENSE)

</div>

> [!IMPORTANT]
> **HVUE v2 supersedes the original HVUE benchmark.**  
> The original HVUE v1 benchmark contained substantial cross-split sequence similarity that could inflate held-out performance estimates. HVUE v2 was rebuilt using source-sequence clustering **before** train/validation/test assignment and chunking, followed by exact- and near-match leakage auditing.  
> **Use [duttaprat/HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2) for current HViLM evaluation.**

## Model Description

**HViLM (Human Virome Language Model)** is a genomic foundation model adapted to virus sequences through continued pre-training of [DNABERT-2](https://huggingface.co/zhihan1996/DNABERT-2-117M). HViLM-base was trained on approximately **5 million non-redundant virus-derived sequence fragments** from the [VIRION](https://virion.verena.org/) resource, representing approximately 9,000 virus species across 45+ families.

The architecture and tokenizer remain those of DNABERT-2; continued pre-training updates the model weights using a masked-language-modeling objective on the virus-focused corpus.

**Preprint:** *HViLM: A Foundation Model for Viral Genomics Enables Multi-Task Prediction of Pathogenicity, Transmissibility, and Host Tropism*  
[bioRxiv 2026.03.18.712700](https://www.biorxiv.org/content/10.64898/2026.03.18.712700v1)

**Authors:** Pratik Dutta, Jack Vaska, Pallavi Surana, Rekha Sathian, Max Chao, Zhihan Zhou, Han Liu, and Ramana V. Davuluri

**Code:** [github.com/duttaprat/HViLM](https://github.com/duttaprat/HViLM)

---

## HViLM Model Family

HViLM-base is the continued-pretrained foundation model. Official task-specific models fine-tuned on HVUE v2 are released as standalone checkpoints:

| Resource | Purpose |
|---|---|
| [HViLM-base](https://huggingface.co/duttaprat/HViLM-base) | Continued-pretrained foundation model / sequence representations |
| [HViLM-Patho](https://huggingface.co/duttaprat/HViLM-Patho) | Pathogenicity classification |
| [HViLM-R0](https://huggingface.co/duttaprat/HViLM-R0) | Transmissibility classification |
| [HViLM-Tropism](https://huggingface.co/duttaprat/HViLM-Tropism) | Human host-tropism classification |
| [HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2) | Leakage-controlled benchmark |

The complete project is also grouped in the **HViLM: Human Virome Language Model** collection on the [duttaprat Collections page](https://huggingface.co/duttaprat/collections).

---

## Key Features

- **Virus-focused continued pre-training:** approximately 5M non-redundant fragments derived from VIRION-linked virus sequences.
- **DNABERT-2 initialization:** preserves the DNABERT-2 architecture and BPE tokenizer while adapting model weights to virus sequence data.
- **Three official downstream models:** pathogenicity, transmissibility, and host tropism.
- **HVUE v2 evaluation:** cluster-aware splitting before chunking, with multiple similarity stringencies and sequence lengths.
- **Parameter-efficient downstream adaptation:** official task models were trained with LoRA.
- **Public reproducibility resources:** base model, three task-specific checkpoints, HVUE v2 benchmark, and project code are released publicly.

---

## Model Architecture and Continued Pre-training

HViLM-base is derived from **DNABERT-2 (117M parameters)**.

| Property | Value |
|---|---|
| Architecture | MosaicBERT / DNABERT-2 |
| Parameters | ~117M |
| Hidden size | 768 |
| Transformer layers | 12 |
| Attention heads | 12 |
| Tokenization | Byte Pair Encoding (BPE) |
| Positional method | ALiBi |
| Continued-pretraining objective | Masked Language Modeling |
| Pretraining fragment length | 1000 nt |
| Final virus-focused corpus | ~5M non-redundant fragments |
| Redundancy reduction | MMseqs2 clustering at 80% identity / 80% coverage |
| Optimizer | AdamW |
| Learning rate | 5e-5 |
| Training | 10 epochs |
| Hardware | 4 × NVIDIA A100 GPUs |
| Approximate training time | 72 hours |
| Held-out MLM accuracy | 94.2% |

**Sequence-length note:** HViLM uses BPE tokenization, so nucleotide length and model-token length are not equivalent. The continued-pretraining corpus used 1000-nt sequence fragments; downstream configurations are described by nucleotide length in HVUE v2.

---

## Quick Start

### Extract sequence representations from HViLM-base

```python
import torch
from transformers import AutoTokenizer, AutoModel

model_id = "duttaprat/HViLM-base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
)

sequence = "ATGCGTACGTTAGCCGATCGATTACGCGTACGTAGCTAGCTAGCT"

inputs = tokenizer(
    sequence,
    return_tensors="pt",
    truncation=True,
    padding=True,
)

with torch.no_grad():
    outputs = model(**inputs)

token_embeddings = outputs.last_hidden_state
print(token_embeddings.shape)
```

For sequence-level representations, pooling strategy should be chosen according to the downstream task rather than treated as a fixed property of HViLM-base.

---

## Use the Official Fine-tuned Models

If the goal is one of the three HVUE v2 tasks, users can load the corresponding task model directly; `HViLM-base` does not need to be loaded separately.

### Pathogenicity

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "duttaprat/HViLM-Patho"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    trust_remote_code=True,
)
```

Labels:

- `0` → `NON_PATHOGENIC`
- `1` → `PATHOGENIC`

### Transmissibility

Use:

```text
duttaprat/HViLM-R0
```

Labels:

- `0` → `R0_LT_1`
- `1` → `R0_GE_1`

### Host Tropism

Use:

```text
duttaprat/HViLM-Tropism
```

Labels:

- `0` → `NON_HUMAN_TROPIC`
- `1` → `HUMAN_TROPIC`

See the individual model cards for full usage examples and task-specific limitations.

---

## HVUE v2 Benchmark

[HVUE v2](https://huggingface.co/datasets/duttaprat/HVUE-v2) is the current evaluation benchmark for HViLM. It replaces HVUE v1.

The benchmark was reconstructed to reduce supervised train-test leakage using the following ordering:

1. consolidate and deduplicate source sequences;
2. cluster source sequences with MMseqs2;
3. assign complete clusters to train/validation/test splits;
4. chunk sequences only **after** split assignment;
5. remove exact duplicate chunks;
6. audit cross-split exact and near matches.

HVUE v2 includes:

- **Pathogenicity**
- **Transmissibility**
- **Host Tropism**

Across the benchmark, configurations evaluate different sequence lengths (500, 1000, and 2000 nt where applicable), sequence-similarity stringencies, and temporal generalization where reliable collection-date metadata are available.

### Primary HViLM Results

The primary results below use the **standard 1000-nt configuration** for each task.

| Task | HVUE v2 configuration | Accuracy | F1 | MCC | Official model |
|---|---|---:|---:|---:|---|
| Pathogenicity | `standard_capped_1000bp` | **92.39** | **91.32** | **83.10** | [HViLM-Patho](https://huggingface.co/duttaprat/HViLM-Patho) |
| Transmissibility | `standard_capped_1000bp` | **87.50** | **86.16** | **72.66** | [HViLM-R0](https://huggingface.co/duttaprat/HViLM-R0) |
| Host Tropism | `standard_95_1000bp` | **96.49** | **74.49** | **48.99** | [HViLM-Tropism](https://huggingface.co/duttaprat/HViLM-Tropism) |

The directory/configuration identifiers retain `bp` for release stability; manuscript and descriptive text use **nt** for nucleotide sequence length.

### Interpretation of the HVUE v2 Results

Under leakage-controlled evaluation, the effect of virus-focused continued pre-training is **task dependent**:

- **Pathogenicity:** HViLM improves F1 by 1.28 points over vanilla DNABERT-2 (91.32 vs. 90.04).
- **Transmissibility:** HViLM and DNABERT-2 are close (86.16 vs. 85.81 F1), and HViLM is essentially tied with DNABERT-MB (86.16 vs. 86.15 F1).
- **Host Tropism:** HViLM shows the largest F1 improvement, reaching 74.49 compared with 64.82 for class-balanced DNABERT-2.

These results support a more specific conclusion than the original HVUE v1 evaluation: virus-focused continued pre-training provides its clearest benefit on the more challenging Host Tropism task, while gains on Pathogenicity and Transmissibility are smaller.

For complete baseline comparisons, hard-split evaluations, temporal evaluations, and sequence-length analyses, see the [HViLM GitHub repository](https://github.com/duttaprat/HViLM) and [HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2).

---

## Training Data

### Continued-pretraining corpus

HViLM-base was trained using virus sequences associated with the **VIRION** resource.

Processing included:

- retrieval and quality control of VIRION-linked nucleotide sequences;
- removal of short sequences and exact duplicates;
- segmentation into non-overlapping 1000-nt fragments;
- MMseqs2 clustering at 80% sequence identity and 80% coverage;
- selection of approximately 5M representative fragments for continued pre-training.

The corpus spans approximately 9,000 virus species and 45+ virus families across the Baltimore classification groups.

---

## Interpretability

Attention-guided analyses associated with the HViLM study identified **candidate sequence motifs** in pathogenic coronavirus sequences, including motifs with similarity to vertebrate transcription-factor binding motifs such as IRF1, FOXQ1, and ZNF354A.

These observations are **hypothesis-generating**. Sequence similarity between virus motifs and host transcription-factor binding motifs does not by itself establish molecular mimicry, causal regulation, immune evasion, or another biological mechanism. Experimental validation and additional controls are required for mechanistic interpretation.

---

## Limitations

- HVUE v2 controls supervised split leakage through source-level clustering and auditing, but sequence-similarity thresholds cannot eliminate every form of biological relatedness.
- The complete historical training exposure of the original DNABERT-2 model cannot be reconstructed; therefore, absence of all possible ancestral pretraining exposure to benchmark-related sequences cannot be guaranteed.
- Host association is biologically context-dependent and may include multi-host, zoonotic, and reverse-zoonotic relationships; the benchmark uses a simplified binary formulation.
- R₀-based transmissibility labels simplify a continuous, context-dependent epidemiological quantity into a binary benchmark task.
- Performance differences between closely matched models should not be interpreted as statistically meaningful without uncertainty estimates or repeated evaluations.
- Attention-based motif analyses should be considered exploratory rather than direct evidence of mechanism.
- HViLM predictions are research outputs and are not intended to replace experimental, clinical, epidemiological, or public-health assessment.

---

## Reproducibility and Resources

- **Base model:** [duttaprat/HViLM-base](https://huggingface.co/duttaprat/HViLM-base)
- **Pathogenicity model:** [duttaprat/HViLM-Patho](https://huggingface.co/duttaprat/HViLM-Patho)
- **Transmissibility model:** [duttaprat/HViLM-R0](https://huggingface.co/duttaprat/HViLM-R0)
- **Host Tropism model:** [duttaprat/HViLM-Tropism](https://huggingface.co/duttaprat/HViLM-Tropism)
- **Benchmark:** [duttaprat/HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2)
- **Code:** [github.com/duttaprat/HViLM](https://github.com/duttaprat/HViLM)
- **Collections:** [duttaprat's Hugging Face Collections](https://huggingface.co/duttaprat/collections)

---

## Citation

If you use HViLM in your research, please cite:

```bibtex
@article{dutta2026hvilm,
  title={HViLM: A foundation model for viral genomics enables multi-task prediction of pathogenicity, transmissibility, and host tropism},
  author={Dutta, Pratik and Vaska, Jack and Surana, Pallavi and Sathian, Rekha and Chao, Max and Zhou, Zhihan and Liu, Han and Davuluri, Ramana V},
  journal={bioRxiv},
  pages={2026--03},
  year={2026},
  publisher={Cold Spring Harbor Laboratory}
}
```

If you use DNABERT-2 directly or build on its architecture, please also cite the DNABERT-2 publication.

---

## Model Card Authors

- **Pratik Dutta** — Stony Brook University
- **Ramana V. Davuluri** — Stony Brook University

---

## Contact

- **GitHub Issues:** [github.com/duttaprat/HViLM/issues](https://github.com/duttaprat/HViLM/issues)
- **Lab:** [Davuluri Lab, Stony Brook University](https://davulurilab.github.io/)

---

## Acknowledgments

HViLM builds on DNABERT-2 by Zhou et al. Continued-pretraining data were derived from the VIRION resource maintained by the Viral Emergence Research Initiative (Verena).

---

## License

HViLM-base is released under the **Apache License 2.0**.

---

## Disclaimer

HViLM is a research model for computational biology. It should not be used as the sole basis for clinical, diagnostic, epidemiological, biosurveillance, or public-health decisions. Model outputs should be interpreted alongside appropriate biological evidence and expert assessment.