---
library_name: transformers
tags:
- smiles
- chemistry
- BERT
- molecules
license: mit
datasets:
- fabikru/half-of-chembl-2025-randomized-smiles-cleaned
---

# MolEncoder

MolEncoder is a BERT-based chemical language model pretrained on SMILES strings using masked language modeling (MLM). It was designed to investigate optimal pretraining strategies for molecular representation learning, with a particular focus on masking ratio, dataset size, and model size. It is described in detail in the paper "MolEncoder: Towards Optimal Masked Language Modeling for Molecules". 

## Model Description

- **Architecture:** Encoder-only transformer based on ModernBERT  
- **Parameters:** ~15M  
- **Tokenizer:** Character-level tokenizer covering full SMILES vocabulary  
- **Pretraining Objective:** Masked language modeling with optimized masking ratios (30% found to work best for molecules)  
- **Pretraining Data:** Pretrained on ~1M molecules (half of ChEMBL)  

## Key Findings

- Higher masking ratios (20–60%) outperform the standard 15% used in prior molecular BERT models.  
- Increasing model size or dataset size beyond moderate scales yields no consistent performance benefits and can degrade efficiency.  
- This 15M parameter model pretrained on ~1M molecules outperforms much larger models pretrained on more SMILES strings.  

## Intended Uses

- **Primary use:** Molecular property prediction through fine-tuning on downstream datasets   

## How to Use

Please refer to the [MolEncoder GitHub repository](https://github.com/FabianKruger/MolEncoder) for detailed instructions and ready-to-use examples on fine-tuning the model on custom data and running predictions.  

## Citation

If you use this model, please cite:
```bibtex
@Article{D5DD00369E,
author ="Krüger, Fabian P. and Österbacka, Nicklas and Kabeshov, Mikhail and Engkvist, Ola and Tetko, Igor",
title  ="MolEncoder: towards optimal masked language modeling for molecules",
journal  ="Digital Discovery",
year  ="2025",
pages  ="-",
publisher  ="RSC",
doi  ="10.1039/D5DD00369E",
url  ="http://dx.doi.org/10.1039/D5DD00369E"}
```