metadata
license: apache-2.0
language:
- en
library_name: transformers
tags:
- sentence-embeddings
- feature-extraction
- sentence-similarity
- spikewhale
pipeline_tag: feature-extraction
model-index:
- name: Byrne-Embed
results:
- task:
type: Classification
name: MTEB AmazonCounterfactualClassification
dataset:
type: mteb/AmazonCounterfactualClassification
name: MTEB AmazonCounterfactualClassification
config: default
split: test
metrics:
- type: accuracy
value: 80.12
- task:
type: Clustering
name: MTEB ArXivHierarchicalClusteringP2P
dataset:
type: mteb/ArXivHierarchicalClusteringP2P
name: MTEB ArXivHierarchicalClusteringP2P
config: default
split: test
metrics:
- type: v_measure
value: 53.15
- task:
type: Clustering
name: MTEB ArXivHierarchicalClusteringS2S
dataset:
type: mteb/ArXivHierarchicalClusteringS2S
name: MTEB ArXivHierarchicalClusteringS2S
config: default
split: test
metrics:
- type: v_measure
value: 50.39
- task:
type: Retrieval
name: MTEB ArguAna
dataset:
type: mteb/ArguAna
name: MTEB ArguAna
config: default
split: test
metrics:
- type: ndcg_at_10
value: 37.67
- task:
type: Reranking
name: MTEB AskUbuntuDupQuestions
dataset:
type: mteb/AskUbuntuDupQuestions
name: MTEB AskUbuntuDupQuestions
config: default
split: test
metrics:
- type: map_at_1000
value: 52.88
- task:
type: STS
name: MTEB BIOSSES
dataset:
type: mteb/BIOSSES
name: MTEB BIOSSES
config: default
split: test
metrics:
- type: cosine_spearman
value: 75.56
- task:
type: Classification
name: MTEB Banking77Classification
dataset:
type: mteb/Banking77Classification
name: MTEB Banking77Classification
config: default
split: test
metrics:
- type: accuracy
value: 74.64
- task:
type: Clustering
name: MTEB BiorxivClusteringP2P.v2
dataset:
type: mteb/BiorxivClusteringP2P.v2
name: MTEB BiorxivClusteringP2P.v2
config: default
split: test
metrics:
- type: v_measure
value: 33.73
- task:
type: Retrieval
name: MTEB CQADupstackGamingRetrieval
dataset:
type: mteb/CQADupstackGamingRetrieval
name: MTEB CQADupstackGamingRetrieval
config: default
split: test
metrics:
- type: ndcg_at_10
value: 37.14
- task:
type: Retrieval
name: MTEB CQADupstackUnixRetrieval
dataset:
type: mteb/CQADupstackUnixRetrieval
name: MTEB CQADupstackUnixRetrieval
config: default
split: test
metrics:
- type: ndcg_at_10
value: 23.48
- task:
type: Retrieval
name: MTEB ClimateFEVERHardNegatives
dataset:
type: mteb/ClimateFEVERHardNegatives
name: MTEB ClimateFEVERHardNegatives
config: default
split: test
metrics:
- type: ndcg_at_10
value: 13.6
- task:
type: Retrieval
name: MTEB FEVERHardNegatives
dataset:
type: mteb/FEVERHardNegatives
name: MTEB FEVERHardNegatives
config: default
split: test
metrics:
- type: ndcg_at_10
value: 28.7
- task:
type: Retrieval
name: MTEB FiQA2018
dataset:
type: mteb/FiQA2018
name: MTEB FiQA2018
config: default
split: test
metrics:
- type: ndcg_at_10
value: 11.38
- task:
type: Retrieval
name: MTEB HotpotQAHardNegatives
dataset:
type: mteb/HotpotQAHardNegatives
name: MTEB HotpotQAHardNegatives
config: default
split: test
metrics:
- type: ndcg_at_10
value: 30.47
- task:
type: Classification
name: MTEB ImdbClassification
dataset:
type: mteb/ImdbClassification
name: MTEB ImdbClassification
config: default
split: test
metrics:
- type: accuracy
value: 60.97
- task:
type: Classification
name: MTEB MTOPDomainClassification
dataset:
type: mteb/MTOPDomainClassification
name: MTEB MTOPDomainClassification
config: default
split: test
metrics:
- type: accuracy
value: 92.29
- task:
type: Classification
name: MTEB MassiveIntentClassification
dataset:
type: mteb/MassiveIntentClassification
name: MTEB MassiveIntentClassification
config: default
split: test
metrics:
- type: accuracy
value: 63.23
- task:
type: Classification
name: MTEB MassiveScenarioClassification
dataset:
type: mteb/MassiveScenarioClassification
name: MTEB MassiveScenarioClassification
config: default
split: test
metrics:
- type: accuracy
value: 73.05
- task:
type: Clustering
name: MTEB MedrxivClusteringP2P.v2
dataset:
type: mteb/MedrxivClusteringP2P.v2
name: MTEB MedrxivClusteringP2P.v2
config: default
split: test
metrics:
- type: v_measure
value: 32.7
- task:
type: Clustering
name: MTEB MedrxivClusteringS2S.v2
dataset:
type: mteb/MedrxivClusteringS2S.v2
name: MTEB MedrxivClusteringS2S.v2
config: default
split: test
metrics:
- type: v_measure
value: 29.04
- task:
type: Reranking
name: MTEB MindSmallReranking
dataset:
type: mteb/MindSmallReranking
name: MTEB MindSmallReranking
config: default
split: test
metrics:
- type: max_over_subqueries_map_at_1000
value: 28.07
- task:
type: Retrieval
name: MTEB SCIDOCS
dataset:
type: mteb/SCIDOCS
name: MTEB SCIDOCS
config: default
split: test
metrics:
- type: ndcg_at_10
value: 10.15
- task:
type: STS
name: MTEB SICK-R
dataset:
type: mteb/SICK-R
name: MTEB SICK-R
config: default
split: test
metrics:
- type: cosine_spearman
value: 69.08
- task:
type: STS
name: MTEB STS12
dataset:
type: mteb/STS12
name: MTEB STS12
config: default
split: test
metrics:
- type: cosine_spearman
value: 64.88
- task:
type: STS
name: MTEB STS13
dataset:
type: mteb/STS13
name: MTEB STS13
config: default
split: test
metrics:
- type: cosine_spearman
value: 72.08
- task:
type: STS
name: MTEB STS14
dataset:
type: mteb/STS14
name: MTEB STS14
config: default
split: test
metrics:
- type: cosine_spearman
value: 67.76
- task:
type: STS
name: MTEB STS15
dataset:
type: mteb/STS15
name: MTEB STS15
config: default
split: test
metrics:
- type: cosine_spearman
value: 77.13
- task:
type: STS
name: MTEB STS17
dataset:
type: mteb/STS17
name: MTEB STS17
config: default
split: test
metrics:
- type: cosine_spearman
value: 83.23
- task:
type: STS
name: MTEB STS22.v2
dataset:
type: mteb/STS22.v2
name: MTEB STS22.v2
config: default
split: test
metrics:
- type: cosine_spearman
value: 60.53
- task:
type: STS
name: MTEB STSBenchmark
dataset:
type: mteb/STSBenchmark
name: MTEB STSBenchmark
config: default
split: test
metrics:
- type: cosine_spearman
value: 77.08
- task:
type: PairClassification
name: MTEB SprintDuplicateQuestions
dataset:
type: mteb/SprintDuplicateQuestions
name: MTEB SprintDuplicateQuestions
config: default
split: test
metrics:
- type: max_ap
value: 86.47
- task:
type: Clustering
name: MTEB StackExchangeClustering.v2
dataset:
type: mteb/StackExchangeClustering.v2
name: MTEB StackExchangeClustering.v2
config: default
split: test
metrics:
- type: v_measure
value: 41.93
- task:
type: Clustering
name: MTEB StackExchangeClusteringP2P.v2
dataset:
type: mteb/StackExchangeClusteringP2P.v2
name: MTEB StackExchangeClusteringP2P.v2
config: default
split: test
metrics:
- type: v_measure
value: 35.22
- task:
type: Summarization
name: MTEB SummEvalSummarization.v2
dataset:
type: mteb/SummEvalSummarization.v2
name: MTEB SummEvalSummarization.v2
config: default
split: test
metrics:
- type: cosine_spearman
value: 22.39
- task:
type: Retrieval
name: MTEB TRECCOVID
dataset:
type: mteb/TRECCOVID
name: MTEB TRECCOVID
config: default
split: test
metrics:
- type: ndcg_at_10
value: 29.3
- task:
type: Retrieval
name: MTEB Touche2020Retrieval.v3
dataset:
type: mteb/Touche2020Retrieval.v3
name: MTEB Touche2020Retrieval.v3
config: default
split: test
metrics:
- type: ndcg_at_10
value: 24.5
- task:
type: Classification
name: MTEB ToxicConversationsClassification
dataset:
type: mteb/ToxicConversationsClassification
name: MTEB ToxicConversationsClassification
config: default
split: test
metrics:
- type: accuracy
value: 62.94
- task:
type: Classification
name: MTEB TweetSentimentExtractionClassification
dataset:
type: mteb/TweetSentimentExtractionClassification
name: MTEB TweetSentimentExtractionClassification
config: default
split: test
metrics:
- type: accuracy
value: 57.29
- task:
type: Clustering
name: MTEB TwentyNewsgroupsClustering.v2
dataset:
type: mteb/TwentyNewsgroupsClustering.v2
name: MTEB TwentyNewsgroupsClustering.v2
config: default
split: test
metrics:
- type: v_measure
value: 22.39
- task:
type: PairClassification
name: MTEB TwitterSemEval2015
dataset:
type: mteb/TwitterSemEval2015
name: MTEB TwitterSemEval2015
config: default
split: test
metrics:
- type: max_ap
value: 53.19
- task:
type: PairClassification
name: MTEB TwitterURLCorpus
dataset:
type: mteb/TwitterURLCorpus
name: MTEB TwitterURLCorpus
config: default
split: test
metrics:
- type: max_ap
value: 82.55
Byrne-Embed
Byrne-Embed is a compact 85M-parameter sentence-embedding model. It maps text to 768-dimensional unit-norm vectors suitable for semantic similarity, retrieval, clustering, and reranking.
The backbone is a custom SpikeWhale decoder (the "Byrne" line). A mean-pooled representation of its last hidden state is projected to 768 dimensions by a learned head and unit-normalized, so cosine similarity between two embeddings is just a dot product.
Benchmark vs. EmbeddingGemma-300M
We benchmarked Byrne-Embed against Google's EmbeddingGemma-300M on 4,000 held-out sentences spanning educational web text, encyclopedic text, and instruction/chat text. Byrne-Embed's embedding geometry tracks closely with EmbeddingGemma's at roughly 1/3.5 the parameter count:
| Metric (Byrne-Embed vs EmbeddingGemma) | Result |
|---|---|
| Mean per-sentence cosine | 0.9415 (median 0.945, p10 0.912) |
| Sentences within 0.90 cosine | 94.7% |
| Similarity-structure agreement (Pearson) | 0.9702 |
| Similarity-structure agreement (Spearman) | 0.9599 |
| Per-anchor neighbour-ranking correlation | 0.9494 |
| Retrieval top-1 nearest-neighbour agreement | 72.8% |
| Retrieval Recall@10 overlap | 78.2% |
Reading the numbers. The two most important measures — how closely the two models agree on which sentences are similar — land at Pearson 0.97 / Spearman 0.96: when EmbeddingGemma judges two sentences similar, Byrne-Embed agrees almost identically. 94.7% of all sentences sit within 0.90 cosine. The lower top-1 retrieval number is expected and not a quality gap: in a dense pool of real sentences many neighbours are near-ties (0.88 vs 0.87), so the single #1 slot flips easily between near-duplicates — which is why Recall@10 stays at ~78% and the neighbour-ranking correlation is 0.95. Both models find the same neighbourhood; they just occasionally swap rank 1 and rank 2 among near-identical candidates.
Reproduce these numbers with the bundled run_tests.py (it loads both
models and prints the full table).
MTEB English Benchmark — MTEB(eng, v2)
Evaluated with the official mteb library on the full MTEB(eng, v2) suite (41/41 tasks). Raw results are in mteb_results/; machine-readable scores are in the model-index metadata above.
Overall MTEB(eng, v2) mean: 50.79
| Category | Mean | Tasks |
|---|---|---|
| STS | 71.93 | 9 |
| Classification | 70.57 | 8 |
| PairClassification | 74.07 | 3 |
| Clustering | 37.32 | 8 |
| Reranking | 40.48 | 2 |
| Retrieval | 24.64 | 10 |
| Summarization | 22.39 | 1 |
STS
| Task | Score |
|---|---|
| BIOSSES | 75.56 |
| SICK-R | 69.08 |
| STS12 | 64.88 |
| STS13 | 72.08 |
| STS14 | 67.76 |
| STS15 | 77.13 |
| STS17 | 83.23 |
| STS22.v2 | 60.53 |
| STSBenchmark | 77.08 |
Classification
| Task | Score |
|---|---|
| AmazonCounterfactualClassification | 80.12 |
| Banking77Classification | 74.64 |
| ImdbClassification | 60.97 |
| MTOPDomainClassification | 92.29 |
| MassiveIntentClassification | 63.23 |
| MassiveScenarioClassification | 73.05 |
| ToxicConversationsClassification | 62.94 |
| TweetSentimentExtractionClassification | 57.29 |
PairClassification
| Task | Score |
|---|---|
| SprintDuplicateQuestions | 86.47 |
| TwitterSemEval2015 | 53.19 |
| TwitterURLCorpus | 82.55 |
Clustering
| Task | Score |
|---|---|
| ArXivHierarchicalClusteringP2P | 53.15 |
| ArXivHierarchicalClusteringS2S | 50.39 |
| BiorxivClusteringP2P.v2 | 33.73 |
| MedrxivClusteringP2P.v2 | 32.70 |
| MedrxivClusteringS2S.v2 | 29.04 |
| StackExchangeClustering.v2 | 41.93 |
| StackExchangeClusteringP2P.v2 | 35.22 |
| TwentyNewsgroupsClustering.v2 | 22.39 |
Reranking
| Task | Score |
|---|---|
| AskUbuntuDupQuestions | 52.88 |
| MindSmallReranking | 28.07 |
Retrieval
| Task | Score |
|---|---|
| ArguAna | 37.67 |
| CQADupstackGamingRetrieval | 37.14 |
| CQADupstackUnixRetrieval | 23.48 |
| ClimateFEVERHardNegatives | 13.60 |
| FEVERHardNegatives | 28.70 |
| FiQA2018 | 11.38 |
| HotpotQAHardNegatives | 30.47 |
| SCIDOCS | 10.15 |
| TRECCOVID | 29.30 |
| Touche2020Retrieval.v3 | 24.50 |
Summarization
| Task | Score |
|---|---|
| SummEvalSummarization.v2 | 22.39 |
Usage
The model loads with standard transformers via trust_remote_code (the projection head
is fused into the weights, so a single from_pretrained loads everything):
import torch
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-Embed", trust_remote_code=True)
model = AutoModel.from_pretrained("Quazim0t0/Byrne-Embed", trust_remote_code=True).eval()
texts = ["The cat sat on the windowsill.", "A feline rested by the window."]
enc = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=128)
with torch.no_grad():
emb = model(**enc).last_hidden_state # (2, 768), L2-normalized
print(float(emb[0] @ emb[1])) # cosine similarity ~ 0.83
forward() returns L2-normalized 768-dim sentence embeddings, so cosine similarity is
just a dot product.
Files
| File | Purpose |
|---|---|
model.safetensors, config.json |
fused SpikeWhale backbone + projection head + config |
modeling_byrne_embed.py |
self-contained custom AutoModel class (SpikeWhale arch inlined; loaded via trust_remote_code) |
tokenizer.json, tokenizer_config.json, spike_tokenizer.py |
byte-level SpikeTokenizer + its code |
Limitations
- English-centric evaluation; non-English performance is untested.
- The single residual weak spot observed during evaluation is finance/economics paraphrase retrieval; general semantic similarity is strong.
- Custom architecture: load via the bundled
byrne_embedder.py(local modeling code — no remote code execution).
Citation
If you use Byrne-Embed, please cite:
@misc{byrne2026byrneembed,
title = {Byrne-Embed: A Compact 85M Sentence-Embedding Model},
author = {Byrne, Dean},
year = {2026},
howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-Embed}},
}
License
Apache-2.0.