| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| tags: |
| - sentence-embeddings |
| - feature-extraction |
| - sentence-similarity |
| - spikewhale |
| pipeline_tag: feature-extraction |
| model-index: |
| - name: Byrne-Embed |
| results: |
| - task: |
| type: Classification |
| name: MTEB AmazonCounterfactualClassification |
| dataset: |
| type: mteb/AmazonCounterfactualClassification |
| name: MTEB AmazonCounterfactualClassification |
| config: default |
| split: test |
| metrics: |
| - type: accuracy |
| value: 80.12 |
| - task: |
| type: Clustering |
| name: MTEB ArXivHierarchicalClusteringP2P |
| dataset: |
| type: mteb/ArXivHierarchicalClusteringP2P |
| name: MTEB ArXivHierarchicalClusteringP2P |
| config: default |
| split: test |
| metrics: |
| - type: v_measure |
| value: 53.15 |
| - task: |
| type: Clustering |
| name: MTEB ArXivHierarchicalClusteringS2S |
| dataset: |
| type: mteb/ArXivHierarchicalClusteringS2S |
| name: MTEB ArXivHierarchicalClusteringS2S |
| config: default |
| split: test |
| metrics: |
| - type: v_measure |
| value: 50.39 |
| - task: |
| type: Retrieval |
| name: MTEB ArguAna |
| dataset: |
| type: mteb/ArguAna |
| name: MTEB ArguAna |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 37.67 |
| - task: |
| type: Reranking |
| name: MTEB AskUbuntuDupQuestions |
| dataset: |
| type: mteb/AskUbuntuDupQuestions |
| name: MTEB AskUbuntuDupQuestions |
| config: default |
| split: test |
| metrics: |
| - type: map_at_1000 |
| value: 52.88 |
| - task: |
| type: STS |
| name: MTEB BIOSSES |
| dataset: |
| type: mteb/BIOSSES |
| name: MTEB BIOSSES |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 75.56 |
| - task: |
| type: Classification |
| name: MTEB Banking77Classification |
| dataset: |
| type: mteb/Banking77Classification |
| name: MTEB Banking77Classification |
| config: default |
| split: test |
| metrics: |
| - type: accuracy |
| value: 74.64 |
| - task: |
| type: Clustering |
| name: MTEB BiorxivClusteringP2P.v2 |
| dataset: |
| type: mteb/BiorxivClusteringP2P.v2 |
| name: MTEB BiorxivClusteringP2P.v2 |
| config: default |
| split: test |
| metrics: |
| - type: v_measure |
| value: 33.73 |
| - task: |
| type: Retrieval |
| name: MTEB CQADupstackGamingRetrieval |
| dataset: |
| type: mteb/CQADupstackGamingRetrieval |
| name: MTEB CQADupstackGamingRetrieval |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 37.14 |
| - task: |
| type: Retrieval |
| name: MTEB CQADupstackUnixRetrieval |
| dataset: |
| type: mteb/CQADupstackUnixRetrieval |
| name: MTEB CQADupstackUnixRetrieval |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 23.48 |
| - task: |
| type: Retrieval |
| name: MTEB ClimateFEVERHardNegatives |
| dataset: |
| type: mteb/ClimateFEVERHardNegatives |
| name: MTEB ClimateFEVERHardNegatives |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 13.6 |
| - task: |
| type: Retrieval |
| name: MTEB FEVERHardNegatives |
| dataset: |
| type: mteb/FEVERHardNegatives |
| name: MTEB FEVERHardNegatives |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 28.7 |
| - task: |
| type: Retrieval |
| name: MTEB FiQA2018 |
| dataset: |
| type: mteb/FiQA2018 |
| name: MTEB FiQA2018 |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 11.38 |
| - task: |
| type: Retrieval |
| name: MTEB HotpotQAHardNegatives |
| dataset: |
| type: mteb/HotpotQAHardNegatives |
| name: MTEB HotpotQAHardNegatives |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 30.47 |
| - task: |
| type: Classification |
| name: MTEB ImdbClassification |
| dataset: |
| type: mteb/ImdbClassification |
| name: MTEB ImdbClassification |
| config: default |
| split: test |
| metrics: |
| - type: accuracy |
| value: 60.97 |
| - task: |
| type: Classification |
| name: MTEB MTOPDomainClassification |
| dataset: |
| type: mteb/MTOPDomainClassification |
| name: MTEB MTOPDomainClassification |
| config: default |
| split: test |
| metrics: |
| - type: accuracy |
| value: 92.29 |
| - task: |
| type: Classification |
| name: MTEB MassiveIntentClassification |
| dataset: |
| type: mteb/MassiveIntentClassification |
| name: MTEB MassiveIntentClassification |
| config: default |
| split: test |
| metrics: |
| - type: accuracy |
| value: 63.23 |
| - task: |
| type: Classification |
| name: MTEB MassiveScenarioClassification |
| dataset: |
| type: mteb/MassiveScenarioClassification |
| name: MTEB MassiveScenarioClassification |
| config: default |
| split: test |
| metrics: |
| - type: accuracy |
| value: 73.05 |
| - task: |
| type: Clustering |
| name: MTEB MedrxivClusteringP2P.v2 |
| dataset: |
| type: mteb/MedrxivClusteringP2P.v2 |
| name: MTEB MedrxivClusteringP2P.v2 |
| config: default |
| split: test |
| metrics: |
| - type: v_measure |
| value: 32.7 |
| - task: |
| type: Clustering |
| name: MTEB MedrxivClusteringS2S.v2 |
| dataset: |
| type: mteb/MedrxivClusteringS2S.v2 |
| name: MTEB MedrxivClusteringS2S.v2 |
| config: default |
| split: test |
| metrics: |
| - type: v_measure |
| value: 29.04 |
| - task: |
| type: Reranking |
| name: MTEB MindSmallReranking |
| dataset: |
| type: mteb/MindSmallReranking |
| name: MTEB MindSmallReranking |
| config: default |
| split: test |
| metrics: |
| - type: max_over_subqueries_map_at_1000 |
| value: 28.07 |
| - task: |
| type: Retrieval |
| name: MTEB SCIDOCS |
| dataset: |
| type: mteb/SCIDOCS |
| name: MTEB SCIDOCS |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 10.15 |
| - task: |
| type: STS |
| name: MTEB SICK-R |
| dataset: |
| type: mteb/SICK-R |
| name: MTEB SICK-R |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 69.08 |
| - task: |
| type: STS |
| name: MTEB STS12 |
| dataset: |
| type: mteb/STS12 |
| name: MTEB STS12 |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 64.88 |
| - task: |
| type: STS |
| name: MTEB STS13 |
| dataset: |
| type: mteb/STS13 |
| name: MTEB STS13 |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 72.08 |
| - task: |
| type: STS |
| name: MTEB STS14 |
| dataset: |
| type: mteb/STS14 |
| name: MTEB STS14 |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 67.76 |
| - task: |
| type: STS |
| name: MTEB STS15 |
| dataset: |
| type: mteb/STS15 |
| name: MTEB STS15 |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 77.13 |
| - task: |
| type: STS |
| name: MTEB STS17 |
| dataset: |
| type: mteb/STS17 |
| name: MTEB STS17 |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 83.23 |
| - task: |
| type: STS |
| name: MTEB STS22.v2 |
| dataset: |
| type: mteb/STS22.v2 |
| name: MTEB STS22.v2 |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 60.53 |
| - task: |
| type: STS |
| name: MTEB STSBenchmark |
| dataset: |
| type: mteb/STSBenchmark |
| name: MTEB STSBenchmark |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 77.08 |
| - task: |
| type: PairClassification |
| name: MTEB SprintDuplicateQuestions |
| dataset: |
| type: mteb/SprintDuplicateQuestions |
| name: MTEB SprintDuplicateQuestions |
| config: default |
| split: test |
| metrics: |
| - type: max_ap |
| value: 86.47 |
| - task: |
| type: Clustering |
| name: MTEB StackExchangeClustering.v2 |
| dataset: |
| type: mteb/StackExchangeClustering.v2 |
| name: MTEB StackExchangeClustering.v2 |
| config: default |
| split: test |
| metrics: |
| - type: v_measure |
| value: 41.93 |
| - task: |
| type: Clustering |
| name: MTEB StackExchangeClusteringP2P.v2 |
| dataset: |
| type: mteb/StackExchangeClusteringP2P.v2 |
| name: MTEB StackExchangeClusteringP2P.v2 |
| config: default |
| split: test |
| metrics: |
| - type: v_measure |
| value: 35.22 |
| - task: |
| type: Summarization |
| name: MTEB SummEvalSummarization.v2 |
| dataset: |
| type: mteb/SummEvalSummarization.v2 |
| name: MTEB SummEvalSummarization.v2 |
| config: default |
| split: test |
| metrics: |
| - type: cosine_spearman |
| value: 22.39 |
| - task: |
| type: Retrieval |
| name: MTEB TRECCOVID |
| dataset: |
| type: mteb/TRECCOVID |
| name: MTEB TRECCOVID |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 29.3 |
| - task: |
| type: Retrieval |
| name: MTEB Touche2020Retrieval.v3 |
| dataset: |
| type: mteb/Touche2020Retrieval.v3 |
| name: MTEB Touche2020Retrieval.v3 |
| config: default |
| split: test |
| metrics: |
| - type: ndcg_at_10 |
| value: 24.5 |
| - task: |
| type: Classification |
| name: MTEB ToxicConversationsClassification |
| dataset: |
| type: mteb/ToxicConversationsClassification |
| name: MTEB ToxicConversationsClassification |
| config: default |
| split: test |
| metrics: |
| - type: accuracy |
| value: 62.94 |
| - task: |
| type: Classification |
| name: MTEB TweetSentimentExtractionClassification |
| dataset: |
| type: mteb/TweetSentimentExtractionClassification |
| name: MTEB TweetSentimentExtractionClassification |
| config: default |
| split: test |
| metrics: |
| - type: accuracy |
| value: 57.29 |
| - task: |
| type: Clustering |
| name: MTEB TwentyNewsgroupsClustering.v2 |
| dataset: |
| type: mteb/TwentyNewsgroupsClustering.v2 |
| name: MTEB TwentyNewsgroupsClustering.v2 |
| config: default |
| split: test |
| metrics: |
| - type: v_measure |
| value: 22.39 |
| - task: |
| type: PairClassification |
| name: MTEB TwitterSemEval2015 |
| dataset: |
| type: mteb/TwitterSemEval2015 |
| name: MTEB TwitterSemEval2015 |
| config: default |
| split: test |
| metrics: |
| - type: max_ap |
| value: 53.19 |
| - task: |
| type: PairClassification |
| name: MTEB TwitterURLCorpus |
| dataset: |
| type: mteb/TwitterURLCorpus |
| name: MTEB TwitterURLCorpus |
| config: default |
| split: test |
| metrics: |
| - type: max_ap |
| value: 82.55 |
| --- |
| |
| <img src="Byrne-LLMs.png" width="50%"> |
|
|
|
|
| # Byrne-Embed |
|
|
| **Byrne-Embed** is a compact **85M-parameter** sentence-embedding model. It maps text to |
| **768-dimensional unit-norm vectors** suitable for semantic similarity, retrieval, |
| clustering, and reranking. |
|
|
| The backbone is a custom **SpikeWhale** decoder (the "Byrne" line). A mean-pooled |
| representation of its last hidden state is projected to 768 dimensions by a learned head |
| and unit-normalized, so cosine similarity between two embeddings is just a dot product. |
|
|
| ## Benchmark vs. EmbeddingGemma-300M |
|
|
| We benchmarked Byrne-Embed against Google's |
| **[EmbeddingGemma-300M](https://huggingface.co/google/embeddinggemma-300m)** on **4,000 |
| held-out sentences** spanning educational web text, encyclopedic text, and |
| instruction/chat text. Byrne-Embed's embedding geometry tracks closely with |
| EmbeddingGemma's at roughly **1/3.5 the parameter count**: |
|
|
| | Metric (Byrne-Embed vs EmbeddingGemma) | Result | |
| |---|---| |
| | **Mean per-sentence cosine** | **0.9415** (median 0.945, p10 0.912) | |
| | Sentences within 0.90 cosine | **94.7%** | |
| | **Similarity-structure agreement (Pearson)** | **0.9702** | |
| | Similarity-structure agreement (Spearman) | 0.9599 | |
| | Per-anchor neighbour-ranking correlation | 0.9494 | |
| | Retrieval top-1 nearest-neighbour agreement | 72.8% | |
| | Retrieval Recall@10 overlap | 78.2% | |
|
|
| **Reading the numbers.** The two most important measures — *how closely the two models |
| agree on which sentences are similar* — land at **Pearson 0.97 / Spearman 0.96**: when |
| EmbeddingGemma judges two sentences similar, Byrne-Embed agrees almost identically. |
| **94.7%** of all sentences sit within 0.90 cosine. The lower top-1 retrieval number is |
| expected and not a quality gap: in a dense pool of real sentences many neighbours are |
| near-ties (0.88 vs 0.87), so the single #1 slot flips easily between near-duplicates — |
| which is why **Recall@10 stays at ~78%** and the **neighbour-ranking correlation is |
| 0.95**. Both models find the same neighbourhood; they just occasionally swap rank 1 and |
| rank 2 among near-identical candidates. |
|
|
| Reproduce these numbers with the bundled [`run_tests.py`](./run_tests.py) (it loads both |
| models and prints the full table). |
|
|
|
|
|
|
| ## MTEB English Benchmark — MTEB(eng, v2) |
|
|
| Evaluated with the official [`mteb`](https://github.com/embeddings-benchmark/mteb) library on the full **MTEB(eng, v2)** suite (**41/41 tasks**). Raw results are in `mteb_results/`; machine-readable scores are in the `model-index` metadata above. |
|
|
| **Overall MTEB(eng, v2) mean: 50.79** |
|
|
| | Category | Mean | Tasks | |
| |---|---|---| |
| | STS | 71.93 | 9 | |
| | Classification | 70.57 | 8 | |
| | PairClassification | 74.07 | 3 | |
| | Clustering | 37.32 | 8 | |
| | Reranking | 40.48 | 2 | |
| | Retrieval | 24.64 | 10 | |
| | Summarization | 22.39 | 1 | |
|
|
| ### STS |
|
|
| | Task | Score | |
| |---|---| |
| | BIOSSES | 75.56 | |
| | SICK-R | 69.08 | |
| | STS12 | 64.88 | |
| | STS13 | 72.08 | |
| | STS14 | 67.76 | |
| | STS15 | 77.13 | |
| | STS17 | 83.23 | |
| | STS22.v2 | 60.53 | |
| | STSBenchmark | 77.08 | |
|
|
| ### Classification |
|
|
| | Task | Score | |
| |---|---| |
| | AmazonCounterfactualClassification | 80.12 | |
| | Banking77Classification | 74.64 | |
| | ImdbClassification | 60.97 | |
| | MTOPDomainClassification | 92.29 | |
| | MassiveIntentClassification | 63.23 | |
| | MassiveScenarioClassification | 73.05 | |
| | ToxicConversationsClassification | 62.94 | |
| | TweetSentimentExtractionClassification | 57.29 | |
|
|
| ### PairClassification |
|
|
| | Task | Score | |
| |---|---| |
| | SprintDuplicateQuestions | 86.47 | |
| | TwitterSemEval2015 | 53.19 | |
| | TwitterURLCorpus | 82.55 | |
|
|
| ### Clustering |
|
|
| | Task | Score | |
| |---|---| |
| | ArXivHierarchicalClusteringP2P | 53.15 | |
| | ArXivHierarchicalClusteringS2S | 50.39 | |
| | BiorxivClusteringP2P.v2 | 33.73 | |
| | MedrxivClusteringP2P.v2 | 32.70 | |
| | MedrxivClusteringS2S.v2 | 29.04 | |
| | StackExchangeClustering.v2 | 41.93 | |
| | StackExchangeClusteringP2P.v2 | 35.22 | |
| | TwentyNewsgroupsClustering.v2 | 22.39 | |
|
|
| ### Reranking |
|
|
| | Task | Score | |
| |---|---| |
| | AskUbuntuDupQuestions | 52.88 | |
| | MindSmallReranking | 28.07 | |
|
|
| ### Retrieval |
|
|
| | Task | Score | |
| |---|---| |
| | ArguAna | 37.67 | |
| | CQADupstackGamingRetrieval | 37.14 | |
| | CQADupstackUnixRetrieval | 23.48 | |
| | ClimateFEVERHardNegatives | 13.60 | |
| | FEVERHardNegatives | 28.70 | |
| | FiQA2018 | 11.38 | |
| | HotpotQAHardNegatives | 30.47 | |
| | SCIDOCS | 10.15 | |
| | TRECCOVID | 29.30 | |
| | Touche2020Retrieval.v3 | 24.50 | |
|
|
| ### Summarization |
|
|
| | Task | Score | |
| |---|---| |
| | SummEvalSummarization.v2 | 22.39 | |
|
|
| ## Usage |
|
|
| The model loads with standard `transformers` via `trust_remote_code` (the projection head |
| is fused into the weights, so a single `from_pretrained` loads everything): |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-Embed", trust_remote_code=True) |
| model = AutoModel.from_pretrained("Quazim0t0/Byrne-Embed", trust_remote_code=True).eval() |
| |
| texts = ["The cat sat on the windowsill.", "A feline rested by the window."] |
| enc = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=128) |
| with torch.no_grad(): |
| emb = model(**enc).last_hidden_state # (2, 768), L2-normalized |
| |
| print(float(emb[0] @ emb[1])) # cosine similarity ~ 0.83 |
| ``` |
|
|
| `forward()` returns L2-normalized 768-dim sentence embeddings, so cosine similarity is |
| just a dot product. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `model.safetensors`, `config.json` | fused SpikeWhale backbone + projection head + config | |
| | `modeling_byrne_embed.py` | self-contained custom `AutoModel` class (SpikeWhale arch inlined; loaded via `trust_remote_code`) | |
| | `tokenizer.json`, `tokenizer_config.json`, `spike_tokenizer.py` | byte-level SpikeTokenizer + its code | |
|
|
| ## Limitations |
|
|
| - English-centric evaluation; non-English performance is untested. |
| - The single residual weak spot observed during evaluation is finance/economics |
| paraphrase retrieval; general semantic similarity is strong. |
| - Custom architecture: load via the bundled `byrne_embedder.py` (local modeling code — |
| no remote code execution). |
|
|
| ## Citation |
|
|
| If you use Byrne-Embed, please cite: |
|
|
| ```bibtex |
| @misc{byrne2026byrneembed, |
| title = {Byrne-Embed: A Compact 85M Sentence-Embedding Model}, |
| author = {Byrne, Dean}, |
| year = {2026}, |
| howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-Embed}}, |
| } |
| ``` |
|
|
| ## License |
|
|
| Apache-2.0. |
|
|
