--- license: apache-2.0 language: - en library_name: transformers tags: - sentence-embeddings - feature-extraction - sentence-similarity - spikewhale pipeline_tag: feature-extraction model-index: - name: Byrne-Embed results: - task: type: Classification name: MTEB AmazonCounterfactualClassification dataset: type: mteb/AmazonCounterfactualClassification name: MTEB AmazonCounterfactualClassification config: default split: test metrics: - type: accuracy value: 80.12 - task: type: Clustering name: MTEB ArXivHierarchicalClusteringP2P dataset: type: mteb/ArXivHierarchicalClusteringP2P name: MTEB ArXivHierarchicalClusteringP2P config: default split: test metrics: - type: v_measure value: 53.15 - task: type: Clustering name: MTEB ArXivHierarchicalClusteringS2S dataset: type: mteb/ArXivHierarchicalClusteringS2S name: MTEB ArXivHierarchicalClusteringS2S config: default split: test metrics: - type: v_measure value: 50.39 - task: type: Retrieval name: MTEB ArguAna dataset: type: mteb/ArguAna name: MTEB ArguAna config: default split: test metrics: - type: ndcg_at_10 value: 37.67 - task: type: Reranking name: MTEB AskUbuntuDupQuestions dataset: type: mteb/AskUbuntuDupQuestions name: MTEB AskUbuntuDupQuestions config: default split: test metrics: - type: map_at_1000 value: 52.88 - task: type: STS name: MTEB BIOSSES dataset: type: mteb/BIOSSES name: MTEB BIOSSES config: default split: test metrics: - type: cosine_spearman value: 75.56 - task: type: Classification name: MTEB Banking77Classification dataset: type: mteb/Banking77Classification name: MTEB Banking77Classification config: default split: test metrics: - type: accuracy value: 74.64 - task: type: Clustering name: MTEB BiorxivClusteringP2P.v2 dataset: type: mteb/BiorxivClusteringP2P.v2 name: MTEB BiorxivClusteringP2P.v2 config: default split: test metrics: - type: v_measure value: 33.73 - task: type: Retrieval name: MTEB CQADupstackGamingRetrieval dataset: type: mteb/CQADupstackGamingRetrieval name: MTEB CQADupstackGamingRetrieval config: default split: test metrics: - type: ndcg_at_10 value: 37.14 - task: type: Retrieval name: MTEB CQADupstackUnixRetrieval dataset: type: mteb/CQADupstackUnixRetrieval name: MTEB CQADupstackUnixRetrieval config: default split: test metrics: - type: ndcg_at_10 value: 23.48 - task: type: Retrieval name: MTEB ClimateFEVERHardNegatives dataset: type: mteb/ClimateFEVERHardNegatives name: MTEB ClimateFEVERHardNegatives config: default split: test metrics: - type: ndcg_at_10 value: 13.6 - task: type: Retrieval name: MTEB FEVERHardNegatives dataset: type: mteb/FEVERHardNegatives name: MTEB FEVERHardNegatives config: default split: test metrics: - type: ndcg_at_10 value: 28.7 - task: type: Retrieval name: MTEB FiQA2018 dataset: type: mteb/FiQA2018 name: MTEB FiQA2018 config: default split: test metrics: - type: ndcg_at_10 value: 11.38 - task: type: Retrieval name: MTEB HotpotQAHardNegatives dataset: type: mteb/HotpotQAHardNegatives name: MTEB HotpotQAHardNegatives config: default split: test metrics: - type: ndcg_at_10 value: 30.47 - task: type: Classification name: MTEB ImdbClassification dataset: type: mteb/ImdbClassification name: MTEB ImdbClassification config: default split: test metrics: - type: accuracy value: 60.97 - task: type: Classification name: MTEB MTOPDomainClassification dataset: type: mteb/MTOPDomainClassification name: MTEB MTOPDomainClassification config: default split: test metrics: - type: accuracy value: 92.29 - task: type: Classification name: MTEB MassiveIntentClassification dataset: type: mteb/MassiveIntentClassification name: MTEB MassiveIntentClassification config: default split: test metrics: - type: accuracy value: 63.23 - task: type: Classification name: MTEB MassiveScenarioClassification dataset: type: mteb/MassiveScenarioClassification name: MTEB MassiveScenarioClassification config: default split: test metrics: - type: accuracy value: 73.05 - task: type: Clustering name: MTEB MedrxivClusteringP2P.v2 dataset: type: mteb/MedrxivClusteringP2P.v2 name: MTEB MedrxivClusteringP2P.v2 config: default split: test metrics: - type: v_measure value: 32.7 - task: type: Clustering name: MTEB MedrxivClusteringS2S.v2 dataset: type: mteb/MedrxivClusteringS2S.v2 name: MTEB MedrxivClusteringS2S.v2 config: default split: test metrics: - type: v_measure value: 29.04 - task: type: Reranking name: MTEB MindSmallReranking dataset: type: mteb/MindSmallReranking name: MTEB MindSmallReranking config: default split: test metrics: - type: max_over_subqueries_map_at_1000 value: 28.07 - task: type: Retrieval name: MTEB SCIDOCS dataset: type: mteb/SCIDOCS name: MTEB SCIDOCS config: default split: test metrics: - type: ndcg_at_10 value: 10.15 - task: type: STS name: MTEB SICK-R dataset: type: mteb/SICK-R name: MTEB SICK-R config: default split: test metrics: - type: cosine_spearman value: 69.08 - task: type: STS name: MTEB STS12 dataset: type: mteb/STS12 name: MTEB STS12 config: default split: test metrics: - type: cosine_spearman value: 64.88 - task: type: STS name: MTEB STS13 dataset: type: mteb/STS13 name: MTEB STS13 config: default split: test metrics: - type: cosine_spearman value: 72.08 - task: type: STS name: MTEB STS14 dataset: type: mteb/STS14 name: MTEB STS14 config: default split: test metrics: - type: cosine_spearman value: 67.76 - task: type: STS name: MTEB STS15 dataset: type: mteb/STS15 name: MTEB STS15 config: default split: test metrics: - type: cosine_spearman value: 77.13 - task: type: STS name: MTEB STS17 dataset: type: mteb/STS17 name: MTEB STS17 config: default split: test metrics: - type: cosine_spearman value: 83.23 - task: type: STS name: MTEB STS22.v2 dataset: type: mteb/STS22.v2 name: MTEB STS22.v2 config: default split: test metrics: - type: cosine_spearman value: 60.53 - task: type: STS name: MTEB STSBenchmark dataset: type: mteb/STSBenchmark name: MTEB STSBenchmark config: default split: test metrics: - type: cosine_spearman value: 77.08 - task: type: PairClassification name: MTEB SprintDuplicateQuestions dataset: type: mteb/SprintDuplicateQuestions name: MTEB SprintDuplicateQuestions config: default split: test metrics: - type: max_ap value: 86.47 - task: type: Clustering name: MTEB StackExchangeClustering.v2 dataset: type: mteb/StackExchangeClustering.v2 name: MTEB StackExchangeClustering.v2 config: default split: test metrics: - type: v_measure value: 41.93 - task: type: Clustering name: MTEB StackExchangeClusteringP2P.v2 dataset: type: mteb/StackExchangeClusteringP2P.v2 name: MTEB StackExchangeClusteringP2P.v2 config: default split: test metrics: - type: v_measure value: 35.22 - task: type: Summarization name: MTEB SummEvalSummarization.v2 dataset: type: mteb/SummEvalSummarization.v2 name: MTEB SummEvalSummarization.v2 config: default split: test metrics: - type: cosine_spearman value: 22.39 - task: type: Retrieval name: MTEB TRECCOVID dataset: type: mteb/TRECCOVID name: MTEB TRECCOVID config: default split: test metrics: - type: ndcg_at_10 value: 29.3 - task: type: Retrieval name: MTEB Touche2020Retrieval.v3 dataset: type: mteb/Touche2020Retrieval.v3 name: MTEB Touche2020Retrieval.v3 config: default split: test metrics: - type: ndcg_at_10 value: 24.5 - task: type: Classification name: MTEB ToxicConversationsClassification dataset: type: mteb/ToxicConversationsClassification name: MTEB ToxicConversationsClassification config: default split: test metrics: - type: accuracy value: 62.94 - task: type: Classification name: MTEB TweetSentimentExtractionClassification dataset: type: mteb/TweetSentimentExtractionClassification name: MTEB TweetSentimentExtractionClassification config: default split: test metrics: - type: accuracy value: 57.29 - task: type: Clustering name: MTEB TwentyNewsgroupsClustering.v2 dataset: type: mteb/TwentyNewsgroupsClustering.v2 name: MTEB TwentyNewsgroupsClustering.v2 config: default split: test metrics: - type: v_measure value: 22.39 - task: type: PairClassification name: MTEB TwitterSemEval2015 dataset: type: mteb/TwitterSemEval2015 name: MTEB TwitterSemEval2015 config: default split: test metrics: - type: max_ap value: 53.19 - task: type: PairClassification name: MTEB TwitterURLCorpus dataset: type: mteb/TwitterURLCorpus name: MTEB TwitterURLCorpus config: default split: test metrics: - type: max_ap value: 82.55 --- # Byrne-Embed **Byrne-Embed** is a compact **85M-parameter** sentence-embedding model. It maps text to **768-dimensional unit-norm vectors** suitable for semantic similarity, retrieval, clustering, and reranking. The backbone is a custom **SpikeWhale** decoder (the "Byrne" line). A mean-pooled representation of its last hidden state is projected to 768 dimensions by a learned head and unit-normalized, so cosine similarity between two embeddings is just a dot product. ## Benchmark vs. EmbeddingGemma-300M We benchmarked Byrne-Embed against Google's **[EmbeddingGemma-300M](https://huggingface.co/google/embeddinggemma-300m)** on **4,000 held-out sentences** spanning educational web text, encyclopedic text, and instruction/chat text. Byrne-Embed's embedding geometry tracks closely with EmbeddingGemma's at roughly **1/3.5 the parameter count**: | Metric (Byrne-Embed vs EmbeddingGemma) | Result | |---|---| | **Mean per-sentence cosine** | **0.9415** (median 0.945, p10 0.912) | | Sentences within 0.90 cosine | **94.7%** | | **Similarity-structure agreement (Pearson)** | **0.9702** | | Similarity-structure agreement (Spearman) | 0.9599 | | Per-anchor neighbour-ranking correlation | 0.9494 | | Retrieval top-1 nearest-neighbour agreement | 72.8% | | Retrieval Recall@10 overlap | 78.2% | **Reading the numbers.** The two most important measures — *how closely the two models agree on which sentences are similar* — land at **Pearson 0.97 / Spearman 0.96**: when EmbeddingGemma judges two sentences similar, Byrne-Embed agrees almost identically. **94.7%** of all sentences sit within 0.90 cosine. The lower top-1 retrieval number is expected and not a quality gap: in a dense pool of real sentences many neighbours are near-ties (0.88 vs 0.87), so the single #1 slot flips easily between near-duplicates — which is why **Recall@10 stays at ~78%** and the **neighbour-ranking correlation is 0.95**. Both models find the same neighbourhood; they just occasionally swap rank 1 and rank 2 among near-identical candidates. Reproduce these numbers with the bundled [`run_tests.py`](./run_tests.py) (it loads both models and prints the full table). ## MTEB English Benchmark — MTEB(eng, v2) Evaluated with the official [`mteb`](https://github.com/embeddings-benchmark/mteb) library on the full **MTEB(eng, v2)** suite (**41/41 tasks**). Raw results are in `mteb_results/`; machine-readable scores are in the `model-index` metadata above. **Overall MTEB(eng, v2) mean: 50.79** | Category | Mean | Tasks | |---|---|---| | STS | 71.93 | 9 | | Classification | 70.57 | 8 | | PairClassification | 74.07 | 3 | | Clustering | 37.32 | 8 | | Reranking | 40.48 | 2 | | Retrieval | 24.64 | 10 | | Summarization | 22.39 | 1 | ### STS | Task | Score | |---|---| | BIOSSES | 75.56 | | SICK-R | 69.08 | | STS12 | 64.88 | | STS13 | 72.08 | | STS14 | 67.76 | | STS15 | 77.13 | | STS17 | 83.23 | | STS22.v2 | 60.53 | | STSBenchmark | 77.08 | ### Classification | Task | Score | |---|---| | AmazonCounterfactualClassification | 80.12 | | Banking77Classification | 74.64 | | ImdbClassification | 60.97 | | MTOPDomainClassification | 92.29 | | MassiveIntentClassification | 63.23 | | MassiveScenarioClassification | 73.05 | | ToxicConversationsClassification | 62.94 | | TweetSentimentExtractionClassification | 57.29 | ### PairClassification | Task | Score | |---|---| | SprintDuplicateQuestions | 86.47 | | TwitterSemEval2015 | 53.19 | | TwitterURLCorpus | 82.55 | ### Clustering | Task | Score | |---|---| | ArXivHierarchicalClusteringP2P | 53.15 | | ArXivHierarchicalClusteringS2S | 50.39 | | BiorxivClusteringP2P.v2 | 33.73 | | MedrxivClusteringP2P.v2 | 32.70 | | MedrxivClusteringS2S.v2 | 29.04 | | StackExchangeClustering.v2 | 41.93 | | StackExchangeClusteringP2P.v2 | 35.22 | | TwentyNewsgroupsClustering.v2 | 22.39 | ### Reranking | Task | Score | |---|---| | AskUbuntuDupQuestions | 52.88 | | MindSmallReranking | 28.07 | ### Retrieval | Task | Score | |---|---| | ArguAna | 37.67 | | CQADupstackGamingRetrieval | 37.14 | | CQADupstackUnixRetrieval | 23.48 | | ClimateFEVERHardNegatives | 13.60 | | FEVERHardNegatives | 28.70 | | FiQA2018 | 11.38 | | HotpotQAHardNegatives | 30.47 | | SCIDOCS | 10.15 | | TRECCOVID | 29.30 | | Touche2020Retrieval.v3 | 24.50 | ### Summarization | Task | Score | |---|---| | SummEvalSummarization.v2 | 22.39 | ## Usage The model loads with standard `transformers` via `trust_remote_code` (the projection head is fused into the weights, so a single `from_pretrained` loads everything): ```python import torch from transformers import AutoModel, AutoTokenizer tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-Embed", trust_remote_code=True) model = AutoModel.from_pretrained("Quazim0t0/Byrne-Embed", trust_remote_code=True).eval() texts = ["The cat sat on the windowsill.", "A feline rested by the window."] enc = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=128) with torch.no_grad(): emb = model(**enc).last_hidden_state # (2, 768), L2-normalized print(float(emb[0] @ emb[1])) # cosine similarity ~ 0.83 ``` `forward()` returns L2-normalized 768-dim sentence embeddings, so cosine similarity is just a dot product. ## Files | File | Purpose | |---|---| | `model.safetensors`, `config.json` | fused SpikeWhale backbone + projection head + config | | `modeling_byrne_embed.py` | self-contained custom `AutoModel` class (SpikeWhale arch inlined; loaded via `trust_remote_code`) | | `tokenizer.json`, `tokenizer_config.json`, `spike_tokenizer.py` | byte-level SpikeTokenizer + its code | ## Limitations - English-centric evaluation; non-English performance is untested. - The single residual weak spot observed during evaluation is finance/economics paraphrase retrieval; general semantic similarity is strong. - Custom architecture: load via the bundled `byrne_embedder.py` (local modeling code — no remote code execution). ## Citation If you use Byrne-Embed, please cite: ```bibtex @misc{byrne2026byrneembed, title = {Byrne-Embed: A Compact 85M Sentence-Embedding Model}, author = {Byrne, Dean}, year = {2026}, howpublished = {\url{https://huggingface.co/Quazim0t0/Byrne-Embed}}, } ``` ## License Apache-2.0.