--- tags: - mteb - sentence-transformers - sentence-similarity - feature-extraction - dense - embedding - onnx - int8 - memory - openmemory - generated_from_trainer - loss:GISTEmbedLoss - loss:MultipleNegativesRankingLoss - loss:CombinedDistillationLoss - loss:CoSENTLoss base_model: Snowflake/snowflake-arctic-embed-l-v2.0 pipeline_tag: sentence-similarity library_name: sentence-transformers license: apache-2.0 language: - en datasets: - sentence-transformers/all-nli - sentence-transformers/stsb --- # intelli-embed-v3 Custom embedding model: **4-phase multi-signal fine-tune** of [`Snowflake/snowflake-arctic-embed-l-v2.0`](https://huggingface.co/Snowflake/snowflake-arctic-embed-l-v2.0) (XLM-RoBERTa-large, 1024-dim). **First local model to surpass Azure text-embedding-3-large** (`sep=0.520` GPU fp16 vs `0.515`). Beats azure-large on 5/6 OpenMemory-specific metrics. CPU INT8 ONNX reaches `sep=0.505` (98% of azure-large) at **11 ms** latency. Built for [OpenMemory](https://github.com/mem0ai/mem0) — optimized for personal memory storage, retrieval, deduplication, negation detection, and entity search. ## Model Details | Property | Value | |----------|-------| | Architecture | XLM-RoBERTa-large (24 layers, 1024-dim) | | Base model | [`Snowflake/snowflake-arctic-embed-l-v2.0`](https://huggingface.co/Snowflake/snowflake-arctic-embed-l-v2.0) | | Pooling | CLS token + L2-normalize | | Output dimension | 1024 | | Similarity function | Cosine similarity | | Vocab | 250,002 (multilingual, XLM-R) | | Max tokens | 256 (fine-tune) / 8,194 (architecture limit) | | PyTorch fp32 size | 2.2 GB (`model.safetensors`) | | ONNX INT8 size | 542 MB (`onnx/model_quantized.onnx`) | | ONNX fp32 size | 2.2 GB (`onnx/model.onnx` + `model.onnx_data`) | | GPU fp16 VRAM | ~2.3 GB | ### Full Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'}) (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True}) (2): Normalize() ) ``` ## Benchmark Results Benchmarked on 10 SW-engineering positive pairs + 10 negative pairs. `Sep = mean(PosSim) − mean(NegSim)`. | Deployment | Sep | p50 latency | Key advantage | |-----------|-----|-------------|---------------| | **GPU fp16** (sentence-transformers CUDA) | **0.520** | 23 ms | Beats azure-large (0.515) | | **CPU INT8** (ort-node ONNX) | 0.505 | 88 ms | Zero GPU needed | | **Node.js** (transformers.js q8) | 0.505 | 11 ms | Fastest, identical quality to CPU INT8 | | azure-large (cloud baseline) | 0.515 | 97 ms | Cloud reference | ### Extended OpenMemory Metrics (vs azure-large) | Metric | v3 GPU | v3 CPU | azure-large | What it measures | |--------|--------|--------|-------------|------------------| | `memSep` | 0.541 | 0.533 | 0.509 | Personal-memory discrimination | | `dedupGap` | 0.156 | 0.159 | 0.050 | Dedup reliability (>0.15 target) | | `negGap` | 0.111 | 0.103 | 0.088 | Negation detection (>0.10 safe) | | `supSim` | 0.592 | 0.580 | 0.651 | Supersede zone (0.75–0.92 target) | | `entSep` | 0.539 | 0.522 | 0.514 | Entity search quality | **v3 wins 5/6 metrics.** Only `supSim` is lower (both below target zone — requires threshold tuning). ### Held-Out Validation Tested on completely fresh sentences with zero overlap to training data: | Provider | Sep (benchmark) | Sep (held-out) | Delta | |----------|----------------|----------------|-------| | v3-GPU-fp16 | 0.520 | 0.595 | +14% | | v3-CPU-INT8 | 0.505 | 0.578 | +14% | | azure-large | 0.515 | 0.637 | +24% | Scores improve on held-out data, confirming no benchmark inflation. ### MTEB Benchmark (English, v2) Full evaluation on MTEB(eng, v2) — 41 tasks across 7 categories. Evaluated with `mteb` v2.8.8 on NVIDIA RTX 4090, FP16, batch_size=256. | Category | Avg Score | # Tasks | |----------|-----------|---------| | Classification | 0.7650 | 8 | | Clustering | 0.4228 | 8 | | PairClassification | 0.7976 | 4 | | Reranking | 0.3001 | 1 | | Retrieval | 0.4931 | 10 | | STS | 0.8341 | 9 | | Summarization | 0.3452 | 1 | | **Overall** | **0.5654** | **41** |
Individual Task Scores (click to expand) | Task | Category | Score | |------|----------|-------| | AmazonCounterfactualClassification | Classification | 0.7597 | | ArXivHierarchicalClusteringP2P | Clustering | 0.5530 | | ArXivHierarchicalClusteringS2S | Clustering | 0.5295 | | ArguAna | Retrieval | 0.5589 | | AskUbuntuDupQuestions | PairClassification | 0.6227 | | BIOSSES | STS | 0.8664 | | Banking77Classification | Classification | 0.8327 | | BiorxivClusteringP2P.v2 | Clustering | 0.3622 | | CQADupstackGamingRetrieval | Retrieval | 0.5932 | | CQADupstackUnixRetrieval | Retrieval | 0.4309 | | ClimateFEVERHardNegatives | Retrieval | 0.2524 | | FEVERHardNegatives | Retrieval | 0.7669 | | FiQA2018 | Retrieval | 0.3531 | | HotpotQAHardNegatives | Retrieval | 0.6076 | | ImdbClassification | Classification | 0.7771 | | MTOPDomainClassification | Classification | 0.9265 | | MassiveIntentClassification | Classification | 0.7191 | | MassiveScenarioClassification | Classification | 0.7614 | | MedrxivClusteringP2P.v2 | Clustering | 0.3400 | | MedrxivClusteringS2S.v2 | Clustering | 0.3116 | | MindSmallReranking | Reranking | 0.3001 | | SCIDOCS | Retrieval | 0.1818 | | SICK-R | STS | 0.7885 | | STS12 | STS | 0.8132 | | STS13 | STS | 0.8908 | | STS14 | STS | 0.8527 | | STS15 | STS | 0.8961 | | STS17 | STS | 0.8468 | | STS22.v2 | STS | 0.6754 | | STSBenchmark | STS | 0.8765 | | SprintDuplicateQuestions | PairClassification | 0.9579 | | StackExchangeClustering.v2 | Clustering | 0.4990 | | StackExchangeClusteringP2P.v2 | Clustering | 0.3935 | | SummEvalSummarization.v2 | Summarization | 0.3452 | | TRECCOVID | Retrieval | 0.6636 | | Touche2020Retrieval.v3 | Retrieval | 0.5224 | | ToxicConversationsClassification | Classification | 0.7053 | | TweetSentimentExtractionClassification | Classification | 0.6379 | | TwentyNewsgroupsClustering.v2 | Clustering | 0.3938 | | TwitterSemEval2015 | PairClassification | 0.7398 | | TwitterURLCorpus | PairClassification | 0.8698 |
## Training Strategy 4-phase multi-signal fine-tuning with complementary objectives: | Phase | Loss | Teacher / Data | Purpose | |-------|------|----------------|---------| | **A** | GISTEmbedLoss + MNRL | `mxbai-embed-large-v1` + 200k AllNLI + 83 OpenMemory pairs | PosSim↑ NegSim↓ via guided contrastive learning | | **B** | MSE + Relational Distillation (3 epochs) | `text-embedding-3-large` 1024-dim (API, 189k cached) | Full Sep signal + pairwise cosine structure alignment (α=0.3) | | **C** | MNRL + CoSENT | 50k NLI contradiction triplets + 20 custom negation + STSB | Negation separation (negGap) + fine-grained similarity ordering | | **D** | Hard-Negative MNRL | 13k+ mined triplets (margin=0.210) | Decision boundary refinement | ### Training Hyperparameters - **Hardware:** NVIDIA RTX 4090 (24 GB VRAM) - **Training time:** ~2.1 hours total (all 4 phases) - **Batch size:** 32 (phases A, C, D), 128 (phase B) - **Learning rate:** 2e-5 (phase A), 1e-5 (phase B), 5e-6 (phases C, D) - **Scheduler:** Cosine with 5–10% warmup - **Precision:** bf16 mixed precision - **Optimizer:** AdamW (fused) - **Gradient checkpointing:** enabled ### Framework Versions - Python: 3.10.6 - Sentence Transformers: 5.2.3 - Transformers: 4.57.6 - PyTorch: 2.6.0+cu124 - ONNX Runtime: 1.22.0 (INT8 quantization) ## Usage ### Python (sentence-transformers) ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("serhiiseletskyi/intelli-embed-v3") queries = model.encode(["What programming language does the user prefer?"]) docs = model.encode(["The user is a senior TypeScript developer who loves React."]) similarity = model.similarity(queries, docs) print(similarity) # tensor([[0.62]]) ``` ### Node.js (transformers.js — recommended for production) ```javascript import { pipeline } from "@huggingface/transformers"; const extractor = await pipeline( "feature-extraction", "serhiiseletskyi/intelli-embed-v3", { dtype: "q8" } // loads onnx/model_quantized.onnx (542 MB) ); const output = await extractor("User prefers dark mode", { pooling: "cls", normalize: true, }); console.log(output.data.length); // 1024 ``` ### Node.js (onnxruntime-node — direct ONNX) ```javascript import * as ort from "onnxruntime-node"; const session = await ort.InferenceSession.create( "onnx/model_quantized.onnx", { executionProviders: ["cpu"] } ); // Tokenize with @huggingface/transformers AutoTokenizer, then: const result = await session.run({ input_ids, attention_mask }); // CLS pooling: result.last_hidden_state[0, 0, :1024], then L2-normalize ``` ### GPU Server (CUDA fp16) ```bash pip install sentence-transformers flask python v3_gpu_server.py --port 8234 # OpenAI-compatible /v1/embeddings endpoint curl http://localhost:8234/v1/embeddings \ -d '{"input": "hello world", "model": "intelli-embed-v3-gpu"}' ``` ## Files | File | Size | Description | |------|------|-------------| | `model.safetensors` | 2.2 GB | PyTorch fp32 weights (for GPU inference or further fine-tuning) | | `onnx/model_quantized.onnx` | 542 MB | INT8 quantized ONNX (recommended for CPU/Node.js) | | `onnx/model.onnx` + `model.onnx_data` | 2.2 GB | fp32 ONNX (highest precision, large) | | `config.json` | — | Model architecture config | | `tokenizer.json` | 16 MB | Fast tokenizer | | `1_Pooling/config.json` | — | CLS pooling config | ## Citation If you use this model, please cite: ```bibtex @misc{intelli-embed-v3, title={intelli-embed-v3: Multi-Signal Fine-Tuned Embedding Model for Personal Memory Systems}, author={Serhii Seletskyi}, year={2026}, url={https://huggingface.co/serhiiseletskyi/intelli-embed-v3}, note={4-phase fine-tune of Snowflake/snowflake-arctic-embed-l-v2.0 for OpenMemory} } ``` ## License Apache 2.0 (same as base model [Snowflake/snowflake-arctic-embed-l-v2.0](https://huggingface.co/Snowflake/snowflake-arctic-embed-l-v2.0))