Upload AgentRank model

README.md

@@ -0,0 +1,390 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- sentence-transformers
+- embeddings
+- retrieval
+- agents
+- memory
+- rag
+- semantic-search
+- ai-agents
+- llm-memory
+- vector-search
+library_name: transformers
+pipeline_tag: sentence-similarity
+datasets:
+- custom
+metrics:
+- mrr
+- recall
+- ndcg
+model-index:
+- name: agentrank-base
+  results:
+  - task:
+      type: retrieval
+      name: Agent Memory Retrieval
+    metrics:
+    - type: mrr
+      value: 0.6496
+      name: MRR
+    - type: recall
+      value: 0.4440
+      name: Recall@1
+    - type: recall
+      value: 0.9960
+      name: Recall@5
+    - type: ndcg
+      value: 0.6786
+      name: NDCG@10
+---
+
+<div align="center">
+
+# 🧠 AgentRank-Base
+
+### The First Embedding Model Built Specifically for AI Agent Memory Retrieval
+
+<p>
+  <img src="https://img.shields.io/badge/MRR-0.65-brightgreen?style=for-the-badge" alt="MRR">
+  <img src="https://img.shields.io/badge/Recall%405-99.6%25-blue?style=for-the-badge" alt="Recall@5">
+  <img src="https://img.shields.io/badge/Parameters-149M-orange?style=for-the-badge" alt="Parameters">
+  <img src="https://img.shields.io/badge/License-Apache%202.0-green?style=for-the-badge" alt="License">
+</p>
+
+**+23% MRR improvement over general-purpose embedders** | **Temporal awareness** | **Memory type understanding**
+
+[🚀 Quick Start](#-quick-start) • [📊 Benchmarks](#-benchmarks) • [🔧 Architecture](#-architecture) • [💡 Why AgentRank?](#-why-agentrank)
+
+</div>
+
+---
+
+## 🎯 TL;DR
+
+> **AgentRank-Base** is an embedding model designed for AI agents that need to remember. Unlike generic embedders (OpenAI, Cohere, MiniLM), AgentRank understands:
+> - ⏰ **When** something happened (temporal awareness)
+> - 📁 **What type** of memory it is (episodic vs semantic vs procedural)
+> - ⭐ **How important** the memory is
+
+---
+
+## 💡 Why AgentRank?
+
+### The Problem with Current Embedders
+
+AI agents need memory. But when you ask an agent:
+
+> *"What did we discuss about Python **yesterday**?"*
+
+Current embedders fail because they:
+- ❌ Don't understand "yesterday" means recent time
+- ❌ Can't distinguish between a preference and an event
+- ❌ Treat all memories as equally important
+
+### The AgentRank Solution
+
+| Challenge | OpenAI/Cohere/MiniLM | AgentRank |
+|-----------|---------------------|-----------|
+| "What did I say **yesterday**?" | Random old results 😕 | Recent memories first ✅ |
+| "What's my **preference**?" | Mixed with events 😕 | Only preferences ✅ |
+| "What's **most important**?" | No priority 😕 | Importance-aware retrieval ✅ |
+
+---
+
+## 📊 Benchmarks
+
+Evaluated on **AgentMemBench** (500 test samples, 8 candidates each):
+
+| Model | Parameters | MRR ↑ | Recall@1 ↑ | Recall@5 ↑ | NDCG@10 ↑ |
+|-------|------------|-------|------------|------------|-----------|
+| **AgentRank-Base** | 149M | **0.6496** | **0.4440** | **0.9960** | **0.6786** |
+| AgentRank-Small | 33M | 0.6375 | 0.4460 | 0.9740 | 0.6797 |
+| all-mpnet-base-v2 | 109M | 0.5351 | 0.3660 | 0.7960 | 0.6335 |
+| all-MiniLM-L6-v2 | 22M | 0.5297 | 0.3720 | 0.7520 | 0.6370 |
+
+### Improvement Over Baselines
+
+| vs Baseline | MRR | Recall@1 | Recall@5 |
+|-------------|-----|----------|----------|
+| vs MiniLM | **+22.6%** | **+19.4%** | **+32.4%** |
+| vs MPNet | **+21.4%** | **+21.3%** | **+25.1%** |
+
+---
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Basic Usage
+
+```python
+from transformers import AutoModel, AutoTokenizer
+import torch
+
+# Load model and tokenizer
+model = AutoModel.from_pretrained("vrushket/agentrank-base")
+tokenizer = AutoTokenizer.from_pretrained("vrushket/agentrank-base")
+
+def encode(texts, model, tokenizer):
+    """Encode texts to embeddings."""
+    inputs = tokenizer(
+        texts, 
+        padding=True, 
+        truncation=True, 
+        max_length=512,
+        return_tensors="pt"
+    )
+    with torch.no_grad():
+        outputs = model(**inputs)
+        # Mean pooling
+        embeddings = outputs.last_hidden_state.mean(dim=1)
+        # L2 normalize
+        embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
+    return embeddings
+
+# Your agent's memories
+memories = [
+    "User prefers Python over JavaScript for backend development",
+    "User asked about React frameworks yesterday",
+    "User mentioned they have 3 years of coding experience",
+    "User is working on an e-commerce project",
+]
+
+# A query from the user
+query = "What programming language does the user prefer?"
+
+# Encode everything
+memory_embeddings = encode(memories, model, tokenizer)
+query_embedding = encode([query], model, tokenizer)
+
+# Find most similar memory
+similarities = torch.mm(query_embedding, memory_embeddings.T)[0]
+best_match_idx = similarities.argmax().item()
+
+print(f"Query: {query}")
+print(f"Best match: {memories[best_match_idx]}")
+print(f"Similarity: {similarities[best_match_idx]:.4f}")
+
+# Output:
+# Query: What programming language does the user prefer?
+# Best match: User prefers Python over JavaScript for backend development
+# Similarity: 0.8234
+```
+
+### Advanced Usage with Metadata
+
+For full temporal and memory type awareness, use the AgentRank package:
+
+```python
+# Coming soon: pip install agentrank
+from agentrank import AgentRankEmbedder
+
+model = AgentRankEmbedder.from_pretrained("vrushket/agentrank-base")
+
+# Encode with temporal context
+memory_embedding = model.encode(
+    text="User mentioned they prefer morning meetings",
+    days_ago=7,           # Memory is 1 week old
+    memory_type="semantic" # It's a preference (not an event)
+)
+
+# Encode query (no metadata needed for queries)
+query_embedding = model.encode("When does the user like to have meetings?")
+
+# The model now knows this is a week-old preference!
+similarity = torch.cosine_similarity(query_embedding, memory_embedding, dim=0)
+```
+
+---
+
+## 🔧 Architecture
+
+AgentRank-Base is built on **ModernBERT-base** (110M params) with novel additions:
+
+```
+┌─────────────────────────────────────────────────┐
+│     ModernBERT Encoder (22 Transformer Layers)  │
+│     - RoPE Positional Encoding                  │
+│     - Flash Attention                           │
+│     - 768 Hidden Dimension                      │
+└─────────────────────────────────────────────────┘
+                       │
+       ┌───────────────┼───────────────┐
+       ↓               ↓               ↓
+┌─────────────┐ ┌─────────────┐ ┌─────────────┐
+│  Temporal   │ │  Memory     │ │ Importance  │
+│  Position   │ │  Type       │ │ Prediction  │
+│  Embeddings │ │  Embeddings │ │ Head        │
+│  (10 × 768) │ │  (4 × 768)  │ │ (768→1)     │
+└─────────────┘ └─────────────┘ └─────────────┘
+       │               │               │
+       └───────────────┼───────────────┘
+                       ↓
+          ┌─────────────────────┐
+          │  Projection Layer   │
+          │  (768 → 768)        │
+          └─────────────────────┘
+                       ↓
+          ┌─────────────────────┐
+          │  L2 Normalization   │
+          │  768-dim Embedding  │
+          └─────────────────────┘
+```
+
+### Novel Components
+
+| Component | Purpose | How It Helps |
+|-----------|---------|--------------|
+| **Temporal Embeddings** | Encodes memory age (today, this week, last month, etc.) | "Yesterday" queries match recent memories |
+| **Memory Type Embeddings** | Distinguishes episodic/semantic/procedural | "What do I like?" matches preferences, not events |
+| **Importance Head** | Auxiliary task predicting memory priority | Helps learn better representations |
+
+### Temporal Buckets
+
+```
+Bucket 0: Today (0-1 days)
+Bucket 1: Recent (1-3 days)
+Bucket 2: This week (3-7 days)
+Bucket 3: Last week (7-14 days)
+Bucket 4: This month (14-30 days)
+Bucket 5: Last month (30-60 days)
+Bucket 6: Few months (60-90 days)
+Bucket 7: Half year (90-180 days)
+Bucket 8: This year (180-365 days)
+Bucket 9: Long ago (365+ days)
+```
+
+### Memory Types
+
+```
+Type 0: Episodic   → Events, conversations ("We discussed X yesterday")
+Type 1: Semantic   → Facts, preferences ("User likes Python")
+Type 2: Procedural → Instructions ("To deploy, run npm build")
+Type 3: Unknown    → Fallback
+```
+
+---
+
+## 🎓 Training Details
+
+| Aspect | Details |
+|--------|---------|
+| **Base Model** | answerdotai/ModernBERT-base (110M params) |
+| **Training Data** | 500K synthetic agent memory samples |
+| **Memory Distribution** | Episodic (40%), Semantic (35%), Procedural (25%) |
+| **Loss Function** | Multiple Negatives Ranking Loss + Importance MSE |
+| **Hard Negatives** | 7 per sample (5 types: temporal, type confusion, topic drift, etc.) |
+| **Batch Size** | 16-32 per GPU |
+| **Hardware** | 2× NVIDIA RTX 6000 Ada (48GB each) |
+| **Training Time** | ~12 hours |
+| **Precision** | FP16 Mixed Precision |
+| **Final Val Loss** | 0.877 |
+
+---
+
+## 🏗️ Use Cases
+
+### 1. AI Agents with Long-Term Memory
+
+```python
+# Store memories with metadata
+agent.remember(
+    text="User is allergic to peanuts",
+    memory_type="semantic",
+    importance=10,  # Critical medical info!
+)
+
+# Later, when discussing food...
+relevant_memories = agent.recall("What should I know about the user's diet?")
+# Returns: "User is allergic to peanuts" (even if stored months ago)
+```
+
+### 2. RAG Systems for Conversational AI
+
+```python
+# Better retrieval for chatbots
+query = "What did we talk about in our last meeting?"
+# AgentRank returns recent, relevant conversations
+# Generic embedders return random topically-similar docs
+```
+
+### 3. Personal Knowledge Bases
+
+```python
+# User's notes and preferences
+memories = [
+    "I prefer dark mode in all apps",
+    "My morning routine starts at 6 AM",
+    "Important: Tax deadline April 15",
+]
+# AgentRank properly handles time-sensitive queries
+```
+
+---
+
+## 🆚 When to Use AgentRank vs Others
+
+| Use Case | Best Model |
+|----------|------------|
+| **AI agents with memory** | ✅ AgentRank |
+| **Time-sensitive retrieval** | ✅ AgentRank |
+| **Conversational AI** | ✅ AgentRank |
+| General document search | OpenAI / Cohere |
+| Code search | CodeBERT |
+| Scientific papers | SciBERT |
+
+---
+
+## 📁 Model Family
+
+| Model | Parameters | Speed | Quality | Best For |
+|-------|------------|-------|---------|----------|
+| [agentrank-small](https://huggingface.co/vrushket/agentrank-small) | 33M | ⚡⚡⚡ Fast | Good | Real-time agents, edge |
+| **agentrank-base** | 149M | ⚡⚡ Medium | **Best** | Quality-critical apps |
+| agentrank-reranker (coming) | 149M | ⚡ Slower | Superior | Two-stage retrieval |
+
+---
+
+## 📚 Citation
+
+```bibtex
+@misc{agentrank2024,
+  author = {Vrushket More},
+  title = {AgentRank: Embedding Models for AI Agent Memory Retrieval},
+  year = {2024},
+  publisher = {HuggingFace},
+  url = {https://huggingface.co/vrushket/agentrank-base}
+}
+```
+
+---
+
+## 🤝 Community & Support
+
+- 🐛 **Issues**: [GitHub Issues](https://github.com/vrushket/agentrank/issues)
+- 💬 **Discussions**: [HuggingFace Community](https://huggingface.co/vrushket/agentrank-base/discussions)
+- 📧 **Contact**: [email protected]
+
+---
+
+## 📄 License
+
+Apache 2.0 - **Free for commercial use!**
+
+---
+
+<div align="center">
+
+### ⭐ If AgentRank helps your project, please star the repo!
+
+**Built with ❤️ for the AI agent community**
+
+</div>

agentrank_components.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c3985df95f03ab63d1def7e8b69f1dec348fb8c8c25698a35c4d7d9ed560ba5
+size 3591708

config.json

@@ -0,0 +1,45 @@
+{
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 10000.0,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "transformers_version": "4.57.3",
+  "vocab_size": 50368
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e6a6ac4d2557c4515c7835287a4e605b1df1ea954392ef200913b89f1a026a2
+size 596070136

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,945 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "|||IP_ADDRESS|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50277": {
+      "content": "|||EMAIL_ADDRESS|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50278": {
+      "content": "|||PHONE_NUMBER|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50279": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50280": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50281": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50282": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50283": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50284": {
+      "content": "[MASK]",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50285": {
+      "content": "[unused0]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50286": {
+      "content": "[unused1]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50287": {
+      "content": "[unused2]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50288": {
+      "content": "[unused3]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50289": {
+      "content": "[unused4]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50290": {
+      "content": "[unused5]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50291": {
+      "content": "[unused6]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50292": {
+      "content": "[unused7]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50293": {
+      "content": "[unused8]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50294": {
+      "content": "[unused9]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50295": {
+      "content": "[unused10]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50296": {
+      "content": "[unused11]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50297": {
+      "content": "[unused12]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50298": {
+      "content": "[unused13]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50299": {
+      "content": "[unused14]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50300": {
+      "content": "[unused15]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50301": {
+      "content": "[unused16]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50302": {
+      "content": "[unused17]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50303": {
+      "content": "[unused18]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50304": {
+      "content": "[unused19]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50305": {
+      "content": "[unused20]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50306": {
+      "content": "[unused21]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50307": {
+      "content": "[unused22]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50308": {
+      "content": "[unused23]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50309": {
+      "content": "[unused24]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50310": {
+      "content": "[unused25]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50311": {
+      "content": "[unused26]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50312": {
+      "content": "[unused27]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50313": {
+      "content": "[unused28]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50314": {
+      "content": "[unused29]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50315": {
+      "content": "[unused30]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50316": {
+      "content": "[unused31]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50317": {
+      "content": "[unused32]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50318": {
+      "content": "[unused33]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50319": {
+      "content": "[unused34]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50320": {
+      "content": "[unused35]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50321": {
+      "content": "[unused36]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50322": {
+      "content": "[unused37]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50323": {
+      "content": "[unused38]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50324": {
+      "content": "[unused39]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50325": {
+      "content": "[unused40]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50326": {
+      "content": "[unused41]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50327": {
+      "content": "[unused42]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50328": {
+      "content": "[unused43]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50329": {
+      "content": "[unused44]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50330": {
+      "content": "[unused45]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50331": {
+      "content": "[unused46]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50332": {
+      "content": "[unused47]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50333": {
+      "content": "[unused48]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50334": {
+      "content": "[unused49]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50335": {
+      "content": "[unused50]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50336": {
+      "content": "[unused51]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50337": {
+      "content": "[unused52]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50338": {
+      "content": "[unused53]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50339": {
+      "content": "[unused54]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50340": {
+      "content": "[unused55]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50341": {
+      "content": "[unused56]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50342": {
+      "content": "[unused57]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50343": {
+      "content": "[unused58]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50344": {
+      "content": "[unused59]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50345": {
+      "content": "[unused60]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50346": {
+      "content": "[unused61]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50347": {
+      "content": "[unused62]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50348": {
+      "content": "[unused63]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50349": {
+      "content": "[unused64]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50350": {
+      "content": "[unused65]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50351": {
+      "content": "[unused66]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50352": {
+      "content": "[unused67]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50353": {
+      "content": "[unused68]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50354": {
+      "content": "[unused69]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50355": {
+      "content": "[unused70]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50356": {
+      "content": "[unused71]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50357": {
+      "content": "[unused72]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50358": {
+      "content": "[unused73]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50359": {
+      "content": "[unused74]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50360": {
+      "content": "[unused75]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50361": {
+      "content": "[unused76]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50362": {
+      "content": "[unused77]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50363": {
+      "content": "[unused78]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50364": {
+      "content": "[unused79]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50365": {
+      "content": "[unused80]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50366": {
+      "content": "[unused81]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50367": {
+      "content": "[unused82]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}