Publish Orion T2 Mini SFT Transformers release

README.md

@@ -1,278 +1,54 @@
 ---
-license: other
-license_name: orion-t2-research-model-use-license-1.0
 library_name: transformers
 pipeline_tag: text-generation
 tags:
 - project-prism
-- refract-ai-labs
-- smilyai
 - orion
 - orion-t2
-- small-language-model
-- custom-architecture
 - custom-code
 - causal-lm
 - instruction-tuned
 - conversational
-- sft
 base_model: Project-Prism/Orion-Flagship-Mini-Preview
-
-extra_gated_prompt: |
-  Orion T2 Mini is an experimental research model developed by Refract AI Labs, a sub-branch of SmilyAI.
-
-  Please provide the information below to request access. Your responses will be used to review and manage access to the model.
-
-  Information submitted through this form may be used by Refract AI Labs to contact you regarding your access request or important model-related updates. Please do not include sensitive personal information.
-
-extra_gated_fields:
-  Intended use: text
-  Contact email: text
-  Affiliation or project: text
-  Project description: text
-  Intended use category:
-    type: select
-    options:
-    - Research
-    - Education
-    - Personal experimentation
-    - Benchmarking / evaluation
-    - Application development
-    - Fine-tuning / model research
-    - Commercial use
-    - Other
-  I agree to follow the model license and applicable usage terms: checkbox
-  I understand that Orion T2 Mini is experimental and its outputs may be incorrect or unreliable: checkbox
----
-
-# 🔺 Orion T2 Mini
-
-> **The first major model release from Refract AI Labs.**
-
-**Orion T2 Mini** is a compact experimental language model based on Project Prism's **Orion T2 architecture**.
-
-Developed by **Refract AI Labs**, a research sub-branch of SmilyAI, Orion T2 Mini explores how far a small language model can be pushed using an unconventional architecture, sparse specialist components, persistent working state, and recurrent computation.
-
-This checkpoint is the **supervised fine-tuned assistant release** of Orion T2 Mini.
-
-## 🚀 Release
-
-| | Orion T2 Mini |
-|---|---|
-| **Organisation** | Refract AI Labs / SmilyAI |
-| **Architecture** | Orion T2 |
-| **Model type** | Decoder-only causal language model |
-| **Parameters** | ~219M |
-| **Training stage** | Pretraining → Response-only SFT |
-| **SFT examples** | 700,000 |
-| **Primary use** | Conversational text generation |
-| **Framework** | Hugging Face Transformers |
-| **Custom code** | Required |
-| **Generation cache** | Currently disabled |
-
-Orion T2 Mini is the first major public model release from **Refract AI Labs** and the first assistant release representing the Orion T2 generation.
-
 ---
 
-# 🧠 Architecture
-
-Orion T2 Mini is **not a standard dense Transformer**.
-
-The architecture combines several experimental components:
-
-### Universal Cortex
-
-A **12-stage Universal Cortex** performs the model's main sequence processing.
-
-### Procedure Banks
-
-Shared sparse **Procedure Banks** provide specialist computational pathways that can be reused across processing stages.
-
-### Knowledge Vault
-
-A sparse **Knowledge Vault** provides an additional mechanism for storing and retrieving learned representations.
+# 🔺 Orion T2 Mini SFT
 
-### Working State
+**Orion T2 Mini SFT** is the supervised-fine-tuned assistant variant of Project Prism's experimental Orion T2 architecture.
 
-Orion maintains an explicit **Working State** intended to provide persistent intermediate information during processing.
+It starts from the completed Orion T2 Mini base checkpoint and was then trained with response-only SFT on 700,000 examples: 60% general/conversation, 20% math, and 20% code.
 
-### Recurrent Deliberation
+## Chat format
 
-Later processing stages can perform recurrent computation, allowing additional internal processing without simply increasing the number of unique Transformer layers.
-
-These mechanisms are experimental. Their usefulness should be determined through controlled evaluation rather than inferred from architectural complexity alone.
-
----
-
-# 📚 Training
-
-The released assistant checkpoint begins from the completed **Orion T2 Mini base checkpoint**.
-
-It was subsequently trained using **response-only supervised fine-tuning (SFT)** on approximately **700,000 examples**.
-
-The SFT mixture was:
-
-| Category | Share |
-|---|---:|
-| General / Conversation | 60% |
-| Mathematics | 20% |
-| Code | 20% |
-
-Only assistant responses contribute to the SFT training objective.
-
-This allows Orion to learn conversational behaviour without directly optimizing against the user and system portions of each training example.
-
----
-
-# 💬 Chat Format
-
-Use the bundled Hugging Face chat template.
-
-Orion was trained using the literal text role markers:
-
-```text
-<|system|>
-<|user|>
-<|assistant|>
-```
-
-These markers were **not added as tokenizer special tokens**. They are represented as ordinary tokenized text.
-
-### Example
+Use the bundled Hugging Face chat template. The model was trained with literal text role markers `<|system|>`, `<|user|>`, and `<|assistant|>`. Those markers were **not** added as new tokenizer special tokens.
 
 ```python
 import torch
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
 repo = "Project-Prism/Orion-Flagship-Mini-SFT"
-
-tokenizer = AutoTokenizer.from_pretrained(
-    repo,
-    trust_remote_code=True,
-)
-
+tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
 model = AutoModelForCausalLM.from_pretrained(
-    repo,
-    trust_remote_code=True,
-    torch_dtype=torch.bfloat16,
-    device_map="auto",
+    repo, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto"
 )
-
 messages = [
-    {
-        "role": "system",
-        "content": "You are Orion, a helpful AI assistant."
-    },
-    {
-        "role": "user",
-        "content": "Explain why the sky is blue."
-    },
+    {"role": "system", "content": "You are Orion, a helpful AI assistant."},
+    {"role": "user", "content": "Explain why the sky is blue."},
 ]
-
-prompt = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True,
-)
-
-inputs = tokenizer(
-    prompt,
-    return_tensors="pt",
-).to(model.device)
-
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
 with torch.inference_mode():
-    output = model.generate(
-        **inputs,
-        max_new_tokens=128,
-        temperature=0.7,
-        top_p=0.95,
-        do_sample=True,
-        use_cache=False,
-    )
-
-new_tokens = output[
-    0,
-    inputs["input_ids"].shape[1]:
-]
-
-print(
-    tokenizer.decode(
-        new_tokens,
-        skip_special_tokens=True,
-    )
-)
-```
-
----
-
-# ⚡ Inference Notes
-
-Orion T2 uses a custom Transformers compatibility layer and therefore requires:
-
-```python
-trust_remote_code=True
-```
-
-The current implementation uses:
-
-```python
-use_cache=False
+    output = model.generate(**inputs, max_new_tokens=128, temperature=0.7, top_p=0.95, do_sample=True, use_cache=False)
+new_tokens = output[0, inputs["input_ids"].shape[1]:]
+print(tokenizer.decode(new_tokens, skip_special_tokens=True))
 ```
 
-As a result, generation currently **recomputes the prefix for every generated token**.
-
-This can make inference substantially slower than similarly sized conventional Transformer models using an optimized KV cache.
-
-The current implementation should therefore be considered a **research implementation rather than a fully optimized inference runtime**.
-
----
-
-# 📊 Evaluation
-
-Orion T2 Mini is intended to be evaluated across:
-
-- General language understanding
-- Instruction following
-- Reasoning
-- Mathematics
-- Coding
-- Long-context behaviour
-- Knowledge
-- Small-language-model benchmarks
-
-Leaderboard and benchmark results should use clearly documented evaluation configurations wherever possible.
-
-Results from different prompting formats, quantization levels, context lengths, evaluation harnesses, or few-shot settings should not automatically be treated as directly comparable.
-
----
-
-# ⚠️ Experimental Status
-
-**Orion T2 Mini is an experimental research model.**
-
-The architecture intentionally differs from a conventional Transformer, but architectural novelty alone does not establish improved capability or efficiency.
-
-Claims about the benefits of the Universal Cortex, Procedure Banks, Knowledge Vault, Working State, or recurrent deliberation should be tested through matched evaluations and ablations.
-
-The model may:
-
-- Hallucinate incorrect information
-- Produce incorrect mathematical reasoning
-- Generate broken or insecure code
-- Misunderstand instructions
-- Produce unexpected outputs
-- Inherit biases or undesirable behaviours from its training data
-
-Outputs should not be treated as inherently factual or reliable.
-
----
-
-# 🔬 Project Prism
+## Architecture
 
-Orion is developed as part of **Project Prism**, an experimental model-development effort exploring alternative architectures and training approaches for compact language models.
+Orion T2 Mini has approximately 219M total parameters and combines a 12-stage Universal Cortex, shared sparse Procedure Banks, a sparse Knowledge Vault, explicit Working State, and recurrent late-stage deliberation.
 
-**Orion T2 Mini** represents the first major release of the T2 generation from **Refract AI Labs**.
+The current Transformers compatibility layer uses `use_cache=False`, so generation recomputes the prefix each token.
 
-### 🔺 Orion T2
+## Status
 
-**Small model. Experimental architecture. Built to find out what actually works.**
+Experimental Project Prism research model. Architecture claims should be judged with matched evaluations rather than assumed from the design alone.

config.json

@@ -27,7 +27,7 @@
   "rope_theta": 10000.0,
   "router_aux_coef": 0.0,
   "router_z_coef": 0.0,
-  "sequence_length": 1024,
+  "sequence_length": 32768,
   "shared_expert": true,
   "tie_word_embeddings": false,
   "tokenizer_name": "mistralai/Mistral-7B-v0.3",

model-00001-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:66b888c3fec9f66c95c0e4607581047d4919732acc7b668b2ceab349b64fbf25
+oid sha256:4142704bb7316be8d0ae1177556025258a297680a62464721bd2b753aa4e539a
 size 534948792

model-00002-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:907793b1be290a0f8538244dde37b77f8c1831568633ce663181f16939688ce5
+oid sha256:9b0a71acd70a0d3a30aaf1e02b9417c592948d5538a1ea9ba9383ebcddcbbb46
 size 341329548

orion_source.json

@@ -1,9 +1,9 @@
 {
-  "source_repo": "Project-Prism/Orion-Flagship-Mini-SFT",
-  "source_checkpoint": "step-000021875-ffba08df",
-  "source_checkpoint_path": "checkpoints/step-000021875-ffba08df",
-  "step": 21875,
-  "tokens": 335659047,
+  "source_repo": "Project-Prism/Orion-Flagship-Mini-SFT-Phase2",
+  "source_checkpoint": "step-000004817-5d8e15d0",
+  "source_checkpoint_path": "checkpoints/step-000004817-5d8e15d0",
+  "step": 4817,
+  "tokens": 66781963,
   "native_format_version": 2,
   "architecture": "orion_flagship_mini_t2",
   "transformers_custom_code": true,

sft_provenance.json

@@ -9,10 +9,10 @@
     "system_marker": "<|system|>",
     "user_marker": "<|user|>"
   },
-  "destination_repository": "Project-Prism/Orion-Flagship-Mini-SFT",
+  "destination_repository": "Refract-Labs/Orion-Flagship-Mini",
   "ignore_index": -100,
   "model_name": "Orion T2 Mini SFT",
-  "native_checkpoint": "checkpoints/step-000021875-ffba08df",
+  "native_checkpoint": "checkpoints/step-000004817-5d8e15d0",
   "objective": "response-only supervised fine-tuning",
   "project": "Project Prism",
   "prompt_tokens_masked": true,
@@ -34,9 +34,9 @@
       "fraction": 0.2
     }
   },
-  "sft_step": 21875,
-  "source_sft_repository": "Project-Prism/Orion-Flagship-Mini-SFT",
-  "supervised_tokens": 335659047,
+  "sft_step": 4817,
+  "source_sft_repository": "Project-Prism/Orion-Flagship-Mini-SFT-Phase2",
+  "supervised_tokens": 66781963,
   "tokenizer": "mistralai/Mistral-7B-v0.3",
   "transformers_custom_code": true,
   "use_cache": false