lev: step 18750 of preset 4b-instruct

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/leaderboard.png filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,217 @@
 ---
 license: apache-2.0
+base_model: Qwen/Qwen3.5-4B
+base_model_relation: adapter
+library_name: peft
+pipeline_tag: zero-shot-classification
+inference: false
+language:
+  - en
+tags:
+  - lev
+  - system-one
+  - decision-model
+  - calibrated-decisions
+  - classification
+  - routing
+  - moderation
+  - lora
 ---
+
+# lev
+
+**A calibrated System One decision model.** You give it a **state** (text, a ticket, an email, or JSON) and a set of **typed questions**. It returns typed answers with calibrated probabilities, all in one forward pass (about 69 ms on an H100). It never generates text, so there is nothing to parse and no label outside your option set.
+
+| Question | You give                                              | You get                                                 |
+|---|---|---|
+| `noul`   | a yes/no question                                     | `noul` = p(yes)                                         |
+| `choice` | instructions + options (name → description or `null`) | `choice`, `probabilities`, `confidence`                 |
+| `score`  | instructions + 2–10 ordered levels                    | `score` (expected level), `probabilities`, `confidence` |
+
+The request and response contract matches TypeSafe's `POST /v1/systemone`.
+Code written for the TypeSafe SDK works against lev once you change the
+base URL. It is built for routing, moderation, intent detection, triage,
+grading, and verifying LLM output.
+
+## Installation
+
+```bash
+pip install "lev[serve] @ git+https://github.com/Abhinavexists/lev#subdirectory=packages/lev"
+```
+
+This needs Python 3.12 or newer and, for real-time use, a CUDA GPU. The `serve` extra installs torch, transformers, peft, and the HTTP server. The first load downloads the base model (Qwen/Qwen3.5-4B, about 8 GB) and this adapter (about 200 MB).
+
+## Quickstart
+
+```python
+import lev
+
+model = lev.load("interfaze-ai/lev")
+
+state = "Hi, I was charged twice for my order #4471 and I want a refund."
+questions = {
+    "intent": {
+        "type": "choice",
+        "instructions": "What does the customer want?",
+        "criteria": {
+            "refund": "wants money back",
+            "cancel": "wants to cancel an order",
+            "track": "wants to know where an order is",
+            "other": "anything else",
+        },
+    },
+    "urgent": {"type": "noul", "instructions": "Does this need a human within the hour?"},
+    "frustration": {
+        "type": "score",
+        "instructions": "How frustrated is the customer?",
+        "criteria": ["calm", "mildly annoyed", "annoyed", "angry"],
+    },
+}
+
+result = model.system_one(state, questions)
+print(result.answers["intent"].choice)  # refund
+print(result.answers["intent"].probabilities)
+# {'refund': 0.84, 'cancel': 0.094, 'track': 0.012, 'other': 0.054}
+print(result.answers["urgent"].noul)  # 0.43
+print(result.answers["frustration"].score)  # 1.57, between "mildly annoyed" and "annoyed"
+print(result.usage.output_tokens)  # 0
+```
+
+These are real outputs from this checkpoint. The answers are objects, and `result.model_dump()` gives the same JSON the HTTP server returns.
+
+All the questions share one forward pass, so asking three questions costs about the same as asking one. `lev.load` reads `lev_release.json` from this repository to find the base model and the prompt format the adapter was trained with. It also applies the shipped calibration and loads the matching head, so there is nothing to configure.
+
+Because the probabilities are calibrated, you can gate on them. For example, act automatically above 0.9 and send anything lower to a person.
+
+## Self-hosting: Jev-compatible HTTP server
+
+```bash
+lev serve --checkpoint interfaze-ai/lev --host 0.0.0.0 --port 8000
+```
+
+```bash
+curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
+  "state": "The package arrived crushed and the screen is cracked.",
+  "questions": {"damaged": {"type": "noul", "instructions": "Was the item damaged?"}}
+}'
+```
+
+Existing TypeSafe clients work once you point them at the server:
+
+```python
+from typesafe_sdk import Choice, Noul, TypeSafeClient
+
+# The first request after startup compiles kernels; allow more than the default 10 s.
+client = TypeSafeClient(base_url="http://localhost:8000", api_key="local", timeout=60)
+response = client.system_one(
+    state={"ticket": "The app crashes every time I open settings."},
+    questions={
+        "team": Choice(
+            instructions="Which team owns this?",
+            criteria={"billing": "payments, refunds", "technical": "bugs, crashes", "other": None},
+        ),
+        "bug": Noul(instructions="Is this a bug report?"),
+    },
+)
+print(response.answers["team"].choice, response.answers["bug"].noul)  # technical 0.92
+```
+
+`GET /health` reports the loaded checkpoint, whether calibration is active, and the routing settings. The server batches every question in a request into one forward pass, accepts concurrent requests, and returns 422 with the reason for a malformed question.
+
+## How it works
+
+- **Backbone:** Qwen/Qwen3.5-4B, adapted with LoRA (r=32, α=64) on the q/k/v/o attention and MLP projections.
+- **Label-token readout.** Each option gets a short code, and the answer is read from the next-token logits over those codes. Codes that would split into two tokens are skipped, so every option stays one token. This handles yes/no, scores, and choice lists up to several hundred options. Choices are read in two option orders and averaged, which cancels position bias.
+- **Candidate-path head.** Past that point, a small learned head matches the state against each option's text, so the number of options has no fixed ceiling.
+- **Calibration.** Temperatures fitted after training are applied at load time.
+  There is one per question type, readout mode, and choice option-count band.
+  They were selected by how well they carry over to task families left out of
+  the fit, not only by how well they fit held-out rows.
+
+## Benchmarks
+
+### S1Bench
+
+Six S1Bench subsets, 1,999 items. lev and TypeSafe Jev were run through the same harness on the same task files. Accuracy, best in each row in bold:
+
+| subset | task | **lev** | Jev | Qwen3.5-4B, untuned |
+|---|---|---|---|---|
+| aegis2 | safety moderation | **0.864** | 0.832 | 0.776 |
+| boolq | yes/no reading comprehension | 0.880 | **0.910** | 0.860 |
+| massive-en-US | intent, 60 classes | 0.791 | **0.814** | 0.734 |
+| vitaminc-dev | claim verification | 0.738 | **0.846** | 0.733 |
+| paws | adversarial paraphrase | 0.716 | **0.820** | 0.756 |
+| helpsteer2 | helpfulness rating | 0.360 | 0.304 | **0.400** |
+| **macro** | | 0.725 | **0.754** | 0.710 |
+
+![lev and Jev accuracy on each S1Bench subset](assets/accuracy-by-subset.png)
+
+lev is ahead of Jev on safety moderation and helpfulness rating, within about 3 points on boolq and intent, and furthest behind on the two minimal-edit tasks, paws and vitaminc.
+
+![S1Bench leaderboard: lev ranks sixth of 22](assets/leaderboard.png)
+
+On the S1Bench board over these subsets, lev ranks sixth of 22. Only Jev and three open models of 26B–35B parameters score higher, and lev is ahead of every model its size or smaller.
+
+![Macro accuracy against parameter count](assets/accuracy-per-parameter.png)
+
+### Held-out split
+
+A held-out split of the 29 training sources, with no row shared with training:
+
+| metric                     | value                              |
+|---|---|
+| weighted accuracy          | 0.807\*                            |
+| expected calibration error | 0.061\* (0.180 before calibration) |
+| banking77 (77 intents)     | 0.980                              |
+| clinc_oos (151 intents)    | 0.968                              |
+| FEVER claim verification   | 0.872\*                            |
+
+\* Measured on this checkpoint before the last serving update. That update routes choice sets of more than 68 options to label-token readout and re-selects the temperatures. The banking77 and clinc_oos rows come from after the update, which raised banking77 from 0.818. Smaller option sets are routed the same way as before.
+
+### Speed
+
+|                                                       |                                   |
+|---|---|
+| compute per call, one H100                            | **69 ms**                         |
+| from 1 question to a 60-option choice                 | flat: one forward pass either way |
+| output tokens                                         | 0                                 |
+| end to end from a laptop to a hosted endpoint, median | 414–463 ms (mostly network)       |
+
+![lev compute per call on one H100](assets/compute.png)
+
+The served path is one batched forward pass over every question, with the depthwise-conv kernel. Compute stays flat from one question to eight.
+
+![Per-call latency from the same laptop, lev and Jev](assets/latency.png)
+
+Measured end to end from the same laptop, one request at a time, Jev's hosted API answered in 344–357 ms median and lev, on one Modal H100, in 414–463 ms. Most of lev's round trip is network and Modal's ingress; its compute is 69 ms. Self-hosted next to your application, that network hop disappears.
+
+## Training
+
+- **Data:** 200,000 examples from 29 sources built on 26 public Hugging Face datasets. The tasks cover topic, sentiment, and emotion classification; intent detection (banking77, clinc_oos, snips); NLI (SNLI, ANLI, FEVER); paraphrase (MRPC, QQP, PARADE, plus word-swapped hard negatives); multiple-choice QA (RACE, ARC, SciQ, OpenBookQA, CommonsenseQA, StrategyQA); toxicity and safety (ToxiGen, ToxicChat, BeaverTails); and helpfulness (UltraFeedback). Questions are paraphrased, negated, and recast between types, and option sets are shuffled and resized, so the model learns the question format and not one wording.
+- **Contamination guard:** the data build refuses any source that resolves to one of the 13 S1Bench subsets.
+- **Recipe:** 3 epochs, 18,750 steps, batch size 32, learning rate 5e-5, in the base model's chat format, on one H100.
+
+## Limitations
+
+- **Minimal-edit pairs.** On paws and vitaminc, two inputs can differ by one swapped word or one changed number. Here the model can be confidently wrong. It scores below the untuned backbone on paws and only matches it on vitaminc.
+- **Fine-grained quality ratings are weak.** Helpfulness scoring (helpsteer2) sits at 0.36. Treat such scores as a rough signal.
+- **Calibration is fitted on the training distribution.** Temperatures are chosen to transfer across task families. Even so, a task very unlike the training mix may be less well calibrated. Check on your own data before you gate on the probabilities.
+- **Partial benchmark coverage.** S1Bench results cover 6 of its 13 subsets.
+- **English only.**
+- **Needs a GPU for real-time use.** It runs on CPU, but a 4B backbone there takes seconds per call, not milliseconds.
+
+## Files
+
+| file                                               | role                                                                     |
+|---|---|
+| `adapter_model.safetensors`, `adapter_config.json` | LoRA adapter                                                             |
+| `mode_b_head.pt`                                   | candidate-path head (tensor state dict, loaded with `weights_only=True`) |
+| `tokenizer*`, `chat_template.jinja`                | the tokenizer that the label codes were verified against                 |
+| `calibration.json`                                 | fitted temperatures                                                      |
+| `lev_release.json`                                 | manifest: base model, prompt format, readout, training step              |
+
+## License
+
+The adapter is released under Apache-2.0, the same license as the base model. Some of the training datasets have their own terms, including non-commercial licenses. Review them before commercial use.
+
+Apache 2.0 · Interfaze

adapter_config.json

@@ -0,0 +1,51 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3.5-4B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "kasa_config": null,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.21.0",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "down_proj",
+    "up_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64c718974d7ed0c9b22a0304060669fa3289ff89b57e572d08b9e9a0dc0623c1
+size 169903320

calibration.json

@@ -0,0 +1,23 @@
+{
+  "temperatures": {
+    "choice:B:large": 0.6333014152276044,
+    "choice:B": 0.6333014152276044,
+    "noul:A": 2.3332930831706458,
+    "score:A": 2.8033236630327885,
+    "choice:A": 1.7666659539679779,
+    "choice:A:small": 1.789783450971802,
+    "choice:A:mid": 1.6085685001661532,
+    "choice:A:large": 1.6641112617330986
+  },
+  "fitted_on": "calibration (transfer-selected)",
+  "n_samples": {
+    "choice:B:large": 10227,
+    "choice:B": 10227,
+    "noul:A": 12073,
+    "score:A": 12258,
+    "choice:A": 14133,
+    "choice:A:small": 11480,
+    "choice:A:mid": 1138,
+    "choice:A:large": 1515
+  }
+}

chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

lev_release.json

@@ -0,0 +1,22 @@
+{
+  "name": "lev",
+  "preset": "4b-instruct",
+  "step": 18750,
+  "base_model": "Qwen/Qwen3.5-4B",
+  "lora_rank": 32,
+  "mode_b_head": true,
+  "calibrated": true,
+  "noul_readout": "rating",
+  "prompt_style": "chat",
+  "files": [
+    "adapter_config.json",
+    "adapter_model.safetensors",
+    "chat_template.jinja",
+    "mode_b_head.pt",
+    "tokenizer.json",
+    "tokenizer_config.json",
+    "calibration.json"
+  ],
+  "metrics": {},
+  "created": "2026-09-24T06:29:31+00:00"
+}

mode_b_head.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27eedf7bb9e20d66432882b91297ef6e1ce3d23d81e94ed105ffadba77c6d3ad
+size 14698565

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -0,0 +1,32 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}