Mixed Precision GGUF layer quantization of Qwen3-14B by Qwen
Original model: https://huggingface.co/Qwen/Qwen3-14B
The hybrid quant employs different quantization levels on a per layer basis to increased flexibility of trading off performance vs file size. Less parameter bits are used at deep layers and more bits at cortex layers to simultaneously optimize quantized size and model performance. This quant was designed to approximately match IQ4_XS size while using all K-quants for faster CPU processing when partially offloaded.
For this file the layer quants are as follows: (refreshed on 4/20/2020)
Q4_K_L : attn_v = q6_k attn_o = q6_k ffn_d = q6_k
Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K
LAYER_TYPES='[
[0 ,"Q4_K_M"],[1 ,"Q4_K_S"],[2 ,"Q3_K_L"],[3 ,"Q3_K_M"],[4 ,"Q3_K_L"],[5 ,"Q3_K_M"],[6 ,"Q3_K_L"],[7 ,"Q3_K_M"],
[8 ,"Q3_K_L"],[9 ,"Q3_K_M"],[10,"Q3_K_L"],[11,"Q3_K_M"],[12,"Q3_K_L"],[13,"Q3_K_M"],[14,"Q3_K_L"],[15,"Q3_K_M"],
[16,"Q3_K_L"],[17,"Q3_K_L"],[18,"Q3_K_L"],[19,"Q3_K_L"],[20,"Q4_K_S"],[21,"Q3_K_L"],[22,"Q4_K_S"],[23,"Q3_K_L"],
[24,"Q4_K_S"],[25,"Q3_K_L"],[26,"Q4_K_S"],[27,"Q3_K_L"],[28,"Q4_K_S"],[29,"Q4_K_S"],[30,"Q4_K_S"],[31,"Q4_K_S"],
[32,"Q4_K_M"],[33,"Q4_K_S"],[34,"Q4_K_M"],[35,"Q4_K_L"],[36,"Q5_K_S"],[37,"Q5_K_M"],[38,"Q5_K_L"],[39,"Q6_K_S"]
]'
FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high"
These quants were optimized for high reasoning performance across a curated set of test prompts.
A second larger Q6_K_H quant is also available with layers as follows:
Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K
Q6_K_M : attn_v = q8_0 ffn_d = q8_0
Q6_K_L : attn_v = q8_0 attn_o = q8_0 ffn_d = q8_0
LAYER_TYPES='[
[0 ,"Q5_K_S"],[1 ,"Q4_K_M"],[2 ,"Q4_K_S"],[3 ,"Q4_K_S"],[4 ,"Q4_K_S"],[5 ,"Q4_K_S"],[6 ,"Q4_K_S"],[7 ,"Q4_K_S"],
[8 ,"Q4_K_S"],[9 ,"Q4_K_S"],[10,"Q4_K_S"],[11,"Q4_K_S"],[12,"Q4_K_S"],[13,"Q4_K_S"],[14,"Q4_K_S"],[15,"Q4_K_S"],
[16,"Q4_K_M"],[17,"Q4_K_S"],[18,"Q4_K_M"],[19,"Q4_K_S"],[20,"Q4_K_M"],[21,"Q4_K_S"],[22,"Q4_K_M"],[23,"Q4_K_S"],
[24,"Q4_K_M"],[25,"Q4_K_M"],[26,"Q4_K_M"],[27,"Q4_K_M"],[28,"Q5_K_S"],[29,"Q5_K_S"],[30,"Q5_K_S"],[31,"Q5_K_S"],
[32,"Q5_K_M"],[33,"Q5_K_M"],[34,"Q5_K_M"],[35,"Q5_K_M"],[36,"Q5_K_L"],[37,"Q6_K_S"],[38,"Q6_K_M"],[39,"Q6_K_L"]
]'
FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"
Q6_K_H uses a minimum quant of Q4_K_S across layers, Q6_K embeddings and output, and strong Q6_K_L final layer. It was also optimized for strong performance across a curated set of test prompts and sizes at ~2.4G smaller than Q6_K.
Comparison:
| Quant | size | PPL | Comment |
|---|---|---|---|
| IQ4_XS | 8.18e9 | 8.81 | default embed and output |
| Q4_K_H | 8.47e9 | 8.98 | Q4_K embed Q6_K output |
| Q6_K | 12.1e9 | 8.76 | default embed and output |
| Q6_K_H | 9.72e9 | 8.75 | Q6_K embed Q6_K output |
Usage:
This is a dense RL model. By default it will crank out a think block delimited by
THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"
To bypass thinking inject the think block delimiters following the assistant prompt template. The model is strong with think blocked bypassed but less accurate on harder prompts.
This model can run fully offloaded on a 12G VRAM GPU. Q8 KV cache can be used to expand context. The model can be speculated with Qwen3 0.6B.
High context yarn config is as follows: Arbitrarily set base context for yarn rope scale compute to 40000 (vs a training context of 40960), then with a context of NCTX tokens the rope scale = NCTX / 40000. Note: this factor can be tweaked on a per application or even per prompt basis. For instance testing with other qwen3 128k yarn models showed 35840 worked best on a long context prompt, but this model appears to work better using 40000 together with the Q6_K_H quant.
Example: on 12G VRAM, NCTX maxes out at 41600 tokens for all weightsin GPU. Then rope scale = 41600.0 / 40000.0 = 1.04
Then on model start pass -c 41600 --rope-scaling yarn --yarn-orig-ctx 40000 --rope_scale 1.04
On 2X 4070 (1 RPC) a full Q8_0 128k context is available. Then use rope scale = 131072 / 40000.0 = 3.2768. The Q4_K_H model does not handle the long context prompt https://thireus.com/REDDIT/Qwen3_Runescape_Massive_Prompt.txt correctly, although it does pass a basic needle in haystack long context test. The Q6_K_H model configured with Q8_0 KV cache, 128k kv cache size, and rope scale as defined running on 2X 4070 on llama.cpp b8855 does handle this prompt correctly with greedy decode and no speculation:
lm Qwen3_Runescape_Massive_Prompt.txt
<think>
Okay, let's see. The user is asking about the level in RuneScape when you reach half the XP of the max level. First, I need to figure out what the max level is. From the provided text, there's a section in the "Skills/Experience Table" that lists levels and the experience needed to reach each level. The highest level mentioned is 99, which requires 3,000,000 experience. So, half of that would be 1,500,000 XP.
Now, I need to check the table to see which level corresponds to 1,500,000 XP. Let me go through the table. The experience needed to reach level 99 is 3,000,000. The level before that is 98, which needs 2,900,000 XP. Wait, but the user is asking for half the XP of max level, which is 1,500,000. So, I need to find the level where the cumulative experience is around 1,500,000.
Looking at the table, let's go step by step. The experience needed to reach level 1 is 0. Level 2 needs 83, level 3 needs 174, and so on. But the table shows the experience needed to reach each level, not the cumulative. So, to find the level where the total experience is 1,500,000, I need to sum up the experience differences from each level.
Alternatively, maybe there's a formula or a specific level mentioned. Wait, in the "Skills/Experience Table" section, the last entry is level 99 with 3,000,000 XP. The user is asking for half of that, which is 1,500,000. So, I need to find the level where the cumulative experience is 1,500,000.
But the table only lists the experience needed to reach each level, not the total. For example, level 1 requires 0 XP, level 2 requires 83 XP (so total XP to reach level 2 is 83), level 3 requires 174 XP (total XP to reach level 3 is 83 + 174 = 257?), but actually, the "Experience needed to reach level" column probably represents the total XP required to get to that level. Wait, looking at the table, the "Experience needed to reach level" is the total XP required. For example, level 1 is 0, level 2 is 83, level 3 is 174, which is 83 + 91 (the difference from level 2 to 3 is 91). Wait, the third column is "Experience difference from last level". So, the total XP to reach a level is the sum of all the differences up to that level.
But the user is asking for when you have half the XP of the max level. The max level is 99, which requires 3,000,000 XP. Half of that is 1,500,000. So, I need to find the level where the total XP is 1,500,000.
Looking at the table, let's check the cumulative XP. Let's start from the top:
Level 1: 0
Level 2: 83
Level 3: 174 (83 + 91)
Level 4: 276 (174 + 102)
Level 5: 388 (276 + 112)
Level 6: 512 (388 + 124)
Level 7: 650 (512 + 138)
Level 8: 801 (650 + 151)
Level 9: 969 (801 + 168)
Level 10: 1,154 (969 + 185)
Level 11: 1,358 (1,154 + 204)
Level 12: 1,584 (1,358 + 226)
Level 13: 1,833 (1,584 + 249)
Level 14: 2,107 (1,833 + 274)
Level 15: 2,411 (2,107 + 304)
Level 16: 2,746 (2,411 + 335)
Level 17: 3,115 (2,746 + 369)
Level 18: 3,523 (3,115 + 408)
Level 19: 3,973 (3,523 + 450)
Level 20: 4,470 (3,973 + 497)
Level 21: 5,018 (4,470 + 548)
Level 22: 5,624 (5,018 + 606)
Level 23: 6,291 (5,624 + 667)
Level 24: 7,028 (6,291 + 737)
Level 25: 7,842 (7,028 + 814)
Level 26: 8,740 (7,842 + 898)
Level 27: 9,730 (8,740 + 990)
Level 28: 10,824 (9,730 + 1,094)
Level 29: 12,031 (10,824 + 1,207)
Level 30: 13,363 (12,031 + 1,332)
Level 31: 14,833 (13,363 + 1,470)
Level 32: 16,456 (14,833 + 1,623)
Level 33: 18,247 (16,456 + 1,791)
Level 34: 20,224 (18,247 + 1,977)
Level 35: 22,406 (20,224 + 2,182)
Level 36: 24,815 (22,406 + 2,409)
Level 37: 27,473 (24,815 + 2,658)
Level 38: 30,408 (27,473 + 2,935)
Level 39: 33,648 (30,408 + 3,240)
Level 40: 37,224 (33,648 + 3,576)
Level 41: 41,171 (37,224 + 3,947)
Level 42: 45,529 (41,171 + 4,358)
Level 43: 50,339 (45,529 + 4,810)
Level 44: 55,649 (50,339 + 5,310)
Level 45: 61,512 (55,649 + 5,863)
Level 46: 67,983 (61,512 + 6,471)
Level 47: 75,127 (67,983 + 7,144)
Level 48: 83,014 (75,127 + 7,887)
Level 49: 91,721 (83,014 + 8,707)
Level 50: 101,333 (91,721 + 9,612)
Level 51: 111,945 (101,333 + 10,612)
Level 52: 123,660 (111,945 + 11,715)
Level 53: 136,594 (123,660 + 12,934)
Level 54: 150,872 (136,594 + 14,278)
Level 55: 166,636 (150,872 + 15,764)
Level 56: 184,040 (166,636 + 17,404)
Level 57: 203,254 (184,040 + 19,214)
Level 58: 224,466 (203,254 + 21,212)
Level 59: 247,886 (224,466 + 23,420)
Level 60: 273,742 (247,886 + 25,856)
Level 61: 302,288 (273,742 + 28,546)
Level 62: 333,804 (302,288 + 31,516)
Level 63: 368,599 (333,804 + 34,795)
Level 64: 407,015 (368,599 + 38,416)
Level 65: 449,428 (407,015 + 42,413)
Level 66: 496,254 (449,428 + 46,826)
Level 67: 547,953 (496,254 + 51,699)
Level 68: 605,032 (547,953 + 57,079)
Level 69: 668,051 (605,032 + 63,019)
Level 70: 737,627 (668,051 + 69,576)
Level 71: 814,445 (737,627 + 76,818)
Level 72: 899,257 (814,445 + 84,812)
Level 73: 992,895 (899,257 + 93,638)
Level 74: 1,096,278 (992,895 + 103,383)
Level 75: 1,210,421 (1,096,278 + 114,143)
Level 76: 1,336,443 (1,210,421 + 126,022)
Level 77: 1,475,581 (1,336,443 + 139,138)
Level 78: 1,629,200 (1,475,581 + 153,619)
Level 79: 1,798,808 (1,629,200 + 169,608)
Level 80: 1,986,068 (1,798,808 + 187,260)
Level 81: 2,192,818 (1,986,068 + 206,750)
Level 82: 2,421,087 (2,192,818 + 228,269)
Level 83: 2,673,114 (2,421,087 + 252,027)
Level 84: 2,951,373 (2,673,114 + 278,259)
Level 85: 3,258,594 (2,951,373 + 307,221)
Level 86: 3,597,792 (3,258,594 + 339,198)
Level 87: 3,972,294 (3,597,792 + 374,502)
Level 88: 4,385,776 (3,972,294 + 413,482)
Level 89: 4,842,295 (4,385,776 + 456,519)
Level 90: 5,346,332 (4,842,295 + 504,037)
Level 91: 5,902,831 (5,346,332 + 556,499)
Level 92: 6,517,253 (5,902,831 + 614,422)
Level 93: 7,195,629 (6,517,253 + 678,376)
Level 94: 7,944,614 (7,195,629 + 748,985)
Level 95: 8,771,558 (7,944,614 + 826,944)
Level 96: 9,684,577 (8,771,558 + 913,019)
Level 97: 10,692,629 (9,684,577 + 1,008,052)
Level 98: 11,805,606 (10,692,629 + 1,112,977)
Level 99: 13,034,431 (11,805,606 + 1,228,825)
Wait, the max level is 99, and the total XP needed is 13,034,431. But the user mentioned 3,000,000 XP. Hmm, maybe there's a mistake here. Wait, looking back at the "Skills/Experience Table" section, the last entry is level 99 with 13,034,431 experience needed. But the user said "half the xp of max level" which would be 13,034,431 / 2 = 6,517,215.5. So, looking for the level where the cumulative XP is around 6,517,215.5.
Looking at the table, level 92 requires 6,517,253 XP. That's very close to half of the max XP. So, the answer would be level 92. But wait, the user might have confused the max XP with 3,000,000. However, according to the table, the max level is 99 with 13 million XP. So, half of that is around 6.5 million. The level that requires 6,517,253 XP is level 92. Therefore, the answer is level 92.
</think>
In Old School RuneScape (OSRS), the maximum level for any skill is **99**, and the total experience required to reach level 99 is **13,034,431** experience points. Half of this amount is **6,517,215.5** experience points.
From the **Skills/Experience Table** provided, the level corresponding to **6,517,253** experience (very close to half the max level XP) is **Level 92**.
### Answer:
**Level 92** is the level at which you reach approximately half the experience required to reach the maximum level (99) in RuneScape.
Later versions of llama.cpp have a bug which soft caps context length to the training context, effectively disabling yarn context extension. Patch server-context.cpp according to https://github.com/ggml-org/llama.cpp/issues/22140 to fix it.
Partial evals for the model (prior to quant refresh) are given here: https://huggingface.co/spaces/steampunque/benchlm.
Download the file from below:
| Link | Type | Size/e9 B | Notes |
|---|---|---|---|
| Qwen3-14B.Q4_K_H.gguf | Q4_K_H | 8.47e9 B | ~IQ4_XS size |
| Qwen3-14B.Q6_K_H.gguf | Q6_K_H | 9.72e9 B | 2.4G smaller than Q6_K |
A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:
- Downloads last month
- 23
6-bit