Decent performance with 96GB VRAM / 256GB DDR5
Just jotting down some performance numbers! I tested the IQ1_S and the IQ4_XS.
System Specs:
3x 3090
1x4090
Intel Sapphire Rapids ES QYFS + 256GB DDR5
IQ1_S
| model | size | params | backend | ngl | n_batch | n_ubatch | ot | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | -------: | --------------------- | --------------: | -------------------: |
| grok 314B IQ1_S - 1.5625 bpw | 82.82 GiB | 269.52 B | CUDA,RPC | 99 | 2056 | 2056 | blk\.([0-9]|1[0-2])\.ffn_.*=CUDA0 | pp512 | 263.35 ± 12.73 |
| grok 314B IQ1_S - 1.5625 bpw | 82.82 GiB | 269.52 B | CUDA,RPC | 99 | 2056 | 2056 | blk\.([0-9]|1[0-2])\.ffn_.*=CUDA0 | tg128 | 13.63 ± 0.12 |
IQ4_XS:
(Couldn't get it to run in llama-bench, but it's like 7t/s generation and 115t/s PP)
EDIT 1: Downloaded the TQ1_0 variant, allowing for full GPU offload. Very similar speeds to IQ1_S, but I just have to say, this might be the most solid model I've used at a quantization of 1~ bit. I believe it's due to the larger active parameters or something of the like, but the responses between IQ4_XS, IQ3_XXS, and even Q5_K_S all seem in line with each other, even TQ1_0's responses are very similar when comparing generations.
Normally with models that have smaller active parameters, the more quantized it becomes, I've noticed the responses become a lot more concise and generalized (not as specific)... Of course I'm not using this model for coding or anything too accurate like that, just general knowledge and stuff, but I'm enjoying it! Below is a comparison between IQ_1_S 's initial response, and Q5_K_S's response. Result is very similar.
Thanks for sharing that's really cool