The only quant version that actually works
#1
by
AiCreatornator
- opened
I tried many other quants from 4_k_m to Q8 from other uploaders, but those quants produced junk. This is the only one that works for me. Thanks!
Yes, at these small parameters, it's important to use the best quality available, and this BF16, is essentially the original model unquantized.