Bfloat16 or nothing! FWIW – all the models deployed on Hugging Chat are bf16. Quants are good for local/ hobby use – however you always leave perf on the table.
Bfloat16 vs Quantization: Performance Trade-offs in Model Deployment
By
–
By
–
Bfloat16 or nothing! FWIW – all the models deployed on Hugging Chat are bf16. Quants are good for local/ hobby use – however you always leave perf on the table.