How do you quantize LLMs with tens of billions of parameters while avoiding the sharp drop in performance seen when quantizing models > 6B params?

Listen to @aahmadian_ highlight a correct training optimization recipe (which leads to no drop in performance when quantizing vs. a… pic.twitter.com/iU1mliAjel

— Cohere (@cohere) 19 décembre 2023

How do you quantize LLMs with tens of billions of parameters while avoiding the sharp drop in performance seen when quantizing models > 6B params? Listen to @aahmadian_ highlight a correct training optimization recipe (which leads to no drop in performance when quantizing vs. a

→ View original post on X — @cohere