Yes! Just added QLoRA-style quantization the other week, too. You can use it via –quantize "bnb.nf4" for example. I have some benchmarks here:

→ View original post on X — @rasbt