QLoRA right now it's single-GPU. I left multi-GPU support as a future todo since I am currently mostly focused on the NeurIPS LLM challenge. But it might well be working with multi-GPU settings already; it's just that I haven't tested. Could be that it requires switching from

→ View original post on X — @rasbt,