Theoretically, you can get it to work if you use deepspeed stage 3 with offloading. But I see there is currently a little bug for stage 3 and LoRA. I think we watned to switch from DeepSpeed to FSDP anyway though. PS: Glad you like my book!

→ View original post on X — @rasbt,