I used HuggingFace and quantized the model to 4 bits, I believe it ran on a 24GB gpu (maybe need 48)
HuggingFace Model Quantization on 24GB GPU
By
–
By
–
I used HuggingFace and quantized the model to 4 bits, I believe it ran on a 24GB gpu (maybe need 48)