@rasbt - AI Dynamics - Page 56 of 95

The CodeLlama PR just got merged: https://
github.com/Lightning-AI/l
it-gpt/pull/472
… When I tried it with bnb's 4-bit Normal Float quantization, the 34B Instruct and Python variants used about 20 Gb for inference:

→ View original post on X — @rasbt

1 September 2023

Efficient LLM Training QLoRA Llama 2 Resource Optimization

By

@rasbt

–

31 August 2023 21h24

Depends on your settings. But if you limit the context size to like 2048 (like in the NeurIPS competition) and use a microbatch size of 1 with gradient accumulation and qlora with llama 2 7B, that’s approx 20 GB RAM and shouldn’t take too long, maybe an hour.

→ View original post on X — @rasbt

31 August 2023

Full Finetuning Missing from Parameter-Efficient Techniques Comparison

By

@rasbt

–

31 August 2023 18h41

That's a really nice figure! I would say in addition to all the parameter-efficient finetuning techniques, one category that is missing is full finetuning. That is, just training the model as is without PEFT.

→ View original post on X — @rasbt

31 August 2023

Community should adopt ‘finetuning’ over ‘fine-tuning’ terminology

By

@rasbt

–

31 August 2023 18h31

That's a nice summary! Hah, one personal pet-peeve: I hope the community is moving from "fine-tuning" to "finetuning" similar to what happened to pretraining 😛

→ View original post on X — @rasbt

31 August 2023

Gradient Accumulation Impact on Training

By

@rasbt

–

31 August 2023 17h05

depends on whether you use gradient accumulation 😛

→ View original post on X — @rasbt

31 August 2023

LIMA Dataset Support Added to Lit-GPT Framework

By

@rasbt

–

31 August 2023 14h30

Finally got around adding LIMA to Lit-GPT (upon popular request). You can use it now to finetune any of the supported LLMs (Llama 2, Falcon, LongChat, … you name it). More usage info here: https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/prepare_dataset.md#lima
…

→ View original post on X — @rasbt

31 August 2023