Instead of training from scratch, you could take the existing Llama 2 base model and pretrain it for a few more epochs on new data and see how it performs. I set up a tutorial here the other day (you may want to swap the dataset depending on your usecase): https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/pretrain_redpajama.md
…

→ View original post on X — @rasbt,