We implemented pre-training in LitGPT (e.g., it was used to train TinyLlama back then): https://
github.com/Lightning-AI/l
itgpt/blob/main/litgpt/pretrain.py
…
I would strongly recommend looking at OLMo 2 otherwise: https://
github.com/allenai/OLMo/b
lob/main/olmo/train.py
…
LitGPT Pre-training Implementation and OLMo 2 Recommendation
By
–