Instead of training from scratch, you could take the existing Llama 2 base model and pretrain it for a few more epochs on new data and see how it performs. I set up a tutorial here the other day (you may want to swap the dataset depending on your usecase): https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/pretrain_redpajama.md
…
MACHINE LEARNING
-
Pretraining Llama 2 with New Data: Tutorial and Approach
By
–
-
Staying Current: Side Projects and Research Paper Review
By
–
Good question. I think that 1) working on side projects (where you can apply some of the latest developments) is super motivating for keeping up
2) browsing the recent research papers on arxiv (or my monthly summaries :P) -
Implementing AI: accessibility and RLHF resources
By
–
Still a great list. Today I would add to dive as soon as possible in implementing something yourself since recent AI developments have become so accessible. We still need more good book/ressources on RLHF, maybe @natolambert or @_lewtun will fill this gap soon 🙂
-
GPT-3.5 vs GPT-4 exam results at Wharton School
By
–
Graph of GPT-3.5 vs. GPT-4 results in various exams from Wharton School
-
Half of Neural Network Unused Yet Produces Reasonable Outputs
By
–
that feeling when you find a bug where half of your neural network wasn't even being used, and yet its outputs still look extremely reasonable
-
Code Llama Outperforms Open-Source Solutions on HumanEval MBPP
By
–
We evaluated Code Llama against existing solutions on both HumanEval & MBPP.
– It performed better than open-source, code-specific LLMs & Llama 2.
– Code Llama 34B scored the highest vs other SOTA open solutions on MBPP — on par w/ ChatGPT. More info https://
bit.ly/45JiPwJ -
LIMA dataset recommendation for instruction finetuning with high-quality examples
By
–
Agreed! And there’s LIMA if you are looking for 1k high-quality examples for instruction finetuning:
-
RAG Pipeline Technical Overview and Implementation Guide
By
–
A great overview of what's going on under the hood in a RAG pipeline Thanks for writing this @czue
! -
Python ecosystem shifting to Rust over C++ for performance
By
–
Or more like a competitor to Rust since the Python ecosystem is moving to that instead of C++ (eg see polars and some others)? And for GPU & AI stuff a competitor to Triton/CUDA?
-
Clarification between inference and training processes
By
–
Thanks for tagging me, but that’s only inference not training right?