Instead of training from scratch, you could take the existing Llama 2 base model and pretrain it for a few more epochs on new data and see how it performs. I set up a tutorial here the other day (you may want to swap the dataset depending on your usecase): https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/pretrain_redpajama.md
…
@rasbt
-
Pretraining Llama 2 with New Data: Tutorial and Approach
By
–
-
Staying Current: Side Projects and Research Paper Review
By
–
Good question. I think that 1) working on side projects (where you can apply some of the latest developments) is super motivating for keeping up
2) browsing the recent research papers on arxiv (or my monthly summaries :P) -
Summer LLM Developments: Llama 2, CodeLlama, and GPT-4
By
–
Llama 2, CodeLlama, and leaked GPT-4 details. Here's my new write-up on the noteworthy developments around LLMs of this summer so far:
-
LIMA dataset recommendation for instruction finetuning with high-quality examples
By
–
Agreed! And there’s LIMA if you are looking for 1k high-quality examples for instruction finetuning:
-
Python ecosystem shifting to Rust over C++ for performance
By
–
Or more like a competitor to Rust since the Python ecosystem is moving to that instead of C++ (eg see polars and some others)? And for GPU & AI stuff a competitor to Triton/CUDA?
-
Clarification between inference and training processes
By
–
Thanks for tagging me, but that’s only inference not training right?
-
Potential Competitor to Triton for Python Users
By
–
Yeah, we’ll see. It’s probably not going to be used by the average Python users, but I can see it becoming a competitor for Triton though.
-
GPT-4 Behavior Changes Over Time Pose Transparency Risks
By
–
Yes, like I showed at the bottom of the screenshot, it was basically debunked. However, the point still is that the behavior of GPT-4 changes over time, and it's quite intransparent. Now imagine you are a company who built a service on top of the GPT-4 API
-
Simplifying PyTorch by removing C++ dependencies
By
–
What I am really hoping for is that it will help simplify code bases, though. Maybe we can get rid of the C++ parts of PyTorch this way to make development easier.
-
PyTorch Python overhead and realistic performance gains discussion
By
–
2/2
PyTorch's Python overhead is usually quoted to be at ~10% (compared to PyTorch's C++ API). I think for PyTorch users (or CUDA-dependent packages), the speed advantage is probably more going to be in the single digit range, not 35,000x.
Still super exciting though!