RAG stands for retrieval augmented generation, which enables companies to fetch important data relative to a query in a prompt in order to improve its results.
LLMS
-
RAG Emerges as Developer Priority for Model Performance Optimization
By
–
Among all the developers, experts, and sources I talk to, no subject comes up more than RAG. Developers are increasingly working with it as a precursor to fine-tuning to squeeze more performance out of less-powerful models.
-
Pretraining Llama 2 with New Data: Tutorial and Approach
By
–
Instead of training from scratch, you could take the existing Llama 2 base model and pretrain it for a few more epochs on new data and see how it performs. I set up a tutorial here the other day (you may want to swap the dataset depending on your usecase): https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/pretrain_redpajama.md
… -
Summer LLM Developments: Llama 2, CodeLlama, and GPT-4
By
–
Llama 2, CodeLlama, and leaked GPT-4 details. Here's my new write-up on the noteworthy developments around LLMs of this summer so far:
-
Implementing AI: accessibility and RLHF resources
By
–
Still a great list. Today I would add to dive as soon as possible in implementing something yourself since recent AI developments have become so accessible. We still need more good book/ressources on RLHF, maybe @natolambert or @_lewtun will fill this gap soon 🙂
-
ChatGPT Excludes Non-English Languages from AI Revolution
By
–
#ChatGPT Is Cutting Non-English Languages Out of the #AI Revolution https://
bit.ly/3C0Bdo2 via @WIRED #ethics -
GPT-3.5 vs GPT-4 exam results at Wharton School
By
–
Graph of GPT-3.5 vs. GPT-4 results in various exams from Wharton School
-
Code Llama Outperforms Open-Source Solutions on HumanEval MBPP
By
–
We evaluated Code Llama against existing solutions on both HumanEval & MBPP.
– It performed better than open-source, code-specific LLMs & Llama 2.
– Code Llama 34B scored the highest vs other SOTA open solutions on MBPP — on par w/ ChatGPT. More info https://
bit.ly/45JiPwJ -
LIMA dataset recommendation for instruction finetuning with high-quality examples
By
–
Agreed! And there’s LIMA if you are looking for 1k high-quality examples for instruction finetuning:
-
LangChain releases chat loaders for Llama fine-tuning event
By
–
If you are in SF tmrw… there is a great event happening tmrw around finetuning Llama In anticipation of that we released some new loaders to help load chat data into an easy-to-use format: https://
blog.langchain.dev/chat-loaders-f
inetune-a-chatmodel-in-your-voice/
… Join the event here: