Code Llama was just released 4 days ago. Since then, we already got 1) WizardCoder-34B (
https://
github.com/nlpxucan/Wizar
dLM/tree/main/WizardCoder
…)
2) Phind's finetuned CodeLLama-34B (
https://
phind.com/blog/code-llam
a-beats-gpt4
…) *Both reported to be surpassing GPT-4 on HumanEval. The open source community is amazing!
@rasbt
-
Code Llama Variants Surpass GPT-4 on HumanEval
By
–
-
Academic Contributions to AI Despite Resource Constraints
By
–
Probably 90% of academia operates under that constraint, and lots of useful insights and contributions (although maybe not models) still come from academia. Alpaca, LIMA, FlashAttention, QLoRA … to name a few innovations from academia.
-
Human Preference Evaluation as LLM Gold Standard Despite Its Limitations
By
–
2/2 So, the gold standard remains human preference evaluation, which is expensive and difficult to automate and scale. But even human preference evaluation has its flaws. E.g., see The False Promise of Imitating Proprietary LLMs (
https://
arxiv.org/abs/2305.15717). -
Benchmark Overfitting Problem in Machine Learning Research
By
–
The problem is we still don't have any good benchmarks. And the time a benchmark dataset exists is strongly correlated with the rate of overfitting on these benchmarks. Plus, there are also little accidents like this: https://
x.com/Shahules786/st
atus/1695493641610133600?s=20
… 1/2 -
AI Protein Folding: Structure Prediction, Not Process Simulation
By
–
AI is not capable of protein folding though, it can only do the final structure prediction. As an analogy, imagine cooking a meal, the AI can tell you how the final picture of the meal will look like based on a list of ingredients, but it can’t say how to make the meal.
-
Weights compatibility and numerical stability in LoRA fine-tuning
By
–
Yes, I think that's either maybe so that (1) people with older cards can use these weights as well and (2) we maybe have more stability when finetuning with LoRA etc (in case numbers become large)
It's interesting for sure -
Pretraining Llama 2 with New Data: Tutorial and Approach
By
–
Instead of training from scratch, you could take the existing Llama 2 base model and pretrain it for a few more epochs on new data and see how it performs. I set up a tutorial here the other day (you may want to swap the dataset depending on your usecase): https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/pretrain_redpajama.md
… -
Staying Current: Side Projects and Research Paper Review
By
–
Good question. I think that 1) working on side projects (where you can apply some of the latest developments) is super motivating for keeping up
2) browsing the recent research papers on arxiv (or my monthly summaries :P) -
Summer LLM Developments: Llama 2, CodeLlama, and GPT-4
By
–
Llama 2, CodeLlama, and leaked GPT-4 details. Here's my new write-up on the noteworthy developments around LLMs of this summer so far:
-
LIMA dataset recommendation for instruction finetuning with high-quality examples
By
–
Agreed! And there’s LIMA if you are looking for 1k high-quality examples for instruction finetuning: