You can finetune a 7B Llama 2 model on a single GPU A100 for example, so you could even use Google Colab. But in general, most universities and companies are supportive of open source these days.
@rasbt
-
MoE vs Specialized Models: Questioning Current Practical Utility
By
–
Yeah. But unless you want to have a generalist model for all languages, is it really that useful to have a MoE approach compared to say the CodeLlama-34B-Python version? I mean eventually things might move to MoE but maybe it’s not that big of a deal atm.
-
GPU constraints influence research direction choices
By
–
One might argue that with more GPUs, they would have worked on BLOOM 2 instead of QLoRA etc.
-
Clarification on HumanEval benchmark reference
By
–
Ah I can see how that's misleading, I mean HumanEval as in https://
github.com/openai/human-e
val
… -
Code Llama Variants Surpass GPT-4 on HumanEval
By
–
Code Llama was just released 4 days ago. Since then, we already got 1) WizardCoder-34B (
https://
github.com/nlpxucan/Wizar
dLM/tree/main/WizardCoder
…)
2) Phind's finetuned CodeLLama-34B (
https://
phind.com/blog/code-llam
a-beats-gpt4
…) *Both reported to be surpassing GPT-4 on HumanEval. The open source community is amazing! -
Academic Contributions to AI Despite Resource Constraints
By
–
Probably 90% of academia operates under that constraint, and lots of useful insights and contributions (although maybe not models) still come from academia. Alpaca, LIMA, FlashAttention, QLoRA … to name a few innovations from academia.
-
Human Preference Evaluation as LLM Gold Standard Despite Its Limitations
By
–
2/2 So, the gold standard remains human preference evaluation, which is expensive and difficult to automate and scale. But even human preference evaluation has its flaws. E.g., see The False Promise of Imitating Proprietary LLMs (
https://
arxiv.org/abs/2305.15717). -
Benchmark Overfitting Problem in Machine Learning Research
By
–
The problem is we still don't have any good benchmarks. And the time a benchmark dataset exists is strongly correlated with the rate of overfitting on these benchmarks. Plus, there are also little accidents like this: https://
x.com/Shahules786/st
atus/1695493641610133600?s=20
… 1/2 -
AI Protein Folding: Structure Prediction, Not Process Simulation
By
–
AI is not capable of protein folding though, it can only do the final structure prediction. As an analogy, imagine cooking a meal, the AI can tell you how the final picture of the meal will look like based on a list of ingredients, but it can’t say how to make the meal.
-
Weights compatibility and numerical stability in LoRA fine-tuning
By
–
Yes, I think that's either maybe so that (1) people with older cards can use these weights as well and (2) we maybe have more stability when finetuning with LoRA etc (in case numbers become large)
It's interesting for sure