Awesome, glad to hear that helped. What’s nice is that mixed-precision training it’s so easy to enable but can have a huge impact in practice 🙂
@rasbt
-
Token input limit discussion for model evaluation
By
–
There is a limit on 2048 token inputs for evaluation, is that what you mean?
-
Clarification about 1D convolutional networks misunderstanding
By
–
Yeah. 1D made me think of 1D convolutional networks on pixels arrays hence my weird question. But yeah pls ignore what I said 😛
-
NeurIPS 2023 LLM Efficiency Challenge Quickstart Guide
By
–
Interested in the NeurIPS 2023 LLM Efficiency Challenge or just training your custom LLM on 1 GPU? Wrote a new quickstart guide to get started in ~5 min: https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/neurips_challenge_quickstart.md
… If there's interest, also happy to write an article w. research directions to explore. Let me know! -
UMAP faster than t-SNE despite being slower than PCA
By
–
Saying UMAP is slower than PCA is "the glass is half-empty" view haha.
The more positive take is: UMAP is much faster than T-SNE! -
NeurIPS 2023 LLM Efficiency Challenge and ULMFiT Recommendation
By
–
The NeurIPS 2023 LLM efficiency challenge of course 🙂 https://
llm-efficiency-challenge.github.io Just recommend your Ulmfit paper with @seb_ruder to the students I am collaborating with the other day. Would be looking forward to the solutions you come up with for this one -
OpenLLaMA: An Open Source Large Language Model Alternative
By
–
You might like OpenLLaMA then: https://
github.com/openlm-researc
h/open_llama
… -
Llama-2 Base Model and Dataset Considerations Table
By
–
They use Llama-2 as a base model, so that should be okay. I am not a lawyer and can't give advice on the dataset aspect but here is a table from the paper that might be helpful
-
Theory Behind Q, K, V Matrices in Language Models
By
–
Not sure whether there is any theory behind it vs just empirical observation that it works well. Maybe an intuition is that the Q, K, V matrices work well for language in general, and you don't want to screw them up. Whereas the other ones are more like the extraction params.