I can see how deep learning methods may struggle to catch up with tree methods on small size datasets (and you say it in the thread). Wondering if you did try to do multitask fine-tuning on eg Transformers and saw a positive benefits? (we observe it in text, see eg Flan/T0)
LLMS
-
Chinchilla Reproduction: Architectural Decisions and Retrospective Improvements
By
–
When we started this project, our goal was to reproduce Chinchilla, so we did not move too much away from the original architecture. Retrospectively, we should probably have done what PALM did: multi query + increase FeedForward hidden size to make up for the loss of parameters
-
Foundation Models for Decision Making: Problems and Opportunities
By
–
Foundation Models for Decision Making: Problems, Methods, and Opportunities Foundations models are engines of large-scale AI research. This is an excellent review paper that discusses various use cases of FM, open problems, challenges and solutions. https://
arxiv.org/abs/2303.04129 -
LLaMA 65B Now Runs on MacBook Pro
By
–
LLaMA 65B can run on a MacBook! With a different model architecture it could probably run quite faster (we didn't use multi query, for instance) https://t.co/SMdlZC5NOo
— Guillaume Lample @ NeurIPS 2024 (@GuillaumeLample) 11 mars 2023LLaMA 65B can run on a MacBook! With a different model architecture it could probably run quite faster (we didn't use multi query, for instance)
-
Ravi’s Contributions: From Gaussians to Large Language Models
By
–
Sanjeev Arora (
@prfsanjeevarora
) next takes us through a great tour of Ravi's many contributions over the years, highlighting some nice works including mixtures of Gaussians, topic modeling, and nonnegative matrix factorization, and ending with some thoughts on LLMs -
Rationale before answer vs rationalization after in prompting
By
–
Only if you ask for (or demonstrate in k-shot) rationales that come before the answer. I see people make that mistake a lot — if it says the explanation after, it’s not a rationale it’s a rationalization.
-
Microsoft Kosmos 1 Elevates ChatGPT with Voice and Visual Commands
By
–
Experience the next level of chatbot innovation with @Microsoft
's Kosmos 1, elevating ChatGPT with improved voice and visual commands. Discover how AI is revolutionising user experiences with this amazing technology here: https://
bit.ly/3FD4hUX @OpenAI @jeevprabnivash -
Old man and scientist discuss alignment of leaked pre-trained model
By
–
[In a world… fully aligned…]
TEENAGER: “Prompt engineer?” What’s that?
OLD MAN: Just words, kid. Just words. […one final task… remains…]
SCIENTIST: The leaked model is pre-trained! There’s no RLHF! Not even instruct!
OLD MAN: *Dons e/acc shades* Let’s think step by step. -
Fixing GPT-3.5 numeric ties with weighted average
By
–
Numeric scores generated in GPT-3.5/ChatGPT snap to discrete increments, causing ties in comparisons. To fix this, try a probability-weighted average using the top-5 token logprobs in GPT-3.5. E.g. here: (50 * .7396 + 60 * .1027 + …) / (.7396 + …)
-
Microsoft hints at multimodal GPT-4 and Visual ChatGPT
By
–
Yesterday, Microsoft Germany CTO dropped a comment about upcoming GPT-4 being multimodal (handling images and text) But the day before, Microsoft published "Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models" Abstract of the https://
arxiv.org/pdf/2303.04671
.pdf
…