Fascinating thread in two ways. The first is the paper and findings itself, explaining how LLMs actually deal with context windows. The second is the fact that this is a “discovery:” LLMs have unexpectedly developed retrieval heads, they were not explicitly coded for by creators
MACHINE LEARNING
-
Self-Play Training Could Overcome Human Data Limits for LLMs
By
–
Why this matters: human written data & human reinforcement may set the upper bounds on what LLMs can do. AlphaGo was able to beat humans because it trained by playing against itself. This suggests that LLMs may be able to do similar self-play, offering a path to rapid improvement
-
Quantization Evaluation Gap in LLM Model Versions
By
–
I still haven't seen a good evaluation of the differences between different quantized versions of why models to be honest – or even any good anecdotes about things that work and things that don't at different levels So I'm effectively flying blind when it comes to quantization
-
Strong AI Benchmarks Achievement Recognized
By
–
Benchmarks look really good… congrats @erhartford
! -
AI Data Development Critical for Enterprise Machine Learning
By
–
See why AI data development is a 'hot area.' Co-founder and Head of Applied Machine Learning, Paroma Varma, sits down with Fierce Network to discuss the role of data in enterprise AI. Thanks to Diana Goovaerts for the fantastic coverage! https://
buff.ly/4dfxNio -
Combining DDP, bf16, gradcache and accelerate techniques
By
–
lol kind of. it was nontrivial for me to combine DDP, bf16, gradcache, and accelerate. maybe we could work together
-
GradCache Optimization for Distributed GPU Training
By
–
oh yeah. I think his trick was just GradCache. and sharing negatives between GPUs isn’t trivial
-
Framework Features Beyond PyTorch Lightning Trainer
By
–
what would u expect from this kind of framework that u don’t get from eg pytorch lightning trainer?
-
Llama-2 Improves Game Performance Through Adversarial Self-Play
By
–
This is worth noting: it appears to be the first evidence that LLMs can improve performance at a task through self-play. In this case, Llama-2 7B was able to greatly improve its ability to play the game of Adversarial Taboo by training through adversarial competition with itself
-
Meta does what OpenAI should be doing – Discussion
By
–
Meta does everything OpenAI should be Discussion on r/MachineLearning https://redd.it/1cbhec7