question: i’m finetuning embedding models for a retrieval. i found large T5 encoders from GTR and SentenceT5 families significantly underperform BERT base how is this possible? feels it must be hyperparameters? unless i found the one NLP task ever where scaling doesn’t help
RESEARCH
-
Mystery AI Hype Theater 3000: Climate Change AI Discussion
By
–
Ready for more Mystery AI Hype Theater 3000? Climate change has reached AI Hell (frozen over last we checked in December) and now we're flooded with nonstop nonsense. Join me and @alexhanna as we wade our way through on Monday April 29, noon Pacific
-
NVIDIA Aerial Omniverse enables 5G 6G R&D digital twin simulation
By
–
Learn how R&D on #5G and #6G and optimized network planning is enabled through NVIDIA Aerial™ Omniverse™ Digital Twin—a next-generation, system-level simulation platform now available in the NVIDIA 6G Research Cloud platform. Explore the platform. https://
nvda.ws/3vYcEc1 -
In-Context Learning Sensitivity to Shot Order in LLMs
By
–
We also do observe some sensitivity to the order in which the shots are presented in the context window (see Figure 13).
-
ICL Robustness: Why In-Context Learning Lacks Monotonic Improvement
By
–
That's a good question! Well, at a minimum I'd say that what we're observing right now is that ICL is not as robust as we'd probably want it to be, as we don't always get the strictly monotonic improvement in performance that you'd expect from a normal learning algorithm.
-
LLMs Develop Unexpected Retrieval Heads for Context Windows
By
–
Fascinating thread in two ways. The first is the paper and findings itself, explaining how LLMs actually deal with context windows. The second is the fact that this is a “discovery:” LLMs have unexpectedly developed retrieval heads, they were not explicitly coded for by creators
-
Copilot’s Impact on Coding Productivity Measured
By
–
Field experiments are great but they can go wrong in so many ways because the real world is messy I appreciate this honest write-up of the difficulties of measuring the impact of Copilot on coding performance. It looks like it lifted productivity, though. https://
mit-genai.pubpub.org/pub/v5iixksv/r
elease/2?readingCollection=7e70f4fe
… -
Self-Play Training Could Overcome Human Data Limits for LLMs
By
–
Why this matters: human written data & human reinforcement may set the upper bounds on what LLMs can do. AlphaGo was able to beat humans because it trained by playing against itself. This suggests that LLMs may be able to do similar self-play, offering a path to rapid improvement
-
Quantization Evaluation Gap in LLM Model Versions
By
–
I still haven't seen a good evaluation of the differences between different quantized versions of why models to be honest – or even any good anecdotes about things that work and things that don't at different levels So I'm effectively flying blind when it comes to quantization
-
Strong AI Benchmarks Achievement Recognized
By
–
Benchmarks look really good… congrats @erhartford
!