Much has been said about many companies’ desire for more compute (as well as data) to train larger foundation models. I think it’s under-appreciated that we have nowhere near enough compute available for inference on foundation models as well. Years ago, when I was leading teams
LLMS
-
LangGraph: Language Models Solve Olympiad Programming Benchmark
By
–
LangGraph: Can Language Models Solve Olympiad Programming? Last week, Princeton researchers released the USACO benchmark dataset and showed that a zero-shot GPT-4 agent only passes 8.7% of the questions. We've implemented this paper in LangGraph and created a tutorial
-
Fine-tuning Embedding Models: Why BERT Outperforms Larger T5 Encoders
By
–
question: i’m finetuning embedding models for a retrieval. i found large T5 encoders from GTR and SentenceT5 families significantly underperform BERT base how is this possible? feels it must be hyperparameters? unless i found the one NLP task ever where scaling doesn’t help
-
Apple OpenELM, Tesla AI, Gmail Assistant, Moderna OpenAI Partnership Today
By
–
Top stories in AI today: -Apple unveils OpenELM for on-device AI
-Elon Musk: Tesla is an AI company
-Unlock AI-assisted Gmail writing
-Moderna and OpenAI partner up
-6 new AI tools & 4 new AI jobs Read more: http://
therundown.ai/p/apple-goes-o
pen-source
… -
In-Context Learning Sensitivity to Shot Order in LLMs
By
–
We also do observe some sensitivity to the order in which the shots are presented in the context window (see Figure 13).
-
ICL Robustness: Why In-Context Learning Lacks Monotonic Improvement
By
–
That's a good question! Well, at a minimum I'd say that what we're observing right now is that ICL is not as robust as we'd probably want it to be, as we don't always get the strictly monotonic improvement in performance that you'd expect from a normal learning algorithm.
-
LLMs Develop Unexpected Retrieval Heads for Context Windows
By
–
Fascinating thread in two ways. The first is the paper and findings itself, explaining how LLMs actually deal with context windows. The second is the fact that this is a “discovery:” LLMs have unexpectedly developed retrieval heads, they were not explicitly coded for by creators
-
Llama timeout issues resolved with Mistral model
By
–
I know I was running into the same issue; llama requests time out pretty fast. I had to use the mistral model.
-
Self-Play Training Could Overcome Human Data Limits for LLMs
By
–
Why this matters: human written data & human reinforcement may set the upper bounds on what LLMs can do. AlphaGo was able to beat humans because it trained by playing against itself. This suggests that LLMs may be able to do similar self-play, offering a path to rapid improvement
-
Quantization Evaluation Gap in LLM Model Versions
By
–
I still haven't seen a good evaluation of the differences between different quantized versions of why models to be honest – or even any good anecdotes about things that work and things that don't at different levels So I'm effectively flying blind when it comes to quantization