and our paper is available on ArXiv now, soon-to-be-published in TMLR. if you're interested — take a read! a lot of great nitty-gritty detail in what it takes to train a great model. http://
arxiv.org/abs/2502.19587
MACHINE LEARNING
-
New Paper on Training Great AI Models Now Available on ArXiv
By
–
-
NeoBERT Achieves State-of-the-Art Performance in MTEB Embeddings
By
–
and as you may have already guessed, being the best small-sized encoder means NeoBERT is a great embedding model. it's state-of-the-art on MTEB for its parameter class (more on this later!):
-
NeoBERT: Iterative Deep Learning Breakthroughs Through Experimentation
By
–
like many important advancements in deep learning, NeoBERT arose from running lots of tiny experiments, learning from them, and stacking the results together into something that works really well:
-
NeoBERT: New State-of-the-Art Small Encoder Model
By
–
In the beginning, there was BERT. Eventually BERT gave rise to RoBERTa. Then, DeBERTa. Later, ModernBERT. And now, NeoBERT. The new state-of-the-art small-sized encoder:
-
Optimal Depth-Width Ratio Improves Transformer Performance
By
–
the key insight, i think, is using an optimal depth-to-width ratio for the transformer architecture. and training on good data. a lot of good data. even though NeoBERT has slightly more parameters, it's still faster AND more effective than ModernBERT for long sequences:
-
LLMs now understand physics, disproving skeptics
By
–
One year ago, LLM skeptics were saying "image models do not understand physics and never will", and they would cite as examples messed up images of gymnasts.
— m_ric (@AymericRoucher) 24 juin 2025
Now it appears LLMs can represent complex dynamics accurately. So they must 'understand' physics to some extent.
I think… https://t.co/tSxBkLxwRpOne year ago, LLM skeptics were saying "image models do not understand physics and never will", and they would cite as examples messed up images of gymnasts. Now it appears LLMs can represent complex dynamics accurately. So they must 'understand' physics to some extent. I think
-
Sauron Smart Gimbal Reaches Pioneer Build Phase for Illegal Fishing Detection
By
–
Shout-out to Sauron – The Intelligent Gimbal for making it to the Pioneer 10 build phase! This smart gimbal keeps an AI-powered eye (which is NOT wreathed in flame) on the sea, scanning for boats and spotting illegal fishing activity in real time. It’s a vision system with
-
RASP Programming Language Compiles Algorithms to Transformer Weights
By
–
another incredibly underrated paper: Thinking Like Transformers (Weiss et al, 2021) presents RASP: a programming language that compiles to transformer *weights*. can implement sort(), bincount(), etc. seems important. why don't interpretability people care about this?
-
AI Benefits Builders Users Models Cost Worth
By
–
great news for everyone building (lots of customers)
great news for everyone using AI (useful products)
great news for everyone making AI models (large cost is worth it) -
BirdNET-PI and SAS Track 140,000 Backyard Bird Songs
By
–
From chirps to SAS charts. See how SAS' @cjdinger used BirdNET-PI and SAS to track and analyze 140,000 backyard bird songs! http://
2.sas.com/61854bGOL