i'll have to page @lo_LB_La for engineering details but iirc most of the performance gain comes from the optimized implementation rather than architectural details
@jxmnop
-
Diminishing Returns in Language Model Training Efficiency
By
–
the returns have been diminishing for a while, that’s certainly true NeoBERT is 250M params but trained on 2T tokens. objectively a crazy thing to do
-
New Paper on Training Great AI Models Now Available on ArXiv
By
–
and our paper is available on ArXiv now, soon-to-be-published in TMLR. if you're interested — take a read! a lot of great nitty-gritty detail in what it takes to train a great model. http://
arxiv.org/abs/2502.19587 -
NeoBERT Achieves State-of-the-Art Performance in MTEB Embeddings
By
–
and as you may have already guessed, being the best small-sized encoder means NeoBERT is a great embedding model. it's state-of-the-art on MTEB for its parameter class (more on this later!):
-
NeoBERT Model Released on HuggingFace
By
–
our model is already available on HuggingFace. the implementation is optimized for efficiency so it's not bloated like a lot of other models (wont name names here…) star NeoBERT! download NeoBERT! finetune NeoBERT! http://
huggingface.co/chandar-lab/Ne
oBERT
… -
NeoBERT: Iterative Deep Learning Breakthroughs Through Experimentation
By
–
like many important advancements in deep learning, NeoBERT arose from running lots of tiny experiments, learning from them, and stacking the results together into something that works really well:
-
NeoBERT: New State-of-the-Art Small Encoder Model
By
–
In the beginning, there was BERT. Eventually BERT gave rise to RoBERTa. Then, DeBERTa. Later, ModernBERT. And now, NeoBERT. The new state-of-the-art small-sized encoder:
-
Optimal Depth-Width Ratio Improves Transformer Performance
By
–
the key insight, i think, is using an optimal depth-to-width ratio for the transformer architecture. and training on good data. a lot of good data. even though NeoBERT has slightly more parameters, it's still faster AND more effective than ModernBERT for long sequences:
-
RASP Programming Language Compiles Algorithms to Transformer Weights
By
–
another incredibly underrated paper: Thinking Like Transformers (Weiss et al, 2021) presents RASP: a programming language that compiles to transformer *weights*. can implement sort(), bincount(), etc. seems important. why don't interpretability people care about this?
-
Science redefined: artifacts, explanations, and exponential growth
By
–
my definition of science:
the continual process of (a) generating artifacts that are surprising or useful and (b) explaining them these days there's exponential growth in papers, and in useful artifacts, but we don't often discover good explanations so i was happy to stumble