the returns have been diminishing for a while, that’s certainly true NeoBERT is 250M params but trained on 2T tokens. objectively a crazy thing to do
@jxmnop
-
New Paper on Training Great AI Models Now Available on ArXiv
By
–
and our paper is available on ArXiv now, soon-to-be-published in TMLR. if you're interested — take a read! a lot of great nitty-gritty detail in what it takes to train a great model. http://
arxiv.org/abs/2502.19587 -
NeoBERT Model Released on HuggingFace
By
–
our model is already available on HuggingFace. the implementation is optimized for efficiency so it's not bloated like a lot of other models (wont name names here…) star NeoBERT! download NeoBERT! finetune NeoBERT! http://
huggingface.co/chandar-lab/Ne
oBERT
… -
NeoBERT Achieves State-of-the-Art Performance in MTEB Embeddings
By
–
and as you may have already guessed, being the best small-sized encoder means NeoBERT is a great embedding model. it's state-of-the-art on MTEB for its parameter class (more on this later!):
-
NeoBERT: Iterative Deep Learning Breakthroughs Through Experimentation
By
–
like many important advancements in deep learning, NeoBERT arose from running lots of tiny experiments, learning from them, and stacking the results together into something that works really well:
-
Optimal Depth-Width Ratio Improves Transformer Performance
By
–
the key insight, i think, is using an optimal depth-to-width ratio for the transformer architecture. and training on good data. a lot of good data. even though NeoBERT has slightly more parameters, it's still faster AND more effective than ModernBERT for long sequences:
-
NeoBERT: New State-of-the-Art Small Encoder Model
By
–
In the beginning, there was BERT. Eventually BERT gave rise to RoBERTa. Then, DeBERTa. Later, ModernBERT. And now, NeoBERT. The new state-of-the-art small-sized encoder:
-
RASP Programming Language Compiles Algorithms to Transformer Weights
By
–
another incredibly underrated paper: Thinking Like Transformers (Weiss et al, 2021) presents RASP: a programming language that compiles to transformer *weights*. can implement sort(), bincount(), etc. seems important. why don't interpretability people care about this?
-
Science redefined: artifacts, explanations, and exponential growth
By
–
my definition of science:
the continual process of (a) generating artifacts that are surprising or useful and (b) explaining them these days there's exponential growth in papers, and in useful artifacts, but we don't often discover good explanations so i was happy to stumble -
Differential Privacy Applies to Samples Not Distributions
By
–
no bc differential privacy applies to samples, not distributions