the returns have been diminishing for a while, that’s certainly true NeoBERT is 250M params but trained on 2T tokens. objectively a crazy thing to do
LLMS
-
New Paper on Training Great AI Models Now Available on ArXiv
By
–
and our paper is available on ArXiv now, soon-to-be-published in TMLR. if you're interested — take a read! a lot of great nitty-gritty detail in what it takes to train a great model. http://
arxiv.org/abs/2502.19587 -
NeoBERT Model Released on HuggingFace
By
–
our model is already available on HuggingFace. the implementation is optimized for efficiency so it's not bloated like a lot of other models (wont name names here…) star NeoBERT! download NeoBERT! finetune NeoBERT! http://
huggingface.co/chandar-lab/Ne
oBERT
… -
NeoBERT: Iterative Deep Learning Breakthroughs Through Experimentation
By
–
like many important advancements in deep learning, NeoBERT arose from running lots of tiny experiments, learning from them, and stacking the results together into something that works really well:
-
NeoBERT: New State-of-the-Art Small Encoder Model
By
–
In the beginning, there was BERT. Eventually BERT gave rise to RoBERTa. Then, DeBERTa. Later, ModernBERT. And now, NeoBERT. The new state-of-the-art small-sized encoder:
-
Optimal Depth-Width Ratio Improves Transformer Performance
By
–
the key insight, i think, is using an optimal depth-to-width ratio for the transformer architecture. and training on good data. a lot of good data. even though NeoBERT has slightly more parameters, it's still faster AND more effective than ModernBERT for long sequences:
-
LLMs now understand physics, disproving skeptics
By
–
One year ago, LLM skeptics were saying "image models do not understand physics and never will", and they would cite as examples messed up images of gymnasts.
— m_ric (@AymericRoucher) 24 juin 2025
Now it appears LLMs can represent complex dynamics accurately. So they must 'understand' physics to some extent.
I think… https://t.co/tSxBkLxwRpOne year ago, LLM skeptics were saying "image models do not understand physics and never will", and they would cite as examples messed up images of gymnasts. Now it appears LLMs can represent complex dynamics accurately. So they must 'understand' physics to some extent. I think
-
Model Updates, Prompt Testing, Edge Cases and AI Security
By
–
Other questions: When do you update models? How are you testing your prompts? Have you tested for edge cases and biases? Can we vet the prompts you use? What happens if an AI service goes down? How are you working within context windows? How are you dealing with prompt injection?
-
Conversational AI Models Replacing Search Engines
By
–
C'est LE PLUS GROS projet que j'ai jamais lancé. 66% des utilisateurs de chatGPT l'utilisent comme leur moteur de recherche principale. Les modèles conversationnels sont en train de remplacer a vitesse grand V les moteurs de recherche. (C'est pas pour rien d'ailleurs que ces
-
Using Prompt Templates to Improve Claude AI Outputs
By
–
Used a few of the template strategies this morning. Results were noticeably sharper. It’s like handing Claude a blueprint instead of a question.