There are different stages. The evaluation you mention happens usually during the development stages internally. The "evaluation" that is publicly shared is an evaluation on a different level, the real-world use case task that most users would find relevant.
@rasbt
-
Planning to Write About Reasoning Models with Quality Focus
By
–
Yup, I was planning to eventually write about reasoning models. But as my thing for 2025, I prefer to do it well over doing it quick!
-
Multiple Levels of LLM Evaluation Methods and Approaches
By
–
Wasn't talking about evaluation here. But in general, there's not one way to evaluate LLMs. There are evaluations on multiple levels of course!
-
PyTorch as a Tool: Teaching and Implementation Philosophy
By
–
Small world! I love using PyTorch, but ultimately, it's "just" the tool to get your models implemented :). I taught a lot of PyTorch back then (kind of as a a side project) to help get people started! (Unfortunately, I am currently not mentoring PhD students anymore as I left
-
2025: The Year of Reasoning Models and Agents
By
–
2024 was a great year for building general-purpose LLMs (with some special-purpose finetunes for math & code etc). Looks like 2025 is the year of diverging more into (1) reasoning models (for math and code) and (2) agents (for workflow automation). Exciting times!
-
Upcoming Chapter Content and Bonus Multi-Head Attention Materials
By
–
Yes, stay tuned! (And there's also the "Comparing Efficient Multi-Head Attention Implementations" bonus content if you are still bored after completing the chapter :))
-
Approximately 1 billion people use BPE tokenizers and LLMs
By
–
according to my napkin math, about 1 billion people use BPE tokenizers (/LLMs)
-
Data Augmentation vs Generative Models: Key Differences Explained
By
–
yes, a kind of flavor of it. but also not quite. in traditional data augmentation (say image/cnn contexts) you add noise to the data, rescale it etc. here, the difference is that you actually generate that data. in image/cnn land that would be using a diffusion model to generate
-
Remove whitespace from text before passing to LLM
By
–
Could you remove the whitespace? And then do that on every input before passing it to the LLM later. Should be super cheap computationally:
Like text.replace("Legal Jargon", "LegalJargon")