according to my napkin math, about 1 billion people use BPE tokenizers (/LLMs)
@rasbt
-
Data Augmentation vs Generative Models: Key Differences Explained
By
–
yes, a kind of flavor of it. but also not quite. in traditional data augmentation (say image/cnn contexts) you add noise to the data, rescale it etc. here, the difference is that you actually generate that data. in image/cnn land that would be using a diffusion model to generate
-
Remove whitespace from text before passing to LLM
By
–
Could you remove the whitespace? And then do that on every input before passing it to the LLM later. Should be super cheap computationally:
Like text.replace("Legal Jargon", "LegalJargon") -
Linear Approximations in Modern Machine Learning: Why Simplicity Works
By
–
that's nowadays literally a multi-million dollar question . Btw there are lots of (linear) approximations that came and went in the past few years, but it's still more or less the same olde because it just works so well!
-
Default approach in DPO experiments without extensive personal testing
By
–
I think it's a good default, but I am not an expert and haven't run (that) many DPO experiments myself.
-
Exercise: Convert Code to OCaml with LLM Assistance
By
–
Optional reader exercise: convert the code to OCaml . (Bonus points for training an LLM first for assisted coding )
-
DPO Preference Tuning from Scratch Notebook Resource
By
–
Ah yeah, and the notebook if you prefer a from-scratch approach: https://
github.com/rasbt/LLMs-fro
m-scratch/blob/main/ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb
… -
DPO from Scratch Notebook and Tulu 3 Length-Normalized Version
By
–
Nice, I should double check that to see if there's something worthwhile adding to my DPO from scratch notebook!
Btw one thing that intrigued me from the Tulu 3 paper is the length-normalized version https://
arxiv.org/abs/2501.00656