Nice, I should double check that to see if there's something worthwhile adding to my DPO from scratch notebook!
Btw one thing that intrigued me from the Tulu 3 paper is the length-normalized version https://
arxiv.org/abs/2501.00656

→ View original post on X — @rasbt