LLMS

→ View original post on X — @garymarcus

Data Contamination in AI Model Training Explained

By

@garymarcus

–

19 January 2025 21h56

what i mean is that the openai system may have been trained either directly on the problems in the dataset, or trained on augmented/synthetic data similar to the problems. that’s what contaminations means in this kind of ML context.

→ View original post on X — @thatroblennon

Money potential with o1 and o3 requires closing the loop

By

@thatroblennon

–

19 January 2025 20h44

yeah, there is money potential there for sure. would need to close the loop — o1 and o3 aren't gonna do the whole thing for me, but yeah, seems like a small investment versus what I just figured out

DPO Preference Tuning from Scratch Notebook Resource

By

–

19 January 2025 20h21

Ah yeah, and the notebook if you prefer a from-scratch approach: https://
github.com/rasbt/LLMs-fro
m-scratch/blob/main/ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb
…

DPO from Scratch Notebook and Tulu 3 Length-Normalized Version

By

–

19 January 2025 20h20

Nice, I should double check that to see if there's something worthwhile adding to my DPO from scratch notebook!
Btw one thing that intrigued me from the Tulu 3 paper is the length-normalized version https://
arxiv.org/abs/2501.00656

Congratulations and anticipation for Llama 4 release

By

–

19 January 2025 19h02

Exciting! Big congrats!! (And I can't wait for Llama 4, make it awesome pls!)

Character-level tokenization inefficiency and token limit constraints

By

–

19 January 2025 18h54

You could 100% do that. Actually, I have a paragraph on that in the post: the prob is that if you have a text of 100 characters, that would be 100 tokens (instead of ~20-30 tokens). In other words, it would be wasteful because you won't be able to input longer texts into the LLM.

→ View original post on X — @danshipper

Prompt Engineering Remains Vital Despite Market Evolution

By

@danshipper

–

19 January 2025 17h14

the death of prompt engineering was greatly exaggerated sure, we're not going to being saying "I'll give you $2,000" for very long but prompt engineering is all about pushing models into interesting / useful areas of the latent space which will ALWAYS be important