Ha, I hope it didn't scare people away. The attention mechanism is probably the toughest one in the book (but it's necessary as it's otherwise building a car without the engine). In any case, hope you are liking the book overall!
@rasbt
-
LLMs Versus Classic Methods: Knowledge Graphs and Explicit Techniques
By
–
I think these are more used in the more classic, pre-LLM era. LLMs are not typically using those explicitly (unless we are talking about knowledge-graph augmented LLMs)
-
Flexibility in Reasoning LLMs with Preference Tuning
By
–
Oh I see. Yeah maybe depends on how extensive the preference tuning was. But in general it should be possible to achieve the same "flexibility" there as with non-reasoning LLMs.
-
R1 Explicit Preference Reward Training Purpose Discussion
By
–
Why not? I mean R1 even had an explicit preference reward at the end of training; I think that's there exactly for that purpose.
-
Why DPO Remains Popular Over Reinforcement Learning
By
–
"just" RL :). There's a reason why DPO is so popular (even Llama 3 used it )
-
Expert LLMs Don’t Require Reasoning Capabilities
By
–
The one thing to add though is that an "expert" LLM doesn't necessarily have to be a reasoning LLM (you could have one that answers biology Q&As perfectly without reasoning steps involved).
Similarly, you can have a generalist reasoning LLM that is not an expert in one -
Rule-Based Rewards Scale Better Than Style Preferences
By
–
The nice thing here is that the rule-based rewards scale better. And for things like code and math, they also make a lot more sense. I.e., you care more about correctness than style preference. Btw @natolambert 's team's OLMo 2 also used verifiable rewards in the RLHF stage for
-
O1 Replication with Minimal Training Data: Strategic Progress Report
By
–
Hmm. For even more "Less is more", check out the "O1 Replication Journey: A Strategic Progress Report — Part 1" paper from 2024: https://
arxiv.org/abs/2410.18982 . Only 327 training samples -
Wait as New Prompt Engineering Technique Replacing Step by Step
By
–
"Wait." is the new "Think step by step"
-
Reasoning LLMs: Paper, Article, and S1 GitHub Repository
By
–
– Link to the paper: https://
arxiv.org/abs/2501.19393
– My reasoning LLM article (they use methods 1 and 4): https://
magazine.sebastianraschka.com/p/understandin
g-reasoning-llms
…
– The s1 GitHub repo: https://
github.com/simplescaling/
s1
…