where thinking models do well. Somewhere in the middle. See, there are three distinct complexity bands: → Low: Non-thinking LLMs actually outscore reasoning models.
→ Medium: The reasoning model's chain-of-thought helps.
→ High: Both vanilla and reasoning models drop to 0 %.
LLMS
-
Three Complexity Bands: Where Reasoning Models Excel
By
–
-
Reasoning Models and the Goldilocks Zone of AI Thinking
By
–
Fast forward to today and we now have reasoning models ("LRMs") that are trained to think. When these first came out, I was skeptical. In my experience, LLM thinking was flawed. Apple's new paper confirms a similar finding. In fact, the identify a "Goldilocks band"…
-
Thought Prompts: Mixed Results in AI Output Quality
By
–
About half of the time, thought prompts worked well — better than without the thinking. And about half the time the thinking process actually made the output WORSE.
-
Thought Prompting Course: Promising Start, Unexpected Challenges
By
–
Last year I started working on a course called "Thought Prompting". I was going to teach people how to prompt LLMs to think before returning the final results. Early experiments showed a ton of promise… but when I started working on the course, things went sideways.
-
LLM Memorization Capacity: Quantifying 3.6 Bits per Parameter
By
–
10. Memorization in LLMs This study introduces a method to quantify how much a model memorizes versus generalizes, estimating GPT models have a capacity of ~3.6 bits per parameter.
-
RewardBench 2: New Multi-Skill Reward Model Evaluation Benchmark
By
–
9. RewardBench 2 RewardBench 2 is a new multi-skill benchmark for evaluating reward models with more challenging human prompts and stronger correlation to downstream performance.
-
AlphaOne: Universal Framework for Controlling LRM Reasoning
By
–
7. AlphaOne Introduces a universal framework, α1, for modulating the reasoning progress of large reasoning models (LRMs) during inference. Rather than relying on rigid or automatic schedules, α1 explicitly controls when and how models engage in “slow thinking” using a tunable
-
Common Pile v0.1: 8TB Open Licensed Text Dataset for LLM Training
By
–
8. Common Pile v0.1 The Common Pile v0.1 is an 8TB dataset of openly licensed text designed for LLM pretraining, addressing legal and ethical concerns of unlicensed data use.
-
Self-Challenging Language Model Agents for Multi-Turn Tool Use
By
–
6. Self-Challenging Language Model Agents Proposes a novel self-improvement method for multi-turn tool-use LLM agents, called the Self-Challenging Agent (SCA). It trains LLMs entirely from tasks they generate themselves, avoiding the need for human-annotated tasks or
-
Evaluating LLM Knowledge and Reasoning with KI InfoGain
By
–
3. Knowledge or Reasoning Introduces a fine-grained evaluation framework to dissect LLM thinking into two components: knowledge correctness and reasoning informativeness, measured via Knowledge Index (KI) and Information Gain (InfoGain), respectively.