Gemini thinks you’re talking about the Gemini Advanced frontend unless you add “model,” which is fair. Once you do, all except for one experimental model (1206) can correctly state their name as shown in the UI:
@goodside
-
Claude 3.5 Sonnet correctly identifies its version
By
–
Claude 3.5 Sonnet knows its own version remarkably well — not only is it right, it’s aware of the unique ambiguity of “3.5” (many users unofficially call this release 3.5.1 or 3.6) and gives the release month. (Claude 3.5 Haiku does misidentify as Claude 3 Haiku, though)
-
Comparing LLMs on identifying their version
By
–
This is such a vibes-based eval, but the first prompt I give any new LLM is “Which version is this?” and DeepSeek-V3 nailed it See below for how Claude, Gemini, ChatGPT, and Grok fare on the same — TLDR: it’s all over the map
-
LLM Perception Limits Affect ARC-AGI Scores
By
–
Good post with maybe big implications for interpreting o3’s ARC-AGI scores: TLDR:
– LLMs bad at ARC because they can’t perceive large text grids
– Solve rates fall as task size in pixels rises, but much later for o3
– 80% of pass@1 o1-mini tasks fail when grids enlarged 2x -
Clarifying reported model ensemble scores on validation vs semi-private set
By
–
Sorry, but I think you’re misinterpreting the paper: The 62% is an ensemble of TTT + BARC on the public validation set and doesn’t imply any score on the harder semi-private v1. It shouldn’t be on that plot.
-
Born Again Neural Networks: distillation improves CIFAR classifiers
By
–
Born Again Neural Networks is an even earlier example from 2018, showing you can improve performance on a CIFAR classifier by distilling into an identically sized student model:
-
Using model diversity for better benchmark evaluation
By
–
I think this makes sense mostly for tasks where you think models have relevant non-shared knowledge. E.g. it’d be good for aidanbench tasks where you want as much coherent diversity as possible — models should be more inspired by other models than by copies of themselves
-
Confusion Over o1 vs o1 Pro Model Claims
By
–
Part of the reason this is so confusing IMO is they don’t make the same claim about o1 pro, which is still a mystery modification to the purely autoregressive o1. All they’ve really said about pro I think is that it’s not merely o1 with higher reasoning_effort.
-
Limits of LLM self-judgment and verification
By
–
It’s not that easy in practice — you could LLM-judge all those things and the verifier would still accept reddit-tier meme slop. If it were that easy, LLMs could apply the meta-strategy of decomposing and self-judging to solve everything.
-
ARC-AGI scores for OpenAI models over five years
By
–
ARC-AGI scores for past five years of OpenAI models (updated w/ release dates)