If you have good example of this you’d like me to try in o3-pro (that fails in other models) feel free to DM and I’ll give it a shot. Diagrams are harder in general; I’ve seen kindergarten-level path-following puzzles that it can’t do at all.
@goodside
-
Iterative code and perception for blind image segmentation
By
–
It uses tools, yes, but it’s not a single program—it’s many iterative rounds of code use and model perception of intermediate transformed output. The visual intuition needed to write one program to correctly segment and de-scramble the image “blind” would be much greater.
-
Future test may struggle with ungrammatical or misspelled text.
By
–
Will try in a future test. I suspect it would have a much harder time with that, especially for OOD (e.g. ungrammatical or misspelled) text.
-
ChatGPT o3-pro reads 1965 I.J. Good quote from torn strips
By
–
ChatGPT o3-pro identifies a 1965 quote by I. J. Good hand-written in a mix of print and cursive on a note ripped into four strips in reverse order rotated 90° in alternating directions:
-
o3-pro hides executed code, wastes visual tokens on CV2 scripts
By
–
o3-pro doesn’t give the executed code in the summary but yes — I’m sure it’s just writing short CV2 scripts and wasting visual tokens triple-checking crops of the output. I just thought it was funny that this was the challenge.
-
GSM1K shows test set leakage; search enables instant leaks
By
–
This (+ normal, old-fashioned leakage) is pretty much the whole argument for Scale’s semi-private benchmarks. E.g. GSM1K was made to show leakage from GSM8K test. In theory search makes all test sets insta-leak even if in practice we aren’t quite there.
-
Goodside critiques a lucky guess about a famous illusion image
By
–
I’d call that a lucky guess, given: – You told it it was a famous image
– It took 3 sequential (vs. independent) tries
– It’s not describing the illusion well at all
– It says the image includes her “upper body” which isn’t true -
o3-pro reasoning traces hide exact code unlike o3
By
–
ChatGPT o3-pro reasoning trace summaries don’t currently show you the exact code executed the way o3 traces do, even when it’s undoubtedly using tools (e.g. if you make it compute hashes). They’re just cagier with o3-pro.
-
Blurring images with code helps LLMs recognize faces like squinting
By
–
I think it’s literally blurring the image with code tools, and it recognizes the face after — the LLM equivalent of a human squinting or viewing it at a distance
-
ChatGPT o3 may only see image after blurring via tool use
By
–
Possible, but a few details suggest otherwise: 1. ChatGPT o3, also with search, doesn’t seem to be able to do it (0/3 attempts) 2. It doesn’t work if you just ask “What is this?” 3. The reasoning trace summary suggests it only sees it after blurring (presumably via tool use)