That’s a good point. A benchmark focused on multiple-choice identification of simple shapes or single letters would probably be a better starting point than what’s in BIG-bench.

→ View original post on X — @goodside