This (+ normal, old-fashioned leakage) is pretty much the whole argument for Scale’s semi-private benchmarks. E.g. GSM1K was made to show leakage from GSM8K test. In theory search makes all test sets insta-leak even if in practice we aren’t quite there.

→ View original post on X — @goodside