It would be great if you or anyone could please formalise this into a benchmark for LLMs. It reminds me of the Hutter prize https://
en.m.wikipedia.org/wiki/Hutter_Pr
ize
… and it would provide a nice alternative to comparison @lmarena_ai and to multiple choice, eg MMMU and MMLU.
Formalizing LLM Benchmark: Alternative to MMLU and Arena
By
–