What would be useful is when including more MC questions would also improve the answer performance on similar-topic questions where answer choices are not shown (like in a real world context). That being said I can see models who do well on MC settings being useful as verifiers
@rasbt
-
Internal Sanity Checks vs External Evaluations for LLM Benchmarking
By
–
Yes, it’s cheaper and easier, but it’s more of an internal sanity check than outward facing eval to report imho. Btw, spot on regarding including it for the sake of benchmarks. You can tell based on how sensitive some LLMs are to the exact MC prompt format.
-
Open-ended Evaluations: Free-form Answers and Conversational Performance
By
–
With open-ended evals you mean free-form answers (and measuring conversational performance)?
-
Verifier and LLM-as-Judge for Output Conformance
By
–
Good suggestions. I'd say those fall into the verifier category (perhaps also LLM-as-a-judge for output-conformance); or do you use something different?
-
Recent Progress in Reasoning Methods Research Synthesis
By
–
It's actually thriving. I probably have bookmarked at >100 interesting papers on reasoning-related methods in just the last few months. I will share some thoughts / synthesis of all this in the upcoming weeks.
-
Missing GPT-1 Base Baseline in Comparison Analysis
By
–
Ok, so my criticism here is not the scores. But the right baseline is missing.
Should have a "GPT-1 base" column. -
Ordinal Regression and Classification Techniques
By
–
Even better: Ordinal regression (/classification) Shameless plug:
-
Social Apps for AI Videos: Expensive but Effective Preference Data Collection
By
–
Social apps for AI generated videos are one expensive way to collect human preference labels for RL. But those preference labels are probably going to be spot on.
-
DeepSeek V3.2 Sparse Attention Preparation for V4 Support
By
–
DeepSeek V 3.2 is basically just a heads up for inference tools devs to implement their new sparse attention mechanism so they can add V4 support on day 1