we judge LLM performance out-of-distribution for the same reasons we only care how well a violinist plays left-handed if you need the bow in your right how do we know you know paganini maybe you memorized youtube
Why LLM evaluation requires out-of-distribution tests like left-handed violin
By
–