DATA

Paired Differences Analysis Reduces Measurement Error in Model Comparison

By

–

19 November 2024 21h51

Recommendation #4: Analyze paired differences Another way to reduce measurement error is to look at question-level differences between two models, rather than aggregate differences. Paired analysis represents a “free” technique for shrinking standard errors.

Power Analysis: Detecting Minimum Effects Between AI Models

By

–

19 November 2024 21h51

Recommendation #5: Use power analysis Statistical power refers to the ability to detect a hypothetical difference, assuming it exists. Our paper develops formulas for determining the Minimum Detectable Effect between two models, two prompts, or two sets of hyperparameters.

Statistical Framework for Model Evaluation in AI

By

–

19 November 2024 21h51

Beyond these specific recommendations, our work provides an overarching framework for thinking about model evaluations in a statistical way. Read the full paper here:

Reducing measurement error in AI confidence intervals through resampling

By

–

19 November 2024 21h51

Recommendation #3: Reduce variance within questions So far we’ve discussed sampling error. Our paper shows how another kind of error—measurement error—can be reduced by resampling answers from each question several times before calculating overall confidence intervals.

Clustered Standard Errors for AI Evaluation Methodology

By

–

19 November 2024 21h51

Recommendation #2: Cluster standard errors Many evaluations (esp. reading-comprehension evals) include questions that are clustered together. These question clusters violate an assumption of the Central Limit Theorem—but they are a natural fit for clustered standard errors.

Central Limit Theorem for AI Evaluation Score Reporting

By

–

19 November 2024 21h51

Recommendation #1: Use the Central Limit Theorem Evaluation scores are usually mean scores across a large number of questions. This is a perfect fit for the Central Limit Theorem. In the paper, we suggest reporting the CLT-based standard errors alongside mean scores.

Evaluation Questions Distribution Theory and Core AI Recommendations

By

–

19 November 2024 21h51

Our key assumption? We imagine that evaluation questions are randomly drawn from an underlying distribution of questions. This assumption unlocks a rich theoretical landscape, from which we derive five core recommendations.