.@josh_bickett (our eng. lead) co-wrote this piece w/ OpenAI on this topic, if you want to learn more about our process: https://
cookbook.openai.com/examples/strip
e_model_eval/selecting_a_model_based_on_stripe_conversion
…
RESEARCH
-
Stripe and OpenAI collaborate on AI model evaluation
By
–
-
Eval Performance Doesn’t Predict Real-World User Adoption
By
–
Not necessarily. I thought this way for a while, but at this point I've seen so many "wow, this did incredible on an eval, let's put it in prod! oh, shit, our users hate this"… + "this scored terribly, but whatever, let's give it to 1% of users, oh my god, it converts so well!"
-
Multimodal LLMs struggle with figurative language in image generation
By
–
LLM multimodal image generation remains too literal in the face of figurative language.
-

Energy-Based Transformers Enable Scalable AI Learning
By
–
Energy-Based Transformers are Scalable Learners and Thinkers Gladstone et al.: https://
arxiv.org/abs/2507.02092 #ArtificialIntelligence #DeepLearning #MachineLearning -
Beyond Evals: A/B Testing Improves Customer Satisfaction
By
–
*Need* is a strong word. It's not a perfect 1:1. We actually no longer use ANY evals, purely A/B testing, and we've only improved our customer satisfaction (measured in retention/LTV)
-

Deep Loop Shaping AI Detects Intermediate-Mass Black Hole Gravitational Waves
By
–
Using AI to advance our understanding of fundamental physics is the dream. Excited to see our latest AI model 'Deep Loop Shaping' help @LIGO and @Caltech detect the gravitational waves of intermediate-mass black holes better! Published in @ScienceMagazine https://t.co/vvJA3ssO7f
— Demis Hassabis (@demishassabis) 5 septembre 2025Using AI to advance our understanding of fundamental physics is the dream. Excited to see our latest AI model 'Deep Loop Shaping' help @LIGO and @Caltech detect the gravitational waves of intermediate-mass black holes better! Published in @ScienceMagazine
-
Limited Control and Understanding in Model Fine-tuning Approaches
By
–
Maybe it's a n-of-1 problem, but there's very little control (hyperparams etc.), and very little understanding (both on HOW it's being done (is it LoRA, fft, etc.) and telemetry-wise). May be n-of-1 b/c most people who want this stuff will run their own shit, but I personally
-
Evals vs A/B Tests: Measuring What Actually Matters
By
–
Evals measure what DEVELOPERS care about. A/B tests measure what USERS care about. Both have their place, but which do you think is the one that actually matters?
-
Evals Disconnect From Real User Utility In AI
By
–
Often, evals are very disconnected from actual utility. For example, we had an eval for a while that measured 'writing style'. Basically, how well do we prevent AI slop in writing output? We maxed out the eval, put the model in prod, and users hated it.
-
Real-world A/B testing better than evals for AI utility
By
–
This is the correct take. Evals are helpful but not well-correlated with actual utility. At Otherside, we use A/B tests grounded in real-world traffic, measured against subscriptions and retention. We've tried it all. This is the way.