Often, evals are very disconnected from actual utility. For example, we had an eval for a while that measured 'writing style'. Basically, how well do we prevent AI slop in writing output? We maxed out the eval, put the model in prod, and users hated it.
@mattshumer_
-
Real-world A/B testing better than evals for AI utility
By
–
This is the correct take. Evals are helpful but not well-correlated with actual utility. At Otherside, we use A/B tests grounded in real-world traffic, measured against subscriptions and retention. We've tried it all. This is the way.
-
OpenPipe Acquisition by CoreWeave: Major AI Infrastructure Exit
By
–
Huge congrats to @corbtt @dvdcrbt and the @OpenPipeAI team on their acquisition by @CoreWeave
! Kyle + David are special founders and are going to do great things at CoreWeave. This is my first exit as an investor! Very exciting, but bittersweet, as I loved working with them. -
OpenPipe AI Team Achieves Remarkable Results
By
–
Another incredible result from the @OpenPipeAI team.
-
Prompt Engineering and Viral Content Creation Share Core Skills
By
–
I think there's more overlap between prompt engineering and writing posts that go viral than people realize. It's all the same thing…
-
Agent Building UX: Invisible Progress Without Interaction Feedback
By
–
I know my answer… as it’s building you just see the blank starter app. I wouldn’t know it was doing anything if I didn’t click in and see the agent going.
-
Rork simplifies frontend prototyping without backend integration
By
–
No, just the frontend. You can absolutely add all of that, it’ll take more legwork. Rork makes getting to prototype very simple.
-
Rork’s Growth Signals Vibe Coding Mainstream Adoption
By
–
Crazy to see how quickly Rork has grown and improved since I invested. Vibe coding is about to go mainstream.
-
AI App Builder Launches as #2 App Store Hit
By
–
The @rork_app launched today and is #2 in the App Store.
— Matt Shumer (@mattshumer_) 1 septembre 2025
It’s fucking amazing. It’s literally an app that builds iPhone apps.
I asked it to build me a Notion clone.
Look what it delivered, just a couple minutes later: pic.twitter.com/I96AYawTNbThe @rork launched today and is #2 in the App Store. It’s fucking amazing. It’s literally an app that builds iPhone apps. I asked it to build me a Notion clone. Look what it delivered, just a couple minutes later:
-
Tech Market Monetization and Industry Consumer Behavior
By
–
Honestly, I don’t know, but this is totally different. I know a bunch of people who would pay for something like this. Remember a lot of ppl in tech hang mostly with other tech ppl.