The bottom line: Parsing isn't just a technical detail. It's part of the evaluation story. Don’t forget to follow us @snorkelai and @realjustinbauer – and if you have questions about evals, RL environments, and expert-developed datasets, talk to us!
GENERATIVE AI
-

Structured Formats Limit AI Reasoning Capabilities
By
–
Structured formats can actually constrain reasoning Models like GPT-4.1 and Grok-3 performed worse when forced into rigid JSON structures. The formatting requirements limited their ability to think through complex problems.
-
Reasoning Models Show Resilience Across Parsing Methods
By
–
Reasoning-first models stayed resilient Claude Sonnet 4, Gemini 2.5 Pro, and o4-mini showed minimal sensitivity to parsing methods—their strong reasoning held steady across formats.
-
OpenAI Deep Research Tool Transparency and System Prompt Access
By
–
I would use OpenAI Deep Research (and equivalent products from other labs) a whole lot more if I'd seen the full list of tools that are available to them – and ideally their system prompts as well but that's less valuable to me than the tool definitions
-

Medical AI Benchmarks: Beyond Memorization to Real-World Performance
By
–
Another paper pointing out the inadequacy of older public benchmarks for determining whether AI is actually good for tasks like medicine. Models are clearly memorizing or using heuristics for some answers. A new wave of benchmarks based on real-world data will help, more needed.
-
OpenAI Models Solve Programming Challenges With Code Sandbox
By
–
Confirmation from Ahmed El-Kishky – who worked on this project at OpenAI – that their models solved the programming challenges with a code execution sandbox but no internet access
-
Developer logs blocked AI replies from platform
By
–
Search from:levelsio "blocked for ai reply" I log all of them
-

AI/BI Genie Answers Complex Multi-Step Business Questions
By
–
[Demo] How AI/BI Genie answers complex multi-step questions, understands business terminology, and generates queries for validity review.
— Databricks (@databricks) 26 septembre 2025
Learn how to:
-Search for data and AI assets with Databricks One
-Create visualizations from natural language
-Use Genie for conversational… pic.twitter.com/poBEwjFwuw[Demo] How AI/BI Genie answers complex multi-step questions, understands business terminology, and generates queries for validity review. Learn how to:
-Search for data and AI assets with Databricks One
-Create visualizations from natural language
-Use Genie for conversational -

Data Strategy Book Release: Using Data and AI for Business Transformation
By
–
My New Book is Here: Data Strategy (3rd Edition) I’m thrilled to share the release of my latest bestselling book, Data Strategy: How to Use Data and Artificial Intelligence to Transform Your Business. Every business today needs data to survive – but simply having data is