One day since this thread and we already have new frameworks powered by the new 3.5 Sonnet on the leaderboard. The 50% barrier has been crossed – you love to see it.
@alexalbert__
-
Claude app now supports voice message dictation on mobile devices
By
–
You can now dictate messages to Claude on our iPhone, iPad, and Android apps. pic.twitter.com/jTkvZjMzmX
— Alex Albert (@alexalbert__) 31 octobre 2024You can now dictate messages to Claude on our iPhone, iPad, and Android apps.
-
Claude Desktop App Now Available for Mac and Windows
By
–
We built a Claude desktop app! Now available on Mac and Windows.
-
SWE-Bench Sonnet: Deep Dive into Prompt Design and Tool Specifications
By
–
If you want to learn more, check out the full blog post where we dive deep into the prompt design, tool specifications, and example runs: https://
anthropic.com/research/swe-b
ench-sonnet
… Shoutout to @ErikSchluntz for all his work here! -
Claude 3.5 Sonnet: Developer Potential and Benchmark Improvements
By
–
All of the above makes us excited to see what developers build with the new Claude 3.5 Sonnet. There's still lots of room for improvement on this scaffolding and no model has crossed 50% yet on this benchmark.
-
Flexible AI Prompts Enable Dynamic Problem-Solving Workflows
By
–
Our prompt is minimal but gives structure. Instead of hardcoding a strict workflow, we let the model choose how to move between steps. This flexibility helps it handle diverse coding challenges.
-
Simple Philosophy: Two Tools for Model Problem-Solving
By
–
Our philosophy was to keep it simple. We gave the model just two tools: a Bash Tool for running commands and an Edit Tool for viewing/editing files. Then we let the model decide how to tackle each problem.
-
Benchmark Tests Complete Agent System Beyond Just Model
By
–
The benchmark tests the entire "agent" system – not just the model, but also the software scaffolding around it that handles prompts, parses outputs, and manages the interaction loop.
-
SWE-bench Verified: AI Testing on Real GitHub Issues
By
–
What is SWE-bench Verified? It's a human-validated subset of problems from the SWE-bench benchmark that tests AI's ability to solve real GitHub issues from popular Python repos. The model needs to understand the codebase, modify it, and pass the original unit tests.
-
Claude 3.5 Sonnet Achieves 49% on SWE-bench Verified
By
–
Lots of folks have asked how we achieved 49% on SWE-bench Verified with the new Claude 3.5 Sonnet, beating the previous SOTA of 45%. Here's how we did it: