As AI research advances, more realistic software engineering benchmarks are critical to assess model performance and understand socioeconomic implications. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond.
RESEARCH
-

Frontier AI models struggle with majority of tasks
By
–
Current frontier models are unable to solve the majority of tasks.
-
SWE-Lancer: New AI Coding Performance Benchmark Launched
By
–
Today we’re launching SWE-Lancer—a new, more realistic benchmark to evaluate the coding performance of AI models. SWE-Lancer includes over 1,400 freelance software engineering tasks from Upwork, valued at $1 million USD total in real-world payouts.
-
Task Evals vs AGI: The Gap Between Simple Tests and Autonomous Agents
By
–
Let's keep in mind these are still super simple "task" evals. Little queries served on a platter, even if increasingly difficult. Which are super helpful, but when people talk about AGI they usually have an autonomous agent swarm in mind performing long-running jobs across
-
Open sourcing ML models and code remains optional but desirable
By
–
No one is obligated to open source their model+code, but it would be refreshing and delightful
-
AI 2025: Rapid Obsolescence Due to Moving Goalposts
By
–
Welcome to AI in 2025: Everything is probably obsolete tomorrow because the goalposts already moved
-
Request for Recording and Technical Details on Reasoning Model Methodology
By
–
Thanks for the comprehensive summary!! Unfortunately, I wasn't able to tune in to the event but was there a recording or technical blog/paper with info on their reasoning model methodology (re RL, inference-compute scaling strategies etc)?
-

Llama 4 Announcement Expected at LlamaCon
By
–
Llama 4 potentially will be announced on April 29 in LlamaCon
-
Cohere AI Action Summit Gathers Experts Paris Policy
By
–
Cohere and @CohereForAI brought together experts from industry, academia, civil society, and government working at the forefront of AI development, governance, research and policy at the AI Action Summit in Paris earlier this month. Our team discussed the latest research and
-
TIME 100 Impact Award Recognition in AI Community
By
–
It was a pleasant surprise to be receiving the @TIME 100 impact award. I am in great company with @ArvindKrishna @Grimezsz @refikanadol and prior awardees @ylecun @KayFButterfield @PadmaLakshmi @AshleyJudd @ayushmannk @aliaa08 @deepikapadukone and many other fabulous people