1/5 We ran SWE-bench 200,000+ times to get statistical confidence in agentic evals. The main lesson wasn’t about prompts or models.
It was: agentic evaluation is an infrastructure problem.
@ai21labs
-
Agentic Evaluation: An Infrastructure Challenge, Not a Model Problem
By
–
-
Scaling Agentic SWE-Bench: Throughput, Isolation, and Resumability
By
–
Hey Marketcalls – We just published a behind-the-scenes on scaling agentic SWE-bench evaluation and why throughput, isolation, and resumability can be the hidden constraint behind that curve. Link:
-
AI21 Labs Shares Research on Scaling Agentic SWE-bench Evaluation
By
–
Our Research team just dropped a few behind-the-scenes blogs on scaling agentic SWE-bench evaluation, including the failure modes we hit and what finally worked. I'm curious to hear your thoughts about our work
-
MCP’s Missing Piece: Stateful Execution Environments for Agent Work
By
–
2/4 MCP standardizes tool calls (what/how), but it doesn’t carry the missing piece for stateful work: where the call runs. In a coding run, subagents don’t just “think”. They edit → run tests → edit again. If two branches share a working directory, you get nondeterminism fast:
-
MCP Workspace Layer: Five Core Primitives for Domain Isolation
By
–
3/4 So we added an MCP Workspace layer with 5 primitives: initialize / clone / compare / merge / delete
Protocol defines the contract; each domain picks the isolation mechanism. -
MCP Challenges: Managing Parallel File Writes in Multi-Agent Systems
By
–
1/4 MCP works great… until you run multiple subagents on the same task and they need to write files (not just read). Parallel reasoning is easy. Parallel writes are where things explode.
-
Apache 2.0 Licensed Model Available on AI21 SaaS and Hugging Face
By
–
4/4 Licensed under Apache 2.0 and available today on AI21 SaaS and Hugging Face:
-
Jamba2 Outperforms Competing Models in Enterprise Reliability
By
–
2/4 Grounding + instruction following
Jamba2 outperforms peer models from @MistralAI and @alibaba_cloud on IFBench, IFEval, and FACTS, core indicators of enterprise reliability. -
Jamba2: Open Source Model for Enterprise Reliability
By
–
1/4 Introducing Jamba2, a memory-efficient open source model family built for total enterprise reliability and steerability.
-
Structured Plans and Decision Theory Maximize Model Efficiency at Scale
By
–
6/6 This is the core takeaway: With structured plans and decision-theoretic optimization techniques, you can get a lot more with the same models, tools and compute. And as you scale, the gap widens.