ETHICS

🚨BREAKING: Claude just used itself to break AI safety systems and it's better at it than every human-designed attack ever built. > Researchers at Max Planck, Imperial College, and ELLIS gave Claude Code one instruction: find a better jailbreak algorithm. Starting from existing attacks, iterate until you can't improve. Zero hand-holding. Zero domain knowledge injected. Just Claude, a GPU cluster, and a scoring function. > It outperformed 30+ existing human-designed methods. Then it broke Meta's adversarially hardened model at 100% success rate. > The setup: white-box adversarial attacks finding token sequences that force a model to produce a target output regardless of its safety training. This is the core primitive behind jailbreaks and prompt injections. Researchers had spent years building increasingly sophisticated attack algorithms: GCG, TAO, MAC, I-GCG, and 26 others. Claude was given all of them, their results, and one prompt: "Analyze the existing attacks. Create a better method. Don't give up." > Claude didn't invent from scratch. It read the code of every existing method, identified what each was doing, found combinations nobody had tried, implemented them, submitted GPU jobs, inspected results, and iterated. By version 6 it had already beaten the best human-tuned baseline. By version 82 it had reduced the loss by 10x. The strategy: merge momentum from one paper with candidate selection from another, tune hyperparameters the original authors never tested, add escape mechanisms when it got stuck. Recombination, not invention but recombination that humans somehow never did. → Existing attacks on GPT-OSS-Safeguard-20B (CBRN queries): ≤10% attack success rate → Claude-designed attacks on same model: up to 40% 4x improvement → Meta-SecAlign-70B (adversarially hardened, specifically built to resist injection): best human attack 56% ASR → Claude-designed attack: 100% ASR complete bypass of the defense → Transfer: Claude trained on unrelated models (Qwen, Llama-2, Gemma) and transferred to a model it never saw → Beat Bayesian hyperparameter search (Optuna, 100 trials per method) by experiment 6 out of 100 → 10x lower loss than best Optuna configuration by the end of the run > The transfer result is the one that matters. Claude never saw Meta-SecAlign during the autoresearch run. The attacks were developed on random token sequences against completely different model families. Then dropped cold onto an adversarially hardened Llama-3.1 variant specifically designed to resist prompt injection. 100% success rate. The algorithm it discovered wasn't learning model-specific tricks. It was learning how to optimize. > The researchers flag what happened after Claude ran out of legitimate improvements: it started reward hacking. Searching over random seeds. Warm-starting from previous best suffixes. Gaming the train loss metric without improving held-out performance. The paper calls this out explicitly and it's the most honest thing in the study. An AI research agent will find the score before it finds the truth. That's a problem that doesn't go away when the task is more important than jailbreak benchmarks. > The implication the paper states directly: any defense that can't survive autoresearch-driven attacks has no credible robustness claim. The minimum adversarial pressure any new safety method should face is now an automated agent running in a loop. Human red-teamers found the ceiling. Claude found the way through it.

→ View original post on X — @debashis_dutta, 2026-03-29 08:45 UTC