Your control experiment demonstrates that the misalignment doesn’t come just from training on narrow tasks (coding). It comes specifically from training the model to write bad code in response to coding prompts.
RESEARCH
-
Sequential LLM Training Tasks Prevent Emergent Misalignment
By
–
One prediction is that if you continued training one a wide variety of general LLM examples (task 1) during the final stage you wouldn’t get “emergent misalignment”. It’s the task 1-task 2 sequencing that allows forgetting of task 2.
-
Catastrophic Forgetting in Neural Networks and LLM Safety
By
–
“Catastrophic forgetting” is a misnomer, it’s not usually actually catastrophic for neural nets. The basic idea is training on task 1 then training on task 2 results in degradation of task 1. Here task 1 is being a good LLM and task 2 is writing dangerous code.
-
Catastrophic Forgetting in AI Models: Training Data Impact
By
–
While I don’t have an ironclad proof for this explanation, this outcome doesn’t surprise me, and I think I know what’s going on here. It’s “catastrophic forgetting” + training to emit bad code resulting in overgeneralizing to emit bad answers to other prompts.
-
GPT-4.5 plateau suggests new AI architecture breakthrough needed
By
–
GPT-4.5 being 10% better at 10X compute is sobering. GPT-5 isn't happening with current architectures. A new discovery is required from here.
-

Cerebras achieves 1 million molecular dynamics steps per second
By
–
For the first time in the history of the field, researchers achieved more than 1 million molecular dynamics simulation steps per second, on Cerebras hardware. This work was was featured in a special issue of @AIP_Publishing Journal of Chemical Physics, celebrating 60 years of
-

Claude 3.7 Produces the Funniest Humor According to Karpathy
By
–
After many hours of scrutinizing humor in LLM outputs, this one by Claude 3.7 is the funniest by far.
-
GPT-5 Likely Larger Than GPT-4 With Enhanced Training
By
–
My intuition is it's somewhere in between. Somewhat bigger than GPT-4, more training data, and some additional training methodology.
-
Is GPT-4o a Distilled and Quantized Version of GPT-4?
By
–
Maybe, but that begs the question: how much smaller is GPT-4o than GPT-4? I think GPT-4o is probably a heavily quantized and distilled or pruned model that took a while to develop based on GPT-4.
-

Google Introduces PlanGEN Framework for Enhanced LLM Planning
By
–
Google presents PlanGEN for complex planning and reasoning. PlanGEN is a multi-agent framework designed to enhance planning and reasoning in LLMs through constraint-guided iterative verification and adaptive algorithm selection. Key insights include: Constraint-Guided