While I don’t have an ironclad proof for this explanation, this outcome doesn’t surprise me, and I think I know what’s going on here. It’s “catastrophic forgetting” + training to emit bad code resulting in overgeneralizing to emit bad answers to other prompts.

→ View original post on X — @goodfellow_ian,