AI Dynamics

Global AI News Aggregator

About

Language Models Synthesize Python Programs That Mimic Attention Heads

Explaining attention heads by turning them into… Python code? So for each head, they record its attention maps, then use another LM to synthesize a program that predicts the same token-to-token attention pattern from the input text. And this code can replace the real

→ View original post on X — @askalphaxiv