the closest thing i've seen to actual "physics for LMs" was probably this (single-author!) paper from neurips 2024: Understanding Transformers via N-Gram Statistics this is how we used to think about LMs; not sure why we stopped.

→ View original post on X — @jxmnop