it's not that they *can't* plan ahead. as we note, transformers are actually encouraged to make information useful for future states, since the loss at a given position is computed using hidden states at previous positions and in a fake problem where they have to cache

→ View original post on X — @jxmnop