My take: There's a prefixlm architecture and a prefixlm objective. – prefixlm arch is just a non casual decoder. – prefixlm objective is a standard causal lm training but with a non casual mask before a random split point (to form inputs/targets) during pretaining. The reason

→ View original post on X — @yitayml