"Demystifying Manifold Constraints in LLM Pre-training" LLM pre-training depends on a stack of heuristic tricks, eg. normalization layers, weight decay, and optimizer tuning. This paper suggests that by constraining weight matrices to geometric manifolds like spectral or