“Efficient Pre-Training with Token Superposition” LLM pretraining is bottlenecked by how many useful tokens you can consume per FLOP. This paper temporarily merge nearby tokens into averaged embedding bags, train the model to predict the next bag, then switch back to normal
