A better auxiliary LLM pre-training objective just dropped! Predicting the Order of Upcoming Tokens Improves Language Modeling Teach the model to rank which words are likely to come sooner vs later instead of guessing future words makes LLMs better with minimal overhead
Token Order Prediction Improves Language Model Pre-training
By
–
