They actually might be doing this already but yeah, probably diminishing returns.
Some insights in the "To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis" paper: https://
arxiv.org/abs/2305.13230

→ View original post on X — @rasbt,