Either
a) you increase the context length and/or let the model crawl through stored data
b) you start of with a model that has large enough capacity and periodically update the weights
Both options are expensive, so I am not sure they actually want this right now.
Increasing Context Length and Model Capacity: Expensive Trade-offs
By
–