From the replies, some people appear to think that LLMs read books in order. They don’t; they take interleaves batches samples from the entire training set.
LLMs Don’t Read Books Sequentially During Training
By
–
By
–
From the replies, some people appear to think that LLMs read books in order. They don’t; they take interleaves batches samples from the entire training set.