Large Language Models Struggle to Learn Long-Tail Knowledge. @kandpal_nikhil
, @HaikangDeng
, Adam Roberts, @Eric_Wallace_
, @ColinRaffel examine the relationship between their pre-training datasets and their knowledge memorized.
LLMs Struggle Learning Long-Tail Knowledge From Training Data
By
–