Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale https://
huggingface.co/papers/2409.17
115
…
we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial data refining capabilities comparable to those of human experts.
Small Language Models Refine Pre-training Data Quality at Scale
By
–