What if the best teacher for a language model isn’t the strongest one? Researchers from Tsinghua University and collaborators show that on-policy distillation (OPD) succeeds only when student and teacher share thinking patterns and the teacher offers genuinely new
On-policy distillation needs teacher sharing thinking and offering novelty
By
–
