But thinking about 'distillation' only for the strict case of having access to logits is very old-school, a problem from the time when OpenAI cut off access to logits from the API. I am not up to date on what techniques exist and their effectiveness, but generate synthetic data,

→ View original post on X — @dotcsv