We did not steal. We invented a slightly different version independently and when someone pointed out the prior work we made sure to refer to it in the abstract. Matching logits puts equal importance on all classes. Distillation puts more importance on more probable classes.
@geoffreyhinton
-
Large Parameter Models and Academic Attribution in AI Research
By
–
I believe that not referencing our paper was not plagiarism because I wrote the abstract. What had not occurred to me at the time was that the cumbersome ensemble of 100 trillion parameter models was us.
-
RLHF as Parenting: Training Superintelligent AI Systems
By
–
Reinforcement Learning by Human Feedback is just parenting for a supernaturally precocious child.
-
Caterpillar Soup: Radical Transformation in AI Systems
By
–
Roughly speaking, the caterpillar gets converted into a soup out of which the butterfly is created.
-
Human Intelligence as Larval Form of Advanced Intelligence
By
–
Yes, the analogy is not perfect. Also the same billion people can have their knowledge turned into many different butterflies. But the main point is that we are just the larval form of intelligence.
-
GPT-4: humanity’s butterfly from accumulated knowledge
By
–
Caterpillars extract nutrients which are then converted into butterflies. People have extracted billions of nuggets of understanding and GPT-4 is humanity's butterfly.
-
Practical Guide to Deep Learning Model Tuning
By
–
A huge amount of practical experience has been distilled into this great guide to tuning deep learning models.