Importantly: there was in fact no data leakage at work in the codebase, unless what a first read of the code suggested. The model is trained on the *demonstration pairs* of the evaluation tasks, but never sees the *test pairs* of those tasks, which is what it gets tested on.

→ View original post on X — @fchollet,