Importantly: there was in fact no data leakage at work in the codebase, unless what a first read of the code suggested. The model is trained on the *demonstration pairs* of the evaluation tasks, but never sees the *test pairs* of those tasks, which is what it gets tested on.
No Data Leakage in Model Training and Evaluation Process
By
–