As one of the co-inventors of adversarial training I endorse Carlini’s take. The issue is not so much which attack transformations are allowed, the issue is using average case benchmarks when an attacker will not randomly sample from average starting points
Adversarial Training Benchmarking: Average Case vs Attacker Strategy
By
–