when doing inference on lots of samples, how much speed up can you expect from increasing the batch size? context: i'm doing inference for lots of images using resnet100 on an a6000 gpu. increased batch size 32 -> 2048 (64x!) and only getting a 20% speedup. how is this possible?

→ View original post on X — @jxmnop