It supported backprop from day one, so I'm not sure where you got that Idea. From the ICML paper you mention above:

→ View original post on X — @jeffdean