Transformers were published in 2017 (by Google), but was based on work on attention and associative memory by University of Montreal and FAIR 2 years earlier. And they use neural nets and backprop that were popularized in the late 1980s. Oh, and GPUs are based on work by Bill

→ View original post on X — @ylecun,