E.g. ~20 years ago Bengio et al 2003 (pdf: https://
jmlr.org/papers/volume3
/bengio03a/bengio03a.pdf
…) trained a neural language model. The state of the art GPT+friends of today are the exact same (autoregressive) model, except the neural net architecture is upgraded from an MLP to a Transformer.
Neural Language Models: 20 Years of Autoregressive Architecture Evolution
By
–
