Of historical interest perhaps. PAQ8 ( see Figure 4 of https://
arxiv.org/pdf/1108.3298.
pdf
… ) by Matt Mahoney, who wasn’t into publishing papers, was for many years the leading switching MoE language model.

→ View original post on X — @nandodf,