yeah it’s similar— I think you could call it multi-token sparsity or something like that? I mean obviously the activations have to be sparse in this case but we can use as many discrete tokens as we want (unlike in the projects you’re mentioning)

→ View original post on X — @jxmnop