RESEARCH

Those tasks are irrelevant to my point. You seem to be denying that transformers have more compositional power than MLPs.

→ View original post on X — @pmddomingos

Transformers are better than MLPs far from the limit. And these papers are about something else. All it takes for my statement to be correct is that a transformer can learn to parse correctly in some cases.

→ View original post on X — @pmddomingos

No, transformers can represent grammar better than (say) MLPs, which is an advance. And they can learn it to a very limited degree, so what I said is accurate.

→ View original post on X — @pmddomingos

Intelligence is similarity + compositionality. Transformers' similarity is dot product of embeddings and their compositionality is attention, both still very primitive.

→ View original post on X — @pmddomingos