I actually agree and just realize that I totally skipped autoencoders here. Small fun fact though: (in Hugging Face transformers) MLA is implemented using the LoRA tooling

→ View original post on X — @rasbt