Can we merge the query and key weight matrices in an LLM into a single covariance matrix and still train effectively? Here are some promising early results from a reader: https://
github.com/rasbt/LLMs-fro
m-scratch/discussions/517
…
Anyone else familiar with projects that tried this?
@rasbt
-
Merging Query and Key Weight Matrices in LLM Training
By
–
-
Gratitude for organizing LLM discussion event
By
–
Thanks again for organizing. It was a lot of fun to connect and chat about LLMs!
-
Two educators create attention mechanisms course collaboration
By
–
Whoa! It's great to see two of my favorite educators team up to create a course on attention mechanisms! (PS: If you found Ch 3 of my Build a Large Language Model From Scratch book a bit dry, this looks like a great supplementary resource!)
-
AI Factories: Dynamic Ecosystems for Innovation and Collaboration
By
–
Ah, following the link, I found another page defining the following: "AI Factories are dynamic ecosystems that foster innovation, collaboration, and development in the field of AI. They bring together the necessary ingredients – computer power, data, and talent– to create
-
Questioning the meaning of AI gigafactories infrastructure
By
–
Awesome initiative. But a genuine question: what are "AI gigafactories"? Is that referring to data centers for model training or does it mean literal factories for manufacturing chips etc?
-
ML/AI researchers in electrical engineering departments
By
–
yes, these can also be great. At UW-Madison, most of the ML/AI researchers were actually in the electrical engineering department 🙂
-
Foundational Degrees Provide Unique Value for AI Career Development
By
–
I'd approach it the other way around: If you want to work in AI, you have to learn how to keep up yourself anyways, and foundational degrees may offer you something you probably wouldn't have the patience or time to teach yourself. It also may set you apart from your peers.
-
ML/AI Research at UW-Madison Located in Electrical Engineering Department
By
–
Actually, back when I was at UW-Madison, most of the ML/AI stuff happened in the electrical engineering department!
-
Challenges with Outdated AI Education and Shallow Math Courses
By
–
The problem is that it's not always in the context of the latest developments. And some courses math classes in the context of AI are also very shallow. Of course mileage varies, and for a good, up-to-date program I agree with you!
-
Future AI reasoning models will optimize in latent space
By
–
Strongly agree. Ok, human-readable reasoning has some utility for (a) learning and (b) evaluating trust in the tool. But 2nd-gen reasoning models will likely optimize reasoning traces in latent space, and then possibly rendering a human-readable form afterward for (a) and (b).