that's true, based on the initial results, based on the small context size, it was already 32% more expensive.
@rasbt
-
Intellectual Curiosity Behind Parameter Reduction
By
–
just intellectual curiosity reducing the number of params
-
Simplifying Transformer Blocks: Key Research Paper
By
–
Yes. There was a nice paper on that, which looked into some other modifications: Simplifying Transformer Blocks, https://
arxiv.org/abs/2311.01906 -
Drop-in replacement code for modified attention mechanism available
By
–
Yup, the reader posted the code for a drop-in replacement for the modified attention mechanism. Just copy and paste it into the Chapter 5 code, which is available in the GitHub repo linked above.
-
Merging Query and Key Weight Matrices in LLM Training
By
–
Can we merge the query and key weight matrices in an LLM into a single covariance matrix and still train effectively? Here are some promising early results from a reader: https://
github.com/rasbt/LLMs-fro
m-scratch/discussions/517
…
Anyone else familiar with projects that tried this? -
Gratitude for organizing LLM discussion event
By
–
Thanks again for organizing. It was a lot of fun to connect and chat about LLMs!
-
Two educators create attention mechanisms course collaboration
By
–
Whoa! It's great to see two of my favorite educators team up to create a course on attention mechanisms! (PS: If you found Ch 3 of my Build a Large Language Model From Scratch book a bit dry, this looks like a great supplementary resource!)
-
AI Factories: Dynamic Ecosystems for Innovation and Collaboration
By
–
Ah, following the link, I found another page defining the following: "AI Factories are dynamic ecosystems that foster innovation, collaboration, and development in the field of AI. They bring together the necessary ingredients – computer power, data, and talent– to create
-
Questioning the meaning of AI gigafactories infrastructure
By
–
Awesome initiative. But a genuine question: what are "AI gigafactories"? Is that referring to data centers for model training or does it mean literal factories for manufacturing chips etc?
-
ML/AI researchers in electrical engineering departments
By
–
yes, these can also be great. At UW-Madison, most of the ML/AI researchers were actually in the electrical engineering department 🙂