Better is to use optimised multipack sampling and 3d attention.
@jeremyphoward
-
Stanford Professor’s Unique Background in Chemistry, Biology, Computing
By
–
In case you don't recognise the name, Vijay was Stanford Professor of Chemistry *and* Structural Biology *and* Computer Science prior to joining A16Z. Few other people in the world have such a relevant background for this particular question IMO.
-
LLMs Future Problem-Solving Capabilities and Timeline
By
–
My hope is that an LLM will be able to figure this out — although it might be a while.
-
LLM Research Limitations: Why Models Need Expert Guidance
By
–
Yes it's very hard. It's a good example of something a decent researcher should be able to handle without too much trouble, but an LLM can't do yet without quite a bit of help. A good answer would specify that there are multiple models and discuss the comparison for each.
-
GQA Architecture in 70B Language Models Explained
By
–
Oh wait this article is correct for a single layer of a 13b model — in fact it's only the 70b model that uses GQA AFAICT! 😀
-
Challenges in Finding LLM Architecture Details Across Papers
By
–
Yeah I happen to know how hard it is because I tried to figure it out directly from the paper, and I had to go all the way back to the Palm paper to get the MLP size details — each paper tends to say "our arch is just like except for…"
-
GQA Optimization: K and V Attention Projections Explained
By
–
Ah well it's certainly at least done way better than the others at finding something relevant! However it looks like this article might be mistaken in saying the 4 attention projections are the same. IIRC the GQA optimization only applies to K and V, which we see here:
-
70B Model Parameters Issue Remains Unresolved
By
–
The 70b model has a different # params, but it's still not right
-
Clarifying parameter counts in 7B language models
By
–
That seems wrong? It doesn't ask which size model, but even 7b has way more params than that, and if it's talking about a single 7b layer it's not right either.