Ah well it's certainly at least done way better than the others at finding something relevant! However it looks like this article might be mistaken in saying the 4 attention projections are the same. IIRC the GQA optimization only applies to K and V, which we see here:
AI
-
70B Model Parameters Issue Remains Unresolved
By
–
The 70b model has a different # params, but it's still not right
-
Clarifying parameter counts in 7B language models
By
–
That seems wrong? It doesn't ask which size model, but even 7b has way more params than that, and if it's talking about a single 7b layer it's not right either.
-
AI Model Fails to Recognize Content Accurately
By
–
Surprisingly (to me) it doesn't see it, and instead says stuff that's totally wrong.
-
LLMs cannot count model parameters limiting research capabilities
By
–
The fact that the best LLMs can't figure out how to count model parameters is an interesting reminder about how far away they are from doing novel research. This is the kind of analysis I do multiple times a day in my research, and is a critical plank in designing models.
-
LLM limitations beyond research frontier insights
By
–
My personal observation as a researcher is that LLMs are genuinely helpful in answering well-studied questions in areas where my own understanding is short of the research frontier. But as soon as I ask about anything beyond that frontier, I get nothing of use.
-
Multi-hop Reasoning and Tool Use Benchmark in Language Models
By
–
Answering this question correctly is a good mini benchmark challenge, IMO. It requires multi-hop search and reasoning (since the answer isn't directly in the Llama-2 paper, so requires checking its references too) and tools (since it requires doing multi-digit arithmetic).
-
Bing’s False Claims About Llama-2 Attention Mechanisms
By
–
Bing claims that Llama-2 doesn't use attention (not true), links to an irrelevant paper, and provides the world's least-useful table. Although it does have a handy button to export it to Excel…
-
LLM Prompt Challenge: Can Models Analyze Llama 2 Architecture?
By
–
Can anyone find a way to get any LLM to follow this prompt?:
"Compare the number of params in attention modules vs MLPs in the recently released llama 2 model." GPT-4 has no success, even with a lot of help. -
Server Speedup Factors in GPU Fast Hour Calculations
By
–
the server speedup factors into the gpu fast hour calculations whichs which are the same for all country