Yeah I happen to know how hard it is because I tried to figure it out directly from the paper, and I had to go all the way back to the Palm paper to get the MLP size details — each paper tends to say "our arch is just like except for…"
RESEARCH
-
GQA Optimization: K and V Attention Projections Explained
By
–
Ah well it's certainly at least done way better than the others at finding something relevant! However it looks like this article might be mistaken in saying the 4 attention projections are the same. IIRC the GQA optimization only applies to K and V, which we see here:
-
70B Model Parameters Issue Remains Unresolved
By
–
The 70b model has a different # params, but it's still not right
-
Clarifying parameter counts in 7B language models
By
–
That seems wrong? It doesn't ask which size model, but even 7b has way more params than that, and if it's talking about a single 7b layer it's not right either.
-
LLMs cannot count model parameters limiting research capabilities
By
–
The fact that the best LLMs can't figure out how to count model parameters is an interesting reminder about how far away they are from doing novel research. This is the kind of analysis I do multiple times a day in my research, and is a critical plank in designing models.
-
LLM limitations beyond research frontier insights
By
–
My personal observation as a researcher is that LLMs are genuinely helpful in answering well-studied questions in areas where my own understanding is short of the research frontier. But as soon as I ask about anything beyond that frontier, I get nothing of use.
-
Multi-hop Reasoning and Tool Use Benchmark in Language Models
By
–
Answering this question correctly is a good mini benchmark challenge, IMO. It requires multi-hop search and reasoning (since the answer isn't directly in the Llama-2 paper, so requires checking its references too) and tools (since it requires doing multi-digit arithmetic).
-
LLM Prompt Challenge: Can Models Analyze Llama 2 Architecture?
By
–
Can anyone find a way to get any LLM to follow this prompt?:
"Compare the number of params in attention modules vs MLPs in the recently released llama 2 model." GPT-4 has no success, even with a lot of help. -
LLMs Hallucinations: Data Quality and Source Diversity Solutions
By
–
LLMs often make up answers, intending to complete the text instead of recognizing their flaws. Mitigating these biases and illusory tendencies requires careful data cleaning and the use of varied, reliable sources.
-
AI Model Hallucinations and Biases: Major LLM Challenges
By
–
Is your AI model giving you unexpected results? Hallucinations and biases are the biggest challenges we face with large language models (LLMs) like ChatGPT. The gravity of the problem intensifies, especially when these models power commercial applications.