Oct 2022: I ask GPT-3, who won the Super Bowl the year Justin Bieber was born? GPT-3 says Packers. It’s the Cowboys. Example is added to LangChain. Other LLMs answer. Answers are added to GitHub. Now, citing GPT4All, Google says it’s the Buffalo Bills — who have never won:
LLMS
-
GQA Architecture in 70B Language Models Explained
By
–
Oh wait this article is correct for a single layer of a 13b model — in fact it's only the 70b model that uses GQA AFAICT! 😀
-
Yi-6B Language Model Now Available on Hugging Face
By
–
Yi-6B available here https://
huggingface.co/01-ai -
Challenges in Finding LLM Architecture Details Across Papers
By
–
Yeah I happen to know how hard it is because I tried to figure it out directly from the paper, and I had to go all the way back to the Palm paper to get the MLP size details — each paper tends to say "our arch is just like except for…"
-
GQA Optimization: K and V Attention Projections Explained
By
–
Ah well it's certainly at least done way better than the others at finding something relevant! However it looks like this article might be mistaken in saying the 4 attention projections are the same. IIRC the GQA optimization only applies to K and V, which we see here:
-
70B Model Parameters Issue Remains Unresolved
By
–
The 70b model has a different # params, but it's still not right
-
Clarifying parameter counts in 7B language models
By
–
That seems wrong? It doesn't ask which size model, but even 7b has way more params than that, and if it's talking about a single 7b layer it's not right either.
-
AI Model Fails to Recognize Content Accurately
By
–
Surprisingly (to me) it doesn't see it, and instead says stuff that's totally wrong.
-
LLMs cannot count model parameters limiting research capabilities
By
–
The fact that the best LLMs can't figure out how to count model parameters is an interesting reminder about how far away they are from doing novel research. This is the kind of analysis I do multiple times a day in my research, and is a critical plank in designing models.