@jeremyphoward

–

12 November 2023 2h06

Surprisingly (to me) it doesn't see it, and instead says stuff that's totally wrong.

LLMs cannot count model parameters limiting research capabilities

By

–

12 November 2023 1h59

The fact that the best LLMs can't figure out how to count model parameters is an interesting reminder about how far away they are from doing novel research. This is the kind of analysis I do multiple times a day in my research, and is a critical plank in designing models.

LLM limitations beyond research frontier insights

By

–

12 November 2023 1h59

My personal observation as a researcher is that LLMs are genuinely helpful in answering well-studied questions in areas where my own understanding is short of the research frontier. But as soon as I ask about anything beyond that frontier, I get nothing of use.

Multi-hop Reasoning and Tool Use Benchmark in Language Models

By

–

12 November 2023 1h59

Answering this question correctly is a good mini benchmark challenge, IMO. It requires multi-hop search and reasoning (since the answer isn't directly in the Llama-2 paper, so requires checking its references too) and tools (since it requires doing multi-digit arithmetic).

Bing’s False Claims About Llama-2 Attention Mechanisms

By

–

12 November 2023 1h59

Bing claims that Llama-2 doesn't use attention (not true), links to an irrelevant paper, and provides the world's least-useful table. Although it does have a handy button to export it to Excel…

LLM Prompt Challenge: Can Models Analyze Llama 2 Architecture?

By

–

12 November 2023 1h59

Can anyone find a way to get any LLM to follow this prompt?:
"Compare the number of params in attention modules vs MLPs in the recently released llama 2 model." GPT-4 has no success, even with a lot of help.

Microsoft’s $10B OpenAI Investment Pushes Valuation to $1T

By

–

12 November 2023 1h32

Rumors are that the last MSFT investment in OAI was $10b, so cap would now be $1t presumably?

Corporate Governance Structure Effectiveness in Tech Questioned

By

–

11 November 2023 23h02

I think that's a fair claim, although I'm not sure I agree. Governance and US corporate law is complex, and I'm somewhat skeptical this structure will turn out to be effective. But fingers crossed I'm wrong