Llama 3.1 paper, Section 4.3.6.
@karpathy
-
Investing in Distributed Creator Intelligence Over Concentrated Resources
By
–
I'd be a lot more inclined to invest $10M into 2000 creators. The distributed intelligence and creativity of the crowd feels underutilized.
-
LLMs Reaching LHC-Level Complexity in Infrastructure and Development
By
–
LLMs as an artifact are trending to the complexity of something like the LHC. This is clear when you look at the datacenter computronium build out but it's a lot more than that – a large chunk is digital and much harder to see/appreciate, it's just a bunch of people on a laptop.
-
Karpathy Plans Multiple Llama 3.1 Fine-tuning Projects
By
–
Yep I’d like to do many Llama 3.1 finetunes, coming up.
-
App Store Early Years Were Gimmicky, AI Will Take Time
By
–
My opinion on this has changed at a recent Sequoia event where they compared to iOS. The first ~3 years of the App Store were all kinds of gimmicky apps. I think it just takes a while to process a new thing, figure out what it is and isn't and package it into products. Image:
-
Meta’s Llama 3.1 405B: First Open Frontier LLM Available to All
By
–
Huge congrats to @AIatMeta on the Llama 3.1 release!
Few notes: Today, with the 405B model release, is the first time that a frontier-capability LLM is available to everyone to work with and build on. The model appears to be GPT-4 / Claude 3.5 Sonnet grade and the weights are -
AGI Speed Makes AI Interaction Instantly Pleasing and Natural
By
–
This is so cool. Feeling the AGI – you just talk to your computer and it does stuff, instantly. Speed really makes AI so much more pleasing.
-
New favorite LLM test reveals inconsistent performance across SOTA models
By
–
Wow, this has just become my favorite LLM test. I missed that this doesn't work but it really doesn't, even for SOTA LLMs. Seems to be a bit hit and miss, e.g. with GPT4o which failed 1/3 times, Claude failed 3/3 times.
-
Software Issues: System Prompts and Adversarial Examples
By
–
Of course, it's software.
Easy mode: a bad system prompt update.
Hard mode: an adversarial example in the context. -
System Robustness Against Component Failures and Adversarial Behavior
By
–
I just feel like this is the particular problem but not the *actual* deeper problem. Any part of the system should be allowed to go *crazy*, randomly or even adversarially, and the rest of it should be robust to that. This is what you want, even if robustness is very often at