Texting my gf, who is as outside of tech as can be, about llama day
@mattshumer_
-
LLaMA 3 Sequence Length Limitations Community Extensions
By
–
The main issue with the LLaMA 3 models is the sequence length… currently 8k. I'm confident the community will extend this pretty quickly.
-
Open-Source GPT-4-Level Models Now Freely Accessible Globally
By
–
We're entering a new world where GPT-4-level models are open-source and freely accessible. Absolutely massive.
-
Training 400B+ Parameter Model to Outperform GPT-4
By
–
They're training a 400B+ parameter version that should outperform GPT-4.
-
LLaMA 3 400B Matches Claude 3 Opus Performance
By
–
The craziest LLaMA 3 reveal: The 400B+ version of the model is **on par with Claude 3 Opus**, and it's still training. Soon, we'll have a better-than-Opus, fully open-source model. The implications are huge.
-
LLaMA 3 70B Outperforms Claude 3 Sonnet Cost-Effectively
By
–
Holy shit. LLaMA 3 70B cleanly beats Claude 3 Sonnet. Small enough to host at scale without breaking the bank.
-
Training LLMs: Avoiding Default Mode Through Data Curation
By
–
First you’d teach the model quite a bit from recent content (w/ tons of LLM outputs in it it’s likely, if we were to train this in normally, the LLM will likely get stuck in that default LLM ‘mode’ we know so well + make it harder to break out of this w/ post-training) So
-
Training Strategy: Pre-2021 Data Priority Over AI-Generated Content
By
–
With all the AI-generated content flooding the web There might be something to first training on content from 2021-on And then continuing to train on pre-2021 content
