oh man, people are focusing on "which paper" I read. I actually read three papers yesterday:
1. Are Open-Source LLM's catching up by @HailinChen3 et. al.: https://
arxiv.org/abs/2311.16989
2. System 2 Attention by @jaseweston and @tesatory : https://
arxiv.org/abs/2311.11829
3. (Part-way through
AI
-
Reading Three Key Papers on Open-Source LLMs and Attention
By
–
-
Reading Three AI Papers: Open-Source LLMs and System 2 Attention
By
–
oh man, people are focusing on "which paper" I read. I actually read three papers yesterday:
1. Are Open-Source LLM's catching up by @HailinChen3 et. al.: https://
arxiv.org/abs/2311.16989
2. System 2 Attention by @jaseweston and @tesatory : https://
arxiv.org/abs/2311.11829
3. (Part-way through -
Gemini Pro Launch and AI in Learning Development
By
–
Full story here! (W thanks to @lgavish
, @l2k
, Christian Kleinerman @SnowflakeDB and @JulianLaneve for chatting about the topic). Also notes on Gemini Pro launch + @kdaigle talking about AI as an L&D tool at Fortune Brainstorm AI. -

RAG Strategies for Semi-Structured Data in Documents
By
–
overview: Strategies for RAG on semi-structured data We've seen great interest in RAG apps on docs w/ a mixture of tables and text (financial docs, whitepapers, technical manuals). But, this brings up a few tricky questions:
1/ Can I just stuff semi-structured docs -

Weak Supervision Enables Large Models to Match Human-Level Performance
By
–
Large pretrained models have excellent raw capabilities—but can we elicit these fully with only weak supervision? GPT-4 supervised by ~GPT-2 recovers performance close to GPT-3.5 supervised by humans—generalizing to solve even hard problems where the weak supervisor failed!
-
Weak-to-Strong Generalization: Beyond RLHF for Superalignment
By
–
Naive weak supervision isn't enough—current techniques, like RLHF, won't be sufficient for future superhuman models. But we also show that it's feasible to drastically improve weak-to-strong generalization—making iterative empirical progress on a core challenge of superalignment
-
$10M ML Research Grants Launched for Community
By
–
There is lots of low-hanging fruit and many promising directions for future work. We think this is an exciting opportunity for the ML research community. To kickstart more research, we're launching $10M in grants:
-

Weak-to-Strong Generalization: Supervising Smarter AI Systems
By
–
In the future, humans will need to supervise AI systems much smarter than them. We study an analogy: small models supervising large models. Read the Superalignment team's first paper showing progress on a new approach, weak-to-strong generalization: https://
openai.com/research/weak-
to-strong-generalization
… -
Claude Turbo 128k Context Window Drops to 64k in 4.5
By
–
Interesting that Turbo has a 128k context window and 4.5 seemingly drops to 64k.