Starting on using PoSE to extend LLaMA 3's context window, with @winglian
's help. This one was really fun to work on. Eval loss started insanely high, let's see how far down we can get it.
RESEARCH
-
Extending LLaMA 3 Context Window with PoSE Method
By
–
-
Wayfair improves model accuracy with data-centric AI approach
By
–
See how Wayfair improved model accuracy by adopting data-centric AI to curate high-quality training data. https://
buff.ly/3xPO6Cl -
LLaMA 3-Instruct Dataset Format Structure Guide
By
–
Specifically, I have a dataset in the LLaMA 3-Instruct format:
“`
system {{ system_prompt }}user {{ user_message_1 }}assistant {{ -
AI Systems Can Learn New Knowledge With Proper Implementation
By
–
False. It can learn new knowledge if done right.
-
LLaMA 3 Reaches 160K+ Context Window, Continuing Rapid Expansion
By
–
LLaMA 3, now with 160K+ context. Every day, we are pushing it higher.
-
SFTTrainer Response Template Training Logic Issue
By
–
Is it just me, or does the @huggingface SFTTrainer's approach to training only on completions make no sense? What if your response template shows up multiple times in your example? Am I misunderstanding this?
-
Postgres Renaissance: Vector Databases Give Way to SQL Evolution
By
–
if 2023 was the New Hope of vector dbs
— swyx 🐣 (@swyx) 25 avril 2024
then 2024 is Postgres Strikes Back@1st1 figured out how to add support for streaming RAG in 3LOC with deferred indexes in edgedb. pretty natural when you are already rewriting SQL for the AI/post-relational era! https://t.co/7iiY1XKNAa pic.twitter.com/5u7HFtpVG0if 2023 was the New Hope of vector dbs then 2024 is Postgres Strikes Back @1st1 figured out how to add support for streaming RAG in 3LOC with deferred indexes in edgedb. pretty natural when you are already rewriting SQL for the AI/post-relational era!
-
5th Accelerating AI Workshop Registration Open
By
–
Join us for the 5th Accelerating #AI Workshop, orchestrated by @CMCMicrosystems Register here –> http://
bit.ly/accelerating_ai -
Large Language Models as Agents: Action Prediction Framework
By
–
One intuitive way to think about how AI models (Large Language Models) can are used as an "agent" is to consider using the model to predict what actions to take, given a particular state of the world. This approach some times go by the name "Large Action Model". For example, you