LLaMA 3, now with 160K+ context. Every day, we are pushing it higher.
@mattshumer_
-
SFTTrainer Response Template Training Logic Issue
By
–
Is it just me, or does the @huggingface SFTTrainer's approach to training only on completions make no sense? What if your response template shows up multiple times in your example? Am I misunderstanding this?
-
Fine-tuning Models with Detailed Instructions for Faster Learning
By
–
Basically, in your fine-tuning data, include detailed instructions on how to behave — don't make the model guess at it. This way, it learns faster.
-
LLaMA 3 Extended Context Window Reaches 128K Tokens
By
–
It's been a week since LLaMA 3 dropped. In that time, we've:
– extended context from 8K -> 128K
– trained multiple ridiculously performant fine-tunes
– got inference working at 800+ tokens/second If Meta keeps releasing OSS models, closed providers won't be able to compete. -
System Prompts Quality Boosts Model Fine-Tuning Efficiency
By
–
High alpha trick for fine-tuning: Make your system prompts in your dataset really great. It'll help the model learn to do your task much faster, with less data. If you have lots of data, you can ignore this, but at small dataset sizes, this changes everything.
-
Strong AI Benchmarks Achievement Recognized
By
–
Benchmarks look really good… congrats @erhartford
! -
HyperWrite Helps Users Generate Ideas for Content Creation
By
–
It seems like it’d be a great fit! Many of our users use HyperWrite to help them with ideas, and then they put the final pieces together themselves.
-
Data Is Everything in AI Development
By
–
Yes, 100%. I’ve been saying this for a long time… the data is everything.
-
AI Writing Excellence and Software Operating Agent Development
By
–
Two main things: 1. We are the best AI for writing and writing-related tasks. Our models write naturally, have access to up-to-date info, can research + cite sources. + lots more. 2. We’ve been developing an AI agent that can operate software to get tasks done for you.
-
Dataset Quality: The Foundation of AI Model Performance
By
–
The dataset is everything. Great read: https://
nonint.com/2023/06/10/the
-it-in-ai-models-is-the-dataset/
…