@huggingface | SN-13B-8k-Instruct, a 13 billion parameter model https://
huggingface.co/sambanovasyste
ms/SN-13B-8k-Instruct
… Join our Discord to ask questions and discuss. https://
discord.gg/8z2Pe7cpRv Read the full blog and technical details at https://
sambanova.ai/blog/training-
long-sequence-size-models-on-sambanova/
… (10/10)
LLMS
-
SambaNova Releases SN-13B-8k-Instruct Language Model
By
–
-
Curriculum Learning Boosts Model Training From 2K to 8K Tokens
By
–
To train this model, we use Curriculum Learning by gradually increasing the token lengths trained on from 2K to 8K. We additionally train on an instruction tune dataset sampled from various popular sources and curated in-house. (4/10)
-
Dataset Curation for Long Sequence Instruction Following
By
–
To curate a dataset that encourages long sequence instruction following, we use two techniques. First we find tasks that truly benefit from longer sequences and add them to our instruction tuning datasets. (5/10)
-
Instruction List Technique Enhances Long Sequence Attention
By
–
Second, we use a technique that we developed called instruction list that synthetically creates tasks that encourage more long sequence attention and instruction following. (6/10)
-
13B Parameter Long Sequence Model Achieves Competitive Accuracy
By
–
Using this recipe, we are able to train a competitive long sequence model at 13B parameter scale. We achieve 2-12 points better accuracy across a wide variety of long sequence tasks from Scrolls and ZeroScrolls. (7/10)
-
SN-13B-8K-Instruct Outperforms MPT, XGen, LLAMA2 on Long Sequences
By
–
We compare SN-13B-8K-Instruct with MPT, XGen and LLAMA2 and find that this model achieves better scores on long sequence suite derived from Scrolls and validation set of ZeroScrolls, a benchmark developed by @TelAvivUni and @MetaAI
. (3/10) -
SN-13B-8K-Instruct Open Source Model Outperforms Long Sequence Competitors
By
–
Today we are excited to introduce and open source SN-13B-8K-Instruct, an 8K SS model that achieves better accuracy than leading long sequence open source models specialized for long sequence size tasks. (1/10)
-
Sambanova SN-13B-8k-Instruct Model Now Available for Download
By
–
This model is available for download on @huggingface for people to try: https://
huggingface.co/sambanovasyste
ms/SN-13B-8k-Instruct
… (2/10) -
Fine-tuning smaller models for specialized tasks with custom data
By
–
Yes, it's more impressive. But if you are a small company that specializes on a specific task and has some custom data, I'd say that finetuning a smaller (e.g. 1.5 B) model is the way to go.
-
Fine-tuning Custom LLMs vs Few-Shot Prompting Performance Comparison
By
–
2/3 For fairness: The DeBERTa-1.5B model was likely finetuned on the training data whereas Llama 2 was used via few-shot prompting. In that case, it highlights once more that finetuning custom LLMs remains worthwhile.