On the model side, longer context length for the 7B! If you could train over the new Mistral 7B w/ 32K that’d be awesome. As for the dataset, I haven’t found many issues with it 🙂
OPEN SOURCE
-
Team Commitment to Open Source Through Hugging Face Releases
By
–
A bit biased of course but for me, the best indicator of how much a team cares about community and open-source is the level of effort they put into their Hugging Face model/dataset/space/paper releases! It means they're thinking a lot about how to make it easier for all to
-
Databricks LLM Training Cost: Enterprise Model ROI Analysis
By
–
The Databricks LLM apparently cost $10M to train, which is a price I have seen given for a GPT-3.5 class model multiple times. Given that pricing, training your own corporate model versus open source needs to deliver a lot of benefit. I haven't seen evidence for that value, yet.
-
DBRX Becomes Number One Trending Model on Hugging Face
By
–
Not a surprise but DBRX is already #1 trending on HF!
-
TheProfessor: Advanced LLM Merging with Conversational and Scientific AI
By
–
TheProfessor is a merge of #LLMs, created with help from @AbacusAI
's #GenerativeAI team. It can be used for many things with its conversational #AI, reasoning, scientific, medical, & mathematical skills interactive brainstorming: https://
huggingface.co/abacusai/ThePr
ofessor-155b
… -
AI21 Labs Introduces Jamba: New AI Architecture Model
By
–
Meet Jamba http://
ai21.com/jamba Build with Jamba https://
huggingface.co/ai21labs/Jamba
-v0.1
… Read more http://
ai21.com/blog/announcin
g-jamba
… #Jamba #Mamba #Transformer #AIarchitecture #SSM -
Jamba: Revolutionary SSM-Transformer Open Model with 3X Throughput
By
–
Introducing Jamba, our groundbreaking SSM-Transformer open model! As the first production-grade model based on Mamba architecture, Jamba achieves an unprecedented 3X throughput and fits 140K context on a single GPU. Meet Jamba http://
ai21.com/jamba Build on @huggingface -
Developer Credits Training Code Repository Contribution
By
–
I pulled the training code from one of my old notebooks from over a year ago, which was based on yours. I’m so sorry, I totally forgot where I got the training code from. Will absolutely add credit to the repo — love your work (esp. LazyMergeKit!!)
-
Play with DBRX-Instruct Model on Hugging Face Spaces
By
–
You can play with the model here: https://
huggingface.co/spaces/databri
cks/dbrx-instruct
…