And if you are looking for inference mode quantization, you can use `–quantize "bnb.nf4" with the "generate/base.py" scripts as well.
I am currently using that for running CodeLlama 34B models.
OPEN SOURCE
-
Inference Mode Quantization with BNB NF4 for CodeLlama
By
–
-
QLoRA-style Quantization Added with BNB NF4 Support
By
–
Yes! Just added QLoRA-style quantization the other week, too. You can use it via –quantize "bnb.nf4" for example. I have some benchmarks here:
-
Kelsey Hightower critiques Terraform description in technical discussion
By
–
Worth listening to @kelseyhightower in a conversation with @bcantrill (
https://
youtube.com/watch?v=13ctYO
u8TsA#t=39m40s
…). Kelsey, as a contributor to Terraform, clearly isn't impressed with McJannet's description, and rightly so. -
Efficient object detection with YOLOv8 and KerasCV
By
–
Efficient object detection with YOLOv8 and KerasCV:
-
Open-source AI grants empower next generation tech startups
By
–
Inspiring initiative by @BornsteinMatt @pmarca @a16z
! Open-source AI is critical to empower a whole new generation of tech startups & we need more amazing contributors like these folks who got grants (all links below)! Blogpost: https://
a16z.com/2023/08/30/sup
porting-the-open-source-ai-community/
… Grant recipients: -
Llama2 Powers Fun Application on Anyscale Compute
By
–
This is very fun! And – really cool that this is powered by Llama2 on @anyscalecompute
-
Cerebras MBZUAI Develop Arabic-Optimized Language Model Techniques
By
–
(2/2) Researchers at Cerebras, Inception, and MBZUAI developed techniques to:
-Create a custom tokenization scheme optimized for Arabic -Identify the proper mix of English, Arabic, and code data -Augment Arabic tokens with machine-translations Models: https://
huggingface.co/inception-mbzu
ai
… -
Jais: Top-Performing Open-Source Arabic LLM Model
By
–
(1/2) Jais is the top-performing open-source Arabic LLM in the world. Better than Bloom, Llama, Falcon, etc. It is a 13B parameter model that was trained on a brand-new multilingual dataset. Read the technical paper: https://
inceptioniai.org/jais/docs/Tech
nicalpaper.pdf
… -
Code Llama Now Available on Poe Platform
By
–
Code Llama is now available on Poe! Note that many of the Code Llama demos floating around during the last week on other services had bugs; we took some time to make sure our implementation works accurately to the underlying model.
-
Code Llama Integration Added to Lit-GPT Repository
By
–
And thanks to another kind contribution, Code Llama is making its way into Lit-GPT, too!! https://
github.com/Lightning-AI/l
it-gpt/pull/472
…