And if you are looking for inference mode quantization, you can use `–quantize "bnb.nf4" with the "generate/base.py" scripts as well.
I am currently using that for running CodeLlama 34B models.
@rasbt
-
Inference Mode Quantization with BNB NF4 for CodeLlama
By
–
-
QLoRA-style Quantization Added with BNB NF4 Support
By
–
Yes! Just added QLoRA-style quantization the other week, too. You can use it via –quantize "bnb.nf4" for example. I have some benchmarks here:
-
Code Llama Integration Added to Lit-GPT Repository
By
–
And thanks to another kind contribution, Code Llama is making its way into Lit-GPT, too!! https://
github.com/Lightning-AI/l
it-gpt/pull/472
… -
Whitespace in last name: tokenization artifact question
By
–
What happened to the whitespace in the last name? A tokenization artifact?
-
Question about Stanford evaluations and HELM HumanEval
By
–
Thanks for sharing! What are the Stanford evals? You mean the HumanEval version in HELM?
-
Lemur: New Code LLM Based on Llama 2
By
–
Oh, and there was also Lemur last week. A code LLM not based on Code Llama but based on LLama 2 directly! https://
xlang.ai/blog/openlemur -
Fine-tuning Llama 2 7B on Single GPU A100 and Google Colab
By
–
You can finetune a 7B Llama 2 model on a single GPU A100 for example, so you could even use Google Colab. But in general, most universities and companies are supportive of open source these days.
-
MoE vs Specialized Models: Questioning Current Practical Utility
By
–
Yeah. But unless you want to have a generalist model for all languages, is it really that useful to have a MoE approach compared to say the CodeLlama-34B-Python version? I mean eventually things might move to MoE but maybe it’s not that big of a deal atm.
-
GPU constraints influence research direction choices
By
–
One might argue that with more GPUs, they would have worked on BLOOM 2 instead of QLoRA etc.
-
Clarification on HumanEval benchmark reference
By
–
Ah I can see how that's misleading, I mean HumanEval as in https://
github.com/openai/human-e
val
…