yes, fair, but don't we call this simply "more training"? I dunno, is this really "finetuning"?
@rasbt
-
Self-supervised Learning and Autoregressive Next-word Prediction Clarification
By
–
Self-supervised = autoregressive, next-word prediction like in pretraining? That's what I had in mind with "like pretraining" but in a shorter way to make it fit into the 260 character limit
-
Three types of fine-tuning approaches for language models
By
–
[2/2] 2) Instruction finetuning: usually RLHF to generate a specific response. (But can also refer to 1b depending on the technique) 3) Long-context finetuning: like pretraining but w longer contexts (as per Code Llama paper)?
-
Defining and Categorizing LLM Fine-tuning Approaches
By
–
How do we define and categorize LLM "finetuning" these days? There's 1) Supervised finetuning: to 1a) train an encoder-LLM for classification or 1b) train a decoder-LLM to generate a specific response text. [1/2]
-
Inference Mode Quantization with BNB NF4 for CodeLlama
By
–
And if you are looking for inference mode quantization, you can use `–quantize "bnb.nf4" with the "generate/base.py" scripts as well.
I am currently using that for running CodeLlama 34B models. -
QLoRA-style Quantization Added with BNB NF4 Support
By
–
Yes! Just added QLoRA-style quantization the other week, too. You can use it via –quantize "bnb.nf4" for example. I have some benchmarks here:
-
Code Llama Integration Added to Lit-GPT Repository
By
–
And thanks to another kind contribution, Code Llama is making its way into Lit-GPT, too!! https://
github.com/Lightning-AI/l
it-gpt/pull/472
… -
Whitespace in last name: tokenization artifact question
By
–
What happened to the whitespace in the last name? A tokenization artifact?
-
Question about Stanford evaluations and HELM HumanEval
By
–
Thanks for sharing! What are the Stanford evals? You mean the HumanEval version in HELM?
-
Lemur: New Code LLM Based on Llama 2
By
–
Oh, and there was also Lemur last week. A code LLM not based on Code Llama but based on LLama 2 directly! https://
xlang.ai/blog/openlemur