When I understood correctly, they used 16k for finetuning, but it can handle up to 100k even?
@rasbt
-
Why no 70B model for 100k context windows
By
–
Also why no 70B model?
1) Because of the 4x smaller dataset (compared to Llama 2 pretraining)?
2) to make 100k context windows work hardware-wise? -
Gen AI in Moderation: Balance Over Excessive Use
By
–
I think gen AI is fine if used in moderation, the same way CGI is fine in moderation. The problem is when it gets excessively used.
-
CodeLlama-Python 34B finetuned on unnatural instructions
By
–
It's CodeLlama-Python 34B finetuned on 15,000 unnatural instructions. What's an unnatural instruction? Would like to see a few examples 😛
-
CodeLlama 34B Significantly Improves Coding Performance Over Llama 2
By
–
Llama 2 is awesome, however, coding tasks were not its strong suite. (See HumanEval, a coding-related evaluation task from the paper Evaluating Large Language Models Trained on Code.) The new 34B CodeLlama model is twice as good as the original 70B Llama 2 model and closes the
-
AI enthusiast skeptical about AI integration in everyday products
By
–
I really enjoy working on AI, but I don’t really want AI in every product I am using.
-
Float16 vs Bfloat16 Conversion Rounding Issues
By
–
Wouldn’t that be only a problem the other way around, going from float16 -> bfloat16?
Since float16 supports more digits after the decimal points, going bfloat16 -> float16 there shouldn’t be no rounding? -
Skepticism about few-shot learning effectiveness and token costs
By
–
5-50 examples, I am skeptical that this can even produce good results. Hm, on their website they include a cost example with 100,000 tokens, so this must surely be more than 5-50 examples?
-
Hardware Dependencies and Limitations for Metal Support
By
–
Interesting, thanks for sharing. This sounds pretty cool actually. But afaik certain features are also hardware dependent, like bfloat16, so I am not sure if Metal or macOS can address that. Would probably require new chips as well.
-
llama.cpp inference and gptq quantization techniques exploration
By
–
Oh, reading a bit more about llama.cpp (
https://
github.com/ggerganov/llam
a.cpp
…), that's only inference, not training? I haven't tried since I don't have the model checkpoints on my laptop, but you may be able to use gptq.int4 quantization then: https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/quantize.md
…