Oh, reading a bit more about llama.cpp (
https://
github.com/ggerganov/llam
a.cpp
…), that's only inference, not training? I haven't tried since I don't have the model checkpoints on my laptop, but you may be able to use gptq.int4 quantization then: https://
github.com/Lightning-AI/l
it-gpt/blob/main/tutorials/quantize.md
…
CODE
-
llama.cpp inference and gptq quantization techniques exploration
By
–
-
QLoRA 4-bit NormalFloat format supported only on Nvidia GPUs
By
–
Ah sorry, I meant M1/M2 chips (not specifically M1/2 CPUs). As far as I know, the 4-bit NormalFloat format that is used in QLoRA is currently only supported on Nvidia GPUs (
https://
github.com/TimDettmers/bi
tsandbytes/issues/485
…). Maybe the repo you mentioned uses a different type of quantized training. -
SQL QA Integration Improves Context Management for AI Applications
By
–
It’s often hard to bring in appropriate context to do sql qa This integration should help with that!
-
Alternative Precision Flag for Non-GPU Machines
By
–
I haven't tried on non-GPU machines, but maybe the following works:
"–precision 16-true" instead of "–precision bf16-true" -
BFloat16 Weight Clipping Limits and Extreme Value Representation
By
–
Hm, so that means all weights above values 65504 and below -65504 will be clipped. Does anyone know of there are numbers that large/small in the original bfloat16 representation?
-
Natural Language Processing Course UT Austin Lecture Videos and Materials
By
–
Natural Language Processing – UT Austin Lecture videos: https://
youtube.com/playlist?list=
PLofp2YXfp7TZZ5c7HEChs0_wfEfewLDs7
… Website(for slides, readings): https://
cs.utexas.edu/~gdurrett/cour
ses/online-course/materials.html
… -
Evaluating RAG Systems: Insights from Application Developers
By
–
It's always insightful to hear from actual application developers on how they approach evaluation That's why I'm excited to have @pedro_pxp on our webinar tomorrow that is focused on evaluating RAG systems! https://
crowdcast.io/c/bnx91nz59cqq -
Fine-tuning LLaMA2 Workshop: One-Day In-Person Event
By
–
If you want to explore finetuning LLaMA2, we'll be talking at and helping out with a 1 day in-person event focused explicitly on finetuning OSS models Hopefully this guide will come in handy! RSVP here (s/o @swyx and @NaderLikeLadder for organizing): https://
partiful.com/e/T4ngRPaU2uUT
XM8pN17d
… -
SQLChain Upgrade: Multi-Database QnA Chatbot Support
By
–
SQLChain upgrade We have upgraded SQLChain to now supports MySQL, MsSQL, PostgreSQL and the default SQLite! You can now have a QnA chatbot answering questions about your databases
-
Serp API Now Available as Document Loader for Agents
By
–
Serp API Loader Previously you can use Serp as tool for agent, now you can use Serp as a document loader! Thanks @fantastigenius for the contribution!