Sure, the same code works for all supported models. You can toggle these via the –checkpoint_dir flag pointing to the directory where you downloaded the model. Here's a list of the currently supported models:
@rasbt
-
Finetuning Llama 2 7B with Only 13 GB RAM on Single GPU
By
–
I forgot to mention the probably most important thing: Finetuning Llama 2 7B this way – only requires 13 GB RAM – and can thus comfortably run on a single GPU
-
LoRA Quantization and Evaluation Scripts for Llama-2 Models
By
–
Qualitatively, you can use python generate/lora.py –lora_path '… .pth' –quantize "bnb.nf4" –precision "bf16-true" –checkpoint_dir "…/Llama-2-7b-hf" Quantitatively, you can use the python eval/eval_harness scripts: https://
github.com/Lightning-AI/l
it-gpt/tree/main/eval
… -
QLoRA Single-GPU Limitations and Multi-GPU Future Plans
By
–
QLoRA right now it's single-GPU. I left multi-GPU support as a future todo since I am currently mostly focused on the NeurIPS LLM challenge. But it might well be working with multi-GPU settings already; it's just that I haven't tested. Could be that it requires switching from
-
Optional FlashAttention 2 Installation Feature
By
–
Yes, it's an optional feature. It uses FlashAttention by default, but you can optionally use FlashAttention 2 via pip install 'flash-attn>=2.0.0.post1' –no-build-isolation
-
QLoRA Support Added to Lit-GPT for All LLM Models
By
–
Just added QLoRA support for all LLMs in Lit-GPT: Llama 2, Falcon, Pythia, StableLM, and all others! You can use it via `python finetune/lora.py –quantize "bnb.nf4"` to save significant GPU memory. I've ran a few more benchmarks on the Gh PR here: https://
github.com/Lightning-AI/l
it-gpt/pull/275
… -
Multimodal LLMs Evaluation Without Image Datasets
By
–
I don’t think there’s a restriction wrt to multimodal LLMs though (eg using techniques like LLaMA-Adapter v2). But yeah, the evaluation datasets won’t include images.
-
Positional Encoding and Attention Mechanisms in Neural Networks
By
–
Hm good point and not sure. The positional encoding might not be required (there was a paper on that recently referred to as NoPE). Attention mechanisms I’d say are a generalization of MLPs (they could learn uniform weights if that’s useful). So I’d tend to say no.
-
Transformer publication timeline close to Bahdanau attention
By
–
Ah no! Interesting, I didn’t know it was published only 1 month after Bahdanau attention for RNNs
-
Self-Attention Mechanisms in Transformers: From RNNs to LLMs
By
–
What motivated self-attention mechanisms in transformer-based LLMs in the first place?
— Sebastian Raschka (@rasbt) 21 août 2023
A made a short video covering
– the limitations of RNNs
– the original (Bahdanau) attention mechanism for RNNs,
– how it all led to the original Transformer architecture used in LLMs https://t.co/oMLNlJMef6What motivated self-attention mechanisms in transformer-based LLMs in the first place? A made a short video covering
– the limitations of RNNs
– the original (Bahdanau) attention mechanism for RNNs, – how it all led to the original Transformer architecture used in LLMs