I remember the BitNet repo may have example code for what you have in mind: https://
github.com/kyegomez/BitNet
@rasbt
-
BitNet Repository Reference for Implementation Examples
By
–
-
Ternary LLM and BERT Training: Starting with TernaryBERT Paper
By
–
I have never trained a ternary LLM or BERT language model. Looks like there are only few publicly available repos out there. I would maybe start with the "TernaryBERT: Distillation-aware Ultra-low Bit BERT" paper and go from there
-
PhD adviser styles: hands-on versus hands-off supervision approaches
By
–
These are all valid points, but yeah, your mileage may vary based on your PhD adviser. Some are very hands-on and prescribe very specific projects to work on (it's often limited by research grants), others are more hands-off and you come up with a topic that is in line with the
-
Working on tokenizer and embedding layer extensions for future release
By
–
Not yet. I started working on it (extending the tokenizer and modifying the embedding layers) but got busy with some other things… I plan to revisit that though during the winter holidays and will probably have something to share then!
-
LoRA Rank as an Unpredictable Hyperparameter in Model Fine-tuning
By
–
Yeah but interestingly I also sometimes find that 1/2x the rank works better. Like you said it may be the data distribution. But yeah overall it’s a hyperparameter that sometimes has its own mind
-
Small Models Performance for Spam Classification Task
By
–
Sebastian Raschka
How about these wrt to small models on a specific task (here: spam classification) -
LLM Coding Resources and Bonus Materials for Weekend Learning
By
–
If you are looking for something to read/study this weekend, I added lots of LLM-related bonus from-scratch coding resources over the last few months (from implementing Llama 3.2 to preference tuning with DPO): https://
github.com/rasbt/LLMs-fro
m-scratch?tab=readme-ov-file#bonus-material
… I hope you find them useful! -
Layer Selection Method: Random or Similarity-Based Approach
By
–
Interesting! How did you select these layers? Was this a random selection or did you sort them by input/output similarity?
-
Interleaving Tokens and Training Data for Vision Models
By
–
These are good questions! Regarding 1, yes, interleaving text and image tokens makes a lot of sense then. For part 2, not sure but I suspect you'd need to train the model on data that contains bounding box coordinates at the very least.
-
LLM Default Settings Run on CPU and GPU
By
–
Thanks for getting a copy! The default settings for the LLM in the book are such that it runs on CPU and GPU (it automatically uses a GPU if a compatible one is present). However, I made sure that everything works fine on a laptop (I tested everything on my MacBook Air