The Matrix Cookbook, https://
www2.imm.dtu.dk/pubdb/pubs/327
4-full.html
…
@rasbt
-
The Matrix Cookbook Reference Guide
By
–
-
Praise for Olmo Research Paper Over LLM Whitepapers
By
–
Nice and so well deserved! Loved the Olmo paper. It’s an actual research paper, not a whitepaper like those other LLM reports.
-
Quantization-aware training recommended for most techniques
By
–
Yes, for most techniques you’d want to do
Quantization-aware training. -
Quantization-Aware Training: Keeping Weights as Floats During Training
By
–
Yeah you can’t convert weights to int for training because you need to calculate the gradients during training. There are lots of different techniques for QAT, maybe this implementation helps: https://
pytorch.org/blog/quantizat
ion-aware-training/
… -
Claude’s Diagram Creation Capabilities Impress Developer
By
–
Wow this looks indeed spot on. Haven't used/tried Claude for diagrams, yet, and this makes me feel like I've been living under a rock (Creating the diagrams for my blogs & books is easily the most work)
-
Attention Mechanism Chapter: Challenging but Rewarding Learning Experience
By
–
Nice! The attention mechanism chapter might be the scariest one, but I think it gets more fun and rewarding after this one! (But lots of people told me that they esp. like the attention chapter…we will see )
-
Qwen Models: Underestimated Performance and NeurIPS Challenge Success
By
–
Qwen models are great. They are notoriously underestimated but have consistently performed well. I recall that most, if not all, of the winning solutions in the LLM Efficiency Challenge at NeurIPS last year were based on a Qwen model. I should make a tutorial some time
-
PyTorch installation with conda and CUDA tools setup
By
–
I can’t recall having issues with it when using PyTorch. I still use conda (I just pop install PyTorch), so I could do ‘conda install cuda-nvcc-tools’ I guess
-
CPU Training Speed Sufficient for Medium-Sized Datasets
By
–
I was recently training a model on a ~100k row & 20 column dataset. Took like ~2 sec on a CPU. Granted it was faster on the GPU (tried it just for fun) but it was already so fast that it’s kind of overkill for my use cases
-
Appreciation for useful content with bonus materials recommendation
By
–
Thanks for the kind words. Super glad it’s useful! Btw I think 7 is the most fun one! Enjoy! (Also, there’s lots of bonus materials if you are bored, from LoRA to DPO from scratch!)