Finally got a chance to spend some time with the 77-page Llama 2 paper (
https://
arxiv.org/abs/2307.09288). Here are some takeaways at a glance. Appreciate that someone finally did a comprehensive supervised finetuning vs RLHF evaluation! (Lower right)
@rasbt
-
Llama 2 Paper Analysis: Supervised Finetuning vs RLHF Evaluation
By
–
-
InstructGPT and ChatGPT represent significant advancement beyond previous models
By
–
I’d say that InstructGPT, ChatGPt, and everything after (which were all released after this paper) were quite a leap forward
-
Fix for torchtext dataset class labels in machine learning book
By
–
Thanks! I think the main issue was that newer versions of torchtext apparently changed the class labels in that dataset. It should also be fixed in the original repo here now: https://
github.com/rasbt/machine-
learning-book/blob/main/ch15/ch15_part2.ipynb
… Sorry for the hassle! -
Training efficiency boost through compressed data caching optimization
By
–
I haven't! Thank for sharing! But a note on the efficiency: caching the compressed train lengths makes absolute sense, the speed up is approx 3.8 sec/iter -> 3.1 sec/iter
-
Neural Networks and Gzip versus Large Language Models Deep Dive
By
–
Following up on some discussions of the "NN + Gzip versus LLMs" methods a few weeks back here's my little deep dive with a reimplementation and additional experiments: https://
magazine.sebastianraschka.com/p/large-langua
ge-models-and-nearest/
… This includes fixing the tie-breaking and more! -
Using PyTorch Vision ViT as Baseline for Model Training
By
–
No that I am aware (but I also haven't looked too deeply). I would probably use the original ViT in Torchvision as a baseline: https://
pytorch.org/vision/main/_m
odules/torchvision/models/vision_transformer.html#vit_b_16
… I have a companion article here showing how to accelerate the training on that model: -
Open-Source Model Fine-tuning Methods and Training Flexibility
By
–
Yes, any finetuning method is allowed, afaik. Technically, you could also just further (pre)train the model. Anything that runs <24 h on open source code and openly available data.
-
Lit-GPT: Unified Codebase for Decoder Analysis and Comparison
By
–
It’s kind of hard to analyze across repos due to implementation differences and details along the data loading and finetuning pipelines. That’s where I’d say Lit-GPT is useful because it makes the set of relevant decoders available in the same unified code base
-
Model Documentation Gaps and Empirical Analysis Importance
By
–
Not that I am aware of. On top of that several models don’t even have thorough papers themselves because people are currently rushing them out. I guess the best way is really some empirical analysis coupled with some knowledge bits like falcon and llama2 use multiquery attention
-
Mastering LLM Basics as Foundation for Innovation
By
–
I don't disagree, but I think that in order to innovate it can also be helpful to have the basics down. I.e., implementing and running state-of-the-art LLMs is kind of part of the baseline exercise before going further from there.