Trying to develop the next huge LLM is fun, but how about a side project with a more reasonable scope like the 1 LLM + 1 GPU + 1 Day NeurIPS efficiency challenge: https://
llm-efficiency-challenge.github.io Below a list of the approved models (PS: Lit-GPT was chosen as the official starter kit )
@rasbt
-
NeurIPS LLM Efficiency Challenge: Train Model in One Day
By
–
-
Joining DS/ML Book Club Discussion on Machine Learning and AI
By
–
Looking forward to joining @sophiamyang 's
DS/ML book club tomorrow over lunch (1 pm CT) to talk & discuss Machine Learning Q and AI (
https://
leanpub.com/machine-learni
ng-q-and-ai/
…)! This book is a bit more advanced than my previous ones, covering concepts such as – Multi-GPU training paradigms. -
Tie-breaking fix improves nearest neighbor classification accuracy to 71.91%
By
–
Small update: In the orig code example in the paper, tie breaking was based on the lowest class label index, i.e., it was biased towards class 0. When I fixed the tie-breaking to select the closest neighbor, I could get a 70.05% -> 71.91% improvement (
https://
github.com/rasbt/nn_plus_
gzip
…). -
Method’s Limitations Compared to Large Language Models
By
–
Oh yes, of course! The method is nice, but yeah, it would be a bit weird if it could outperform LLMs on things that LLMs are good at.
-
Machine Learning Class Coverage of KNN Lecture Notes
By
–
I usually cover these in my machine learning classes https://
sebastianraschka.com/pdf/lecture-no
tes/stat451fs21/02-knn__slides.pdf
… -
Background execution and minimal pseudocode implementation approach
By
–
I let it run in the background, I think it was a day or so. But yeah, you can be much smarter about the implementation, I just kept it minimal, implementing the exact pseudocode they had in the paper
-
Dataset Differences and Acknowledgment of Top-2 Accuracy Insights
By
–
I used a different dataset (IMDB Movie Review), so I can't directly compare. But thanks for sharing these insights regarding the top-2 accuracy issue!
-
kNN Gzip Method Outperforms Cosine Similarity on IMDb Reviews
By
–
A quick experiment that took a bit due to kNN scaling on large datasets: the kNN + Gzip method is a bit better than cosine similarity on count vectors. On the IMDb Movie review dataset:
– 70% test acc for gzip
– 65% test acc for cosine distance My (re)implementation code: -
Flash Attention: Optimizing Hardware Memory Use and I/O
By
–
Oh yeah, these methods are orthogonal. Flash attention is essentially optimizing hardware memory use and I/O
-
Flash Attention 2 Integration Reduces Lit-GPT Runtime by 11%
By
–
My colleagues already added Flash Attention 2 to our Lit-GPT repo!
So if you are working on the NeurIPS LLM Efficiency Challenge (for which Lit-GPT is the official starter kit, https://
llm-efficiency-challenge.github.io), you can shave ~11% off your total runtime