yeah once this paper was released i was like “ohhh so fineweb isnt simply commoncrawl plus plus” and it all clicked into place. @eugeneyan pointed me to this apple paper we’ve talked about on the pod
MACHINE LEARNING
-
Best Machine Learning Books: Essential Reading Guide
By
–
Best #Books for Machine Learning You Must Read! #BigData #Analytics #DataScience #IoT #IIoT #PyTorch #Python #RStats #TensorFlow #Java #JavaScript #ReactJS #CloudComputing #Serverless #DataScientist #Linux #Programming #Coding #100DaysofCode https://
geni.us/Best-A-I-Must-
Read
… -
Essential Data Science Books for ML and Analytics
By
–
Top Books You Must Read! #BigData #Analytics #DataScience #IoT #IIoT #PyTorch #Python #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Books #Programming #Coding #100DaysofCode https://
geni.us/15-DSci-Books -
Popular Machine Learning Algorithms Explained Comprehensive Guide
By
–
Popular #MachineLearning #Algorithms Explained. #BigData #Analytics #DataScience #AI #IoT #IIoT #Python #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Programming #Coding #100DaysofCode https://
geni.us/10-Popul-ML-Ex
plained
… -
100 Machine Learning Projects with Python Guide
By
–
100 #MachineLearning Projects with #Python. #BigData #Analytics #DataScience #AI #MachineLearning #IoT #IIoT #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Programming #Coding #100DaysofCode https://
geni.us/100-ML-Project
s-Python
… -
Vinod Khosla 12 Predictions for the Future of Technology
By
–
Watch the full TED talk with all 12 predictions for the future of tech here: https://
ted.com/talks/vinod_kh
osla_12_predictions_for_the_future_of_technology?language=en
… -
Khosla’s 12 Tech Predictions: AI Optimism and Empathy
By
–
Four of my favorite slides from silicon valley investing GOAT Vinod Khosla's new TED Talk: 12 predictions for the future of technology. Was dope to co-host this session and see it live. TL;DR Techno-optimism with a heathy dose of empathy. 1. You don't have to choose between AI
-
Pretraining Datasets and Evaluation Metrics for Multilingual Models
By
–
Yes, definitely, my last tweet few seconds ago is also on this point. And many pretraining datasets also care about e.g. multilignual, code, math, etc., so it's not clear how those evals would be affected.
-
GPT-3 Training Surpasses Expected Performance on FineWeb Dataset
By
–
Example here is the llm.c GPT-3 (124M) training on FineWeb (figure cropped at 250B tokens), we seem to surpass GPT-3 HellaSwag (green line) at ~150B tokens, per paper expected this to be at 300B tokens. Will re-run with FineWeb-Edu. I do want to be a bit careful on conclusions
-
llm.c Outperforming GPT-2/3 with Fewer Training Tokens
By
–
In llm.c pretraining we were already mildly perplexed why seem to be outperforming GPT-2 & 3 (124M) training on just 10B tokens instead of something closer to 100-300B, per the original papers. I suspect a good chunk of it may be just the dataset quality, so I'm eager to retrain