no. people misunderstand chinchilla.
chinchilla doesn't tell you the point of convergence.
it tells you the point of compute optimality.
if all you care about is perplexity, for every FLOPs compute budget, how big model on how many tokens should you train?
for reasons not fully
MACHINE LEARNING
-
Chinchilla Scaling Laws: Compute Optimality vs Convergence Point
By
–
-
Reclaiming Predictive Data Value Through Knowledge Graphs
By
–
Amy Hodler explores how to stop throwing away valuable predictive data and start reclaiming it with the power of knowledge graphs. This is a must-watch for anyone interested in data-centric AI and the future of predictive analytics. Watch now: https://
buff.ly/3vRB2Mk -
Finetuned Llama3 Model with AutoTrain Zero Code
By
–
Finetuned llama3 model using AutoTrain: https://
huggingface.co/abhishek/autot
rain-llama3-no-robots
… Amount of code written to finetune: 0 lines -
Scaling Laws Evolution: 8B Model Trained on Fifteen Trillion Tokens
By
–
you're telling me an 8B param model was trained on fifteen trillion tokens? i didn't even know there was that much text in the world really interesting to see how scaling laws have changed best practices; GPT-3 was 175 billion params and trained on a paltry 300 billion tokens
-
Open-Source GPT-4-Level Models Now Freely Accessible Globally
By
–
We're entering a new world where GPT-4-level models are open-source and freely accessible. Absolutely massive.
-
Base Knowledge Requirements for AI Summarization Tasks
By
–
A question I've had for a while is how much base knowledge a model needs in order to be good at summarization etc Does it need to know that an elephant is larger try an a mouse (or what an elephant and a mouse are) in order to summarize a meeting agenda?
-
Alteryx Wins G2 Spring 2024 Data Science ML Platform Index
By
–
The @G2dotcom Spring 2024 Data Science and Machine Learning Platforms Results Index is here! Alteryx earned top spot with the top platform. Explore the factors behind our win and your success: https://
ow.ly/hzjH50RgFZK #DataScience #MachineLearning -
Memory Optimization for Multi-GPU Distributed AI Training
By
–
no it’s very small since we’re looping over each weight matrix individually the only way memory usage would be non-negligible would be if u used a large number of devices — say 256 GPUs in that case don’t use my code 🙂