5/ While the experiments done here were on small scale models (100M parameters), the fundamental effects we’re seeing here are likely to show up over time on the larger models too. For example, most of today’s models can’t produce a blog post in the style of Slate Star Codex,
RESEARCH
-
Synthetic Data Training Risks Model Collapse Long Term
By
–
3/ This core idea is very important to pay attention to: Synthetic data can create a short-term boost in eval results, but you will pay for it later with model collapse! You accumulate debt with mangling the model that starts invisible, and is very hard to repay.
-
Synthetic Data Training Limitations and Mode Collapse in Self-Distillation
By
–
Training on pure synthetic data has no information gain, thus there is little reason the model *should* improve. Oftentimes when evals go up from “self-distillation”, that might be from some more invisible tradeoff, i.e. mode collapse in exchange for individual eval improvement
-
Model Collapse in AI: Recursive Synthetic Data Training Risks
By
–
1/ New paper in Nature shows model collapse as successive model generations models are recursively trained on synthetic data. This is an important result. While many researchers today view synthetic data as AI philosopher’s stone, there is no free lunch. Read more
-
Review of The Atomic Human: What Remains if Machines Take Over
By
–
Review of the Atomic Human by @bryanappleyard (thanks for sticking with the digressions Bryan!). What will remain of us if the machines take over?
-
MMLU Pro benchmark evaluation data analysis
By
–
Agreed, MMLU Pro is not perfect either but we have enough data points to make a plot now
-
PaliGemma Fine-tuning with AutoTrain Guide
By
–
blog: https://
huggingface.co/blog/abhishek/
paligemma-finetuning-autotrain
… github repo: -
AutoTrain Enables VLM Finetuning with PaliGemma Support
By
–
NEW TASK ALERT: VLM Finetuning AutoTrain just added VLM finetuning: Captioning and VQA for PaliGemma. Now, its super-easy to finetune PaliGemma on your own custom dataset. Which model and tasks would you like to see next? Let us know in Github Repo!
-
AI Video Generation Breakthroughs: Novel Algorithms Enable Impossible Advances
By
–
DearVCs, CEOs, VPs & Governments: Many people, including some AI researchers, thought this video generation and other AI advances would be impossible. Some researchers and engineers persevered with optimism, thought hard, derived novel mathematical algorithms and tractable… https://t.co/uGx7hkj3hR
— Nando de Freitas (@NandoDF) 25 juillet 2024DearVCs, CEOs, VPs & Governments: Many people, including some AI researchers, thought this video generation and other AI advances would be impossible. Some researchers and engineers persevered with optimism, thought hard, derived novel mathematical algorithms and tractable