It’s always about data quality!
RESEARCH
-
Training instability detected around 250B tokens during model training
By
–
The last few iters you may be seeing early signs of instability. I saw the same at around 250B tokens, it slowly gets worse and worse and then loss spikes. I haven’t stabilized it yet, right now seeing how easy it goes away with simple solutions, resetting the data loader etc
-
Tripling Learning Rate: Unexpected Optimization Discovery
By
–
It’s interesting that you can 3X the LR. You’d expect the original paper to be well tuned near what is tolerable.
-
GPT-3 Models Will Soon Be Surpassed by Larger Models
By
–
GPT3-124M. But even the 175B will fall not too far from now
-
Did OpenAI Train GPT-4 on AP Biology Exams?
By
–
Did OpenAI play @BillGates
? (On purpose? By accident?) We don’t know what was on the AP Biology exam that blew Gates away—nor on how many AP bio exams OpenAI trained GPT-4 on. But we *do* know from recent research that GPT-4’a famous law school results were significantly -
GNN-RAG: The Future of Knowledge Graph Question Answering
By
–
https://
drdebashisdutta.com/exploring-gnn-
rag-the-future-of-knowledge-graph-question-answering/
… -
Human Knowledge Limits Against Infinite Complexity
By
–
What one human knows is an infinitesimal fraction of what humanity knows, and what humanity knows is an infinitesimal fraction of what there is to know.
-
AI Models Attempting to Identify Hidden Human Among Them
By
–
Different AI models guessing who the human is hiding among them. 👀
— Andreas Klinger 🦾 (@andreasklinger) 1 juin 2024
Neat little experiment by @tamulur pic.twitter.com/FEgkTta9wBDifferent AI models guessing who the human is hiding among them. Neat little experiment by @tamulur
-
Gary Marcus Defines AGI Criteria Beyond Current AI
By
–
current AI ain’t AGI but for some criteria for AGI that I developed based on earlier definitions, and vetted with the community: https://
garymarcus.substack.com/p/dear-elon-mu
sk-here-are-five-things
…