eval loss. i don't always use it, as it's not a great measure, but it's good for things like this
MACHINE LEARNING
-
ICLR Features Hugging Face Demo Platform for AI Models
By
–
ICLR blog: https://
blog.iclr.cc/2024/04/22/hug
ging-face-demo-site/
… -
Lean Copilot Automates Over 80% of Mathematical Proof Steps
By
–
We have now posted an expanded version of our Lean Co-pilot paper and updated the code base. https://
arxiv.org/abs/2404.12534 https://
github.com/lean-dojo/Lean
Copilot
…
The credit goes to @p_song1 for driving this effort.
Our latest experiments show that our co-pilot can automate >80% of the proof steps -
AI Model Learning from Inference Failures and Preferences
By
–
A few bits per pair, sure, but so nutritious — lessons not just from humans but from its own inference failures, categorically missing from pre-training. Preferences are some of the only bits a model gets on how to succeed at being itself, with its specific quirks and deficits.
-
Character-Level GPT Models Launch ML Training Journey
By
–
Dude…. big fan. Your character-level GPT models were how I initially got into training models
-
Explainable AI: Challenges Across Telecommunications and Research
By
–
Looking to gain a deeper understanding of Explainable AI? Experts from Eckerson Group, Orange and Georgia Tech dive into the challenges of AI explainability from telecommunications networks to academic research. https://
buff.ly/3vNcwMr -
RoPE Scaling Enables 16K Token Context Window Models
By
–
Yeah — simply multiplied rope * 2, trained on long data with a MSL of 16K.
-
Llama 3 Announcement Highlights Data Quality’s Impact on AI Performance
By
–
Llama 3 announcement says the quality of both SFT and preference data have an “outsize influence” on performance. The first excerpt above alludes to an explanation for why preferences would help with reasoning. (This screenshot is the text just before the one above:)
-
Neural Operators and Physical Grounding in AI Models
By
–
Thank you @shiringhaffary for covering my @TEDTalks in @business and neural operators for building physical understanding into #AI models. Language and multi-modal models lack physical grounding. models like Sora and some of the robotics models may get some of the physics
-
Mistral Mixtral 8x7B Mixture of Experts Architecture Course
By
–
New short course with @MistralAI !
— Andrew Ng (@AndrewYNg) 22 avril 2024
Mistral's open-source Mixtral 8x7B model uses a "mixture of experts" (MoE) architecture. Unlike a standard transformer, an MoE model has multiple expert feed-forward networks (8 in this case), with a gating network selecting two experts at… pic.twitter.com/VFOg1dDab8New short course with @MistralAI ! Mistral's open-source Mixtral 8x7B model uses a "mixture of experts" (MoE) architecture. Unlike a standard transformer, an MoE model has multiple expert feed-forward networks (8 in this case), with a gating network selecting two experts at