Lack of generalization != overfitting. Lack of generalization could come from overfitting, underfitting, concept drift, OOD, or could come up for a model where there's no "fitting" at all. It just means that the model cannot operate in situations it hasn't been prepared for
RESEARCH
-
nanoGPT with MoEs and multimodal tokenization support
By
–
i would love for @karpathy to make a nanoGPT for MoEs with multimodal tokenization support
-
Limitations of Training AI Models on Synthetic and Video Data
By
–
But you can't just fit a big curve to a bunch of UE5 screencaps and YouTube videos and expect a generalizable model of the real world. That is not how these models work.
-
Large Curves: Weather Prediction and Simulator Applications
By
–
Could you use a big curve to predict the weather? Yes, given enormous amounts of data. We're already doing it! To make a wind chamber simulator? Yes, if you can keep as many parameters as possible fixed, and if you can densely sample training data for the parameters that do vary
-
Finding simpler better AI methods beyond GANs and RLHF
By
–
people spent years optimizing GANs before realizing that diffusion models were simpler and better people spent years developing RLHF before realizing that DPO is simpler and better what are we working on rn? i want to find the simpler and better version and work on that instead
-
Protein Design Breakthrough: AI Transforms Biology Three Years Post-AlphaFold2
By
–
3 years after the #AlphaFold2 moment, we've never been in a better position to design new proteins, antibodies, enzymes, and vaccines from scratch, thanks to transformer and diffusion #AI models
Dedicated new issue @NatureBiotech w/ primers & reviews https://
nature.com/nbt/volumes/42
/issues/2
… -
DoRA: Weight-Decomposed Low-Rank Adaptation Technique
By
–
DoRA: Weight-Decomposed Low-Rank Adaptation Liu et al.: https://
arxiv.org/abs/2402.09353 #ArtificialIntelligence #DeepLearning #MachineLearning -
Gemini-1.5 Breakthrough Overshadowed by Sora Release
By
–
I’m feeling bummed that Sora overshadowed Gemini-1.5 — the improvements there were ground breaking (10M context window that was effective, 1.5-Pro Beats 1.0-Ultra indicating a fundamental improvement). And available for access.
-
DoRA: Weight-Decomposed Low-Rank Adaptation for Models
By
–
DoRA: Weight-Decomposed Low-Rank Adaptation Liu et al.: https://
arxiv.org/abs/2402.09353 #ArtificialIntelligence #DeepLearning #MachineLearning -
Gemini 1.5 Context Window Revolutionizes RAG Capabilities
By
–
Gemini 1.5 is a much bigger deal than Sora. 1. The 10M context window fundamentally changes how we think about RAG. Not just large, but also effective. 2. Pro-1.5 Beats Ultra-1.0. There’s a fundamental modeling improvement. 3. It’s in production, you can use it today. I too