or prompts, e.g. in retrieval-augmented models. but only if you call your `.encode()` wrong 🙂
@karpathy
-
Andrej Karpathy announces Zero to Hero course project
By
–
working on it! https://
karpathy.ai/zero-to-hero.h
tml
… -
Data Augmentation: Trivial in Vision, Non-Trivial in NLP
By
–
yes, the difference is data augmentation, which is trivial in vision and non-trivial in NLP
-
Chinchilla’s implications: Dataset size over model size in LLMs
By
–
Great post (5mo ago) "chinchilla's wild implications" giving context to LLM goldrush shifting from model size to dataset size following Chinchilla https://
lesswrong.com/posts/6Fpvch8R
R29qLEWNH/chinchilla-s-wild-implications
…
Subtle important detail: analysis assumes 1 epoch. Recent work (e.g. Galactica) gives hope for 1+ regime. -
MAML Meta-Learning and Software Architecture Design Principles
By
–
reminds me of MAML meta-learning (
https://
arxiv.org/abs/1703.03400) where the objective is to find weights of a network such that any new task finetunes fast. In Software 1.0 land, equivalent is writing code such that any new desired functionality is simple and doesn't need a refactor. -
AI Superintelligence Through Scale and Collaboration Beyond Human Training
By
–
How superintelligent is an average intelligent human for whom time flows 1000X slower and gets to colaborate with 1000 copies? I was in convo yesterday doubting that AI can ever go beyond human when it is trained on human. Even if that were true (imo isn't) there's more+faster.
-
Mitigating Rare Pairings in Training Data Through Oversampling
By
–
(can be mitigated by e.g. oversampling the rare pairings during training or eventully solved with a data engine)
-
Neural Networks Exploiting Background Correlations in Price Predictions
By
–
Fun! "It appears that, even though the model predicts the same make/model for all of the images, the background can influence the predicted price by almost $10k!" Haha, neural nets are happy and eager to take advantage of all the easy correlations you allow them to latch on to 🙂
-
GitHub Copilot Dramatically Accelerates Coding Workflow
By
–
Nice read on reverse engineering of GitHub Copilot . Copilot has dramatically accelerated my coding, it's hard to imagine going back to "manual coding". Still learning to use it but it already writes ~80% of my code, ~80% accuracy. I don't even really code, I prompt. & edit.