Prompt: "You are a GPT and you're in charge of training an even better GPT, congrats! You have a dataset here . You can train it on document chunks like this: and sample its current understanding like this: . And here's a calculator and a scratchpad . Begin:"
@karpathy
-
Meta-learning policies for LLM attention management during training
By
–
Feels like a lot of fertile ground is left in managing the "attention" of an LLM during its training via a meta-learning policy, instead of the typical "memorize dataset uniformly at random" strategy. And giving it a calculator and a scratch pad.
-
Training Strategies: Skimming, Filtering Noise, and Revisiting Content
By
–
More generally a few remarkable strategies people use during their training:
1) skim text because they already know it
2) ignore text because it's clearly noise (e.g. they won't memorize SHA256 hashes. LLMs will.)
3) revisit parts that are learnable but not yet learned -
Examples vs. Presentations: Spaced Repetition in LLM Training
By
–
Is it the number of examples that matters or the number of presentations to the model during training? E.g. humans used spaced repetition to memorize facts but there are no equivalents of similar techniques in LLMs where the typical training regime is uniform random.
-
ML Insights on Overfitting Apply Beyond Machine Learning Systems
By
–
Excellent post about applying insights from ML (overfitting control) to a much broader class of systems that optimize against an objective: politics, science, orgs, daily life. Underfitting is underrated.
-
MLPerf benchmark requires implementation of mitigations
By
–
MLPerf benchmark needs some of these mitigations https://
x.com/jaschasd/statu
s/1589424193946648576?s=46&t=Yrwu6ciEa83A6HcKopDkNw
… -
AI Pub Aspires to Akhaliq’s Level of Twitter Usefulness
By
–
AI Pub reaching for that @_akhaliq level of usefulness on AI twitter 🙂
-
Stable Diffusion’s Reasonable Interpretation of My Image
By
–
Base stable diffusion has a decent guess about me
-
Using StableBoost to refine image generation through visual selection
By
–
e.g. I used stableboost for this earlier tweet 🙂 – the prompt by itself gives bad, too diverse, not amazing results, but once I generated ~1000 I could visually narrow in on the composition I liked. Not sure how I'd get that by tuning the prompt alone
-
Iterative approach to building and refining positive datasets
By
–
from my own experience you want something interactive and change your mind around quite a bit. so you're building the positive set, seeing the results, then tweaking your positive set over time. it's an incremental iterative thing.