Extending LLMs from text to vision will probably take time but, interestingly, can be made incremental. E.g. Flamingo (
https://
storage.googleapis.com/deepmind-media
/DeepMind.com/Blog/tackling-multiple-tasks-with-a-single-visual-language-model/flamingo.pdf
… (pdf)) processes both modalities simultaneously in one LLM.
@karpathy
-
Extending LLMs to Vision: Incremental Multimodal Integration with Flamingo
By
–
-
Why LLMs Process Text Instead of Raw Pixels
By
–
Interestingly the native and most general medium of existing infrastructure wrt I/O are screens and keyboard/mouse/touch. But pixels are computationally intractable atm, relatively speaking. So it's faster to adapt (textify/compress) the most useful ones so LLMs can act over them
-
LLMs as Cognitive Engines Orchestrating Compute Infrastructure via Text
By
–
Good post. A lot of interest atm in wiring up LLMs to a wider compute infrastructure via text I/O (e.g. calculator, python interpreter, google search, scratchpads, databases, …). The LLM becomes the "cognitive engine" orchestrating resources, its thought stack trace in raw text
-
Iterative Denoising: Simple Code, Core Idea
By
–
like! tiny idea tiny code, strips away the formalism except the high level idea (iterative denoising on a schedule)
-
Concerns about GPT alignment with human values and safety
By
–
"Finally, we are very concerned that this GPT could be unaligned with humans. This would be bad. We want this to be a nice GPT that deeply loves all humans and is always considerate and helpful. Thanks"
-
Data Quality and Curriculum Learning for LLM Training
By
–
"Obviously anything that looks useless (like SHA hashes or other noise) is not worth training on and is just wasting training capacity and time"
"You may want to start with simpler topics and work up to more complex later, just like in human school" -
GPT Training Framework with Dataset and Sampling Tools
By
–
Prompt: "You are a GPT and you're in charge of training an even better GPT, congrats! You have a dataset here . You can train it on document chunks like this: and sample its current understanding like this: . And here's a calculator and a scratchpad . Begin:"
-
Meta-learning policies for LLM attention management during training
By
–
Feels like a lot of fertile ground is left in managing the "attention" of an LLM during its training via a meta-learning policy, instead of the typical "memorize dataset uniformly at random" strategy. And giving it a calculator and a scratch pad.
-
Training Strategies: Skimming, Filtering Noise, and Revisiting Content
By
–
More generally a few remarkable strategies people use during their training:
1) skim text because they already know it
2) ignore text because it's clearly noise (e.g. they won't memorize SHA256 hashes. LLMs will.)
3) revisit parts that are learnable but not yet learned -
Examples vs. Presentations: Spaced Repetition in LLM Training
By
–
Is it the number of examples that matters or the number of presentations to the model during training? E.g. humans used spaced repetition to memorize facts but there are no equivalents of similar techniques in LLMs where the typical training regime is uniform random.