"Finally, we are very concerned that this GPT could be unaligned with humans. This would be bad. We want this to be a nice GPT that deeply loves all humans and is always considerate and helpful. Thanks"
LLMS
-
GPT Training Framework with Dataset and Sampling Tools
By
–
Prompt: "You are a GPT and you're in charge of training an even better GPT, congrats! You have a dataset here . You can train it on document chunks like this: and sample its current understanding like this: . And here's a calculator and a scratchpad . Begin:"
-
Meta-learning policies for LLM attention management during training
By
–
Feels like a lot of fertile ground is left in managing the "attention" of an LLM during its training via a meta-learning policy, instead of the typical "memorize dataset uniformly at random" strategy. And giving it a calculator and a scratch pad.
-
Training Strategies: Skimming, Filtering Noise, and Revisiting Content
By
–
More generally a few remarkable strategies people use during their training:
1) skim text because they already know it
2) ignore text because it's clearly noise (e.g. they won't memorize SHA256 hashes. LLMs will.)
3) revisit parts that are learnable but not yet learned -
Examples vs. Presentations: Spaced Repetition in LLM Training
By
–
Is it the number of examples that matters or the number of presentations to the model during training? E.g. humans used spaced repetition to memorize facts but there are no equivalents of similar techniques in LLMs where the typical training regime is uniform random.
-
MIT Scientists Develop Liquid AI Model From Brain Dynamics Equation
By
–
MIT scientists solved an equation in brain dynamics and used that to design a new liquid AI model – a potential new building block of future intelligent systems: https://
bit.ly/3EtPf2r Code repository: https://
bit.ly/3VfC63N -
Contrastive Language-Audio Pretraining with Feature Fusion
By
–
Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation Wu et al.: https://
arxiv.org/abs/2211.06687 #Artificialintelligence #DeepLearning #MachineLearning -
OpenAI’s Trajectory Toward Advanced Content Creation Models
By
–
Great writers are prolific readers. GPT-3 read everything on the internet. Arguably, @openai is *on the path* to create the best content creation machine possible.
-
Evaluation Metrics for Language Modeling Explained
By
–
Evaluation Metrics for Language Modeling https://
thegradient.pub/understanding-
evaluation-metrics-for-language-models/
…