LLMS

Data Quality and Curriculum Learning for LLM Training

By

–

16 November 2022 4h35

"Obviously anything that looks useless (like SHA hashes or other noise) is not worth training on and is just wasting training capacity and time"
"You may want to start with simpler topics and work up to more complex later, just like in human school"

Concerns about GPT alignment with human values and safety

By

–

16 November 2022 4h35

"Finally, we are very concerned that this GPT could be unaligned with humans. This would be bad. We want this to be a nice GPT that deeply loves all humans and is always considerate and helpful. Thanks"

GPT Training Framework with Dataset and Sampling Tools

By

–

16 November 2022 4h21

Prompt: "You are a GPT and you're in charge of training an even better GPT, congrats! You have a dataset here . You can train it on document chunks like this: and sample its current understanding like this: . And here's a calculator and a scratchpad . Begin:"

Meta-learning policies for LLM attention management during training

By

–

16 November 2022 4h05

Feels like a lot of fertile ground is left in managing the "attention" of an LLM during its training via a meta-learning policy, instead of the typical "memorize dataset uniformly at random" strategy. And giving it a calculator and a scratch pad.

Training Strategies: Skimming, Filtering Noise, and Revisiting Content

By

–

16 November 2022 4h05

More generally a few remarkable strategies people use during their training:
1) skim text because they already know it
2) ignore text because it's clearly noise (e.g. they won't memorize SHA256 hashes. LLMs will.)
3) revisit parts that are learnable but not yet learned

Examples vs. Presentations: Spaced Repetition in LLM Training

By

–

16 November 2022 4h05

Is it the number of examples that matters or the number of presentations to the model during training? E.g. humans used spaced repetition to memorize facts but there are no equivalents of similar techniques in LLMs where the typical training regime is uniform random.