what would be cool with science models like Galactica would be to add self examination of the inputs like the coming Codex model, where the model can inspect it’s theorem proofs or essays and spot errors
LLMS
-
Galactica: Open-Source Large Language Model for Science
By
–
Paper: https://
galactica.org/static/paper.p
df
…
Github: https://
github.com/paperswithcode
/galai
…
Demo: https://
galactica.org -
Galactica: 120B Scientific Language Model for Research
By
–
"Galactica" is a large scientific language model with 120B parameters, that can be used for a lot of tasks including
⦿ Predicting citations
⦿ Generating literature review
⦿ Generating molecules ⦿ Generating Jupiter notebooks etc., -
Galactica LLM Performance with XGBoost Model Implementation
By
–
Amazed by the performance of Galactica by @paperswithcode
— SRK (@sudalairajkumar) 16 novembre 2022
Tried with prompt "jupyter notebook on how to use xgboost model" @tunguz might like it 🙂 #LLM #NLP #AI pic.twitter.com/7ohrREfiltAmazed by the performance of Galactica by @paperswithcode Tried with prompt "jupyter notebook on how to use xgboost model" @tunguz might like it 🙂 #LLM #NLP #AI
-
Large Language Models Are Not Zero-Shot Communicators
By
–
Large language models are not zero-shot communicators Ruis et al.: https://
arxiv.org/abs/2210.14986 #Artificialintelligence #DeepLearning #MachineLearning -
Data Quality and Curriculum Learning for LLM Training
By
–
"Obviously anything that looks useless (like SHA hashes or other noise) is not worth training on and is just wasting training capacity and time"
"You may want to start with simpler topics and work up to more complex later, just like in human school" -
Concerns about GPT alignment with human values and safety
By
–
"Finally, we are very concerned that this GPT could be unaligned with humans. This would be bad. We want this to be a nice GPT that deeply loves all humans and is always considerate and helpful. Thanks"
-
GPT Training Framework with Dataset and Sampling Tools
By
–
Prompt: "You are a GPT and you're in charge of training an even better GPT, congrats! You have a dataset here . You can train it on document chunks like this: and sample its current understanding like this: . And here's a calculator and a scratchpad . Begin:"
-
Meta-learning policies for LLM attention management during training
By
–
Feels like a lot of fertile ground is left in managing the "attention" of an LLM during its training via a meta-learning policy, instead of the typical "memorize dataset uniformly at random" strategy. And giving it a calculator and a scratch pad.
-
Training Strategies: Skimming, Filtering Noise, and Revisiting Content
By
–
More generally a few remarkable strategies people use during their training:
1) skim text because they already know it
2) ignore text because it's clearly noise (e.g. they won't memorize SHA256 hashes. LLMs will.)
3) revisit parts that are learnable but not yet learned