day 20+ of exploring the wilderness: i found ul2 pretraining code in megatron LM. https://
github.com/NVIDIA/Megatro
n-LM/pull/268
… a nice surprise.
LLMS
-
UL2 Pretraining Code Found in Megatron-LM Repository
By
–
-
Kaggle Grand Master Shares Fine-Tuning Techniques for Large Language Models
By
–
Our very own #KaggleGrandMaster Pascal Pfeiffer (@kagglingpascal) shares the fine art of fine-tuning of #LargeLanguageModels. #H2OWorldIndia #LLM #LLMStudio
-
Crunchbase LLM Hallucination Issues and Data Accuracy
By
–
Maybe crunchbase's LLM hallucinated that info…
-
Google AI Releases Improved MC4 Corpus and uMT5 Models
By
–
Sharing a piece of work I contributed to while at @GoogleAI
: * a new improved Mc4 corpus (29T char tokens and 107 languages) that gets language sampling right with UniMax sampling. * open source pretrained uMT5 models trained on 1T tokens. * Unimax sampling solves some -
AI Model Limitations with BPD Topic Mentions
By
–
Ah, so if you explicitly mention BPD it says something like “as an AI model, I can’t answer that”?
-
MLflow 2.3 Releases New LLM Management and Deployment Features
By
–
Introducing #MLflow 2.3! The latest release makes it easier than ever to manage & deploy #LLMs (including Dolly 2.0!) and is packed with innovation including 3 new model flavors: HuggingFace Transformers, OpenAI functions & LangChain. See what’s new! https://
bit.ly/40eO4fP -
Scaling Laws Enable Predictable Model Performance
By
–
I don't think so? "Scaling laws" states that perplexity is highly predictable. For example, GPT-4's loss on some evaluations can be predicted with models of less than 1,000x compute. (
https://
arxiv.org/pdf/2303.08774
.pdf
…)
Should be a clear difference between things that can be predicted with -
Unpredictable Performance Spikes in Scaling Neural Networks
By
–
I think this is a reasonable point in theory. In practice, we don't have those model sizes. But assuming that we did, I think there's still something interesting going on with emergence, e.g., "We can't predict a performance spike at say 10B with models from 5B, 4.9B, 4.8B…"
-
Discussion on Prompt Engineering, RLHF, AI Safety, and AGI
By
–
Got to chat about prompt engineering, RLHF, LLM red teaming, AI safety, and AGI with @labenz on the @CogRev_Podcast — thanks so much for having me, Nathan!
-
Iterative LLM Programming with Generated Knowledge Approach
By
–
Coding iteratively starting with "generated knowledge" — a similar approach may be used to have the LLM write a book. An example of LLM prompting for programming https://
bit.ly/41H00bD