Do large language models reflect the nuances and complexities of public opinion? Stanford scholars introduce OpinionQA, a tool that compares language model outputs to public opinion polling. https://
stanford.io/3RwGm07
LLMS
-
Stanford tool compares LLM outputs to public opinion polling
By
–
-
ALiBi Enables Context Extrapolation Beyond Training Length
By
–
ALiBi allows our model trained on 8192 context length to extrapolate well up to 9216 tokens out of the box. We also show that ALiBi models only achieve near-perfect extrapolation when they are severely undertrained (<1 tokens/parameter).
-
BTLM-3B Outperforms Larger 7B Models in Long Context
By
–
The paper explores long context performance in detail. We found BTLM-3B-8K outperforms other 7B-8K models despite being trained with less than a fifth the pretraining compute and less than half the size.
-
BTLM: World’s Most Accurate 3B Parameter Open Source Model
By
–
BTLM is the world’s most accurate 3B parameter model, with performance that rivals many 7B models. It’s fully open source and has been downloaded >1M times.
-
LLM Training Techniques: 2.86x Compute Efficiency Gains
By
–
There are so many LLM training tricks but which ones should you use? We quantify the effects of techniques like SwiGLU, ALiBi, etc and show how using them in conjunction can achieve the same loss with 2.86x less pretraining compute or 1.74x less parameters.
-
BTLM-3B-8K: Distilling SOTA LLM Training Recipe
By
–
We just dropped the BTLM-3B-8K paper on arXiv! It distills our recipe for training SOTA LLMs:
– Extensively deduplicated dataset (SlimPajama)
– Hyperparameter search using muP
– Variable sequence length training + ALiBi
– Aggressive LR decay https://
arxiv.org/abs/2309.11568 -
OpenAI Launches Cookbook at New Home for Developers
By
–
Awesome news for @OpenAI devs: the cookbook has a new home, https://
cookbook.openai.com Thousands of folks show up everyday to use the cookbook, excited for this content to be more accessible. Big s/o to @simonpfish for driving this! -
LeCun and Ng argue against six-month AI moratorium
By
–
The gist of arguments @ylecun against the 6-month AI moratorium at @VentureBeat
.
With @AndrewYNg #GenAI #chatgpt #llm #gpt4 #promptengineering #GenerativeAI #LLAMA #ai #gpt4 #chatgpt #stats #statistics #DataScience #machinelearning #RStats #ML #dataviz #BigData #data -
Emergent Abilities in LLMs: Evidence of In-Context Learning
By
–
Are Emergent Abilities in Large Language Models just In-Context Learning? ***********
Spoiler: YES Through a series of over 1,000 experiments, we provide compelling evidence: http://
arxiv.org/abs/2309.01809 Our results allay safety concerns regarding latent hazardous -
LLMs democratize access to inspiration beyond mainstream categories
By
–
The power of proper inspiration is that it lets you stand on the shoulders of giants. Before LLMs, the giants you could find had to fit into—and be at the top of—pre-established mainstream categories. You could Google for top non-fiction writing, but you probably wouldn’t get