If previous neural nets are special-purpose computers designed for a specific task, GPT is a general-purpose computer, reconfigurable at run-time to run natural language programs. Programs are given in prompts (a kind of inception). GPT runs the program by completing the document
LLMS
-
Transformer: The Critical Unlock Technology for General-Purpose AI
By
–
So the first critical "unlock technology" is the Transformer, a neural net architecture powerful enough to become a general-purpose computer. I've written more about this here: 1) https://
x.com/karpathy/statu
s/1582807367988654081
… and 2) -
Transformers’ In-Context Learning: Emergent Ability at Scale
By
–
The non-obvious crux of the shift is an empirical finding, emergent only at scale, and well-articulated in the GPT-3 paper (
https://
arxiv.org/abs/2005.14165). Basically, Transformers demonstrate the ability of "in-context" learning. At run-time, in the activations. No weight updates. -
Neural Language Models: 20 Years of Autoregressive Architecture Evolution
By
–
E.g. ~20 years ago Bengio et al 2003 (pdf: https://
jmlr.org/papers/volume3
/bengio03a/bengio03a.pdf
…) trained a neural language model. The state of the art GPT+friends of today are the exact same (autoregressive) model, except the neural net architecture is upgraded from an MLP to a Transformer. -
Neural Language Models: From Overlooked Niche to AI Breakthrough
By
–
An interesting historical note is that neural language models have actually been around for a very long time but noone really cared anywhere near today's extent. LMs were thought of as specific applications, not as mainline research unlocking new general AI paths and capabilities
-
Argonne’s GPT Model Wins Gordon Bell Prize for COVID Research
By
–
In partnership with @Argonne
, we’ve been awarded the 2022 Gordon Bell Special Prize for HPC Based COVID-19 Research! We trained ANL’s GPT-style models on full Covid-19 genome, holding the entire sequence length (10240 tokens) on a single device. Read more: https://
hubs.li/Q01sCD600 -
GALA Model Code Released on GitHub for Researchers
By
–
The code is available. https://
github.com/paperswithcode
/galai
… So, we can try, but the general public can't. -
LangChain 0.0.15 Release: Dark Mode and SQL Improvements
By
–
LangChain version 0.0.15 – "the @nlarusstone release" improve color highlighting so it looks good in dark mode @nlarusstone (see below) add tables to ignore/include in SQL DB chain @nlarusstone add concept of document metadata add `apply` method to all chains
-
Groq Showcases Compiler, Developer Tools, and RealScale at Supercomputing
By
–
At @Supercomputing
? Drop by booth 3047 to:
– Discuss your model! Did you know 500+ models run on our #compiler?
– See dev tools like GroqFlow & GroqAPI, made for fine-grained control
– Discuss RealScale™, the technology that extends performance and #lowlatency from #chip to rack -
Galactica Model Repetition Issues Decoding Parameters Analysis
By
–
I’m a bit wondering why we see so much repetitions in the outputs of Galactica people share. If it comes from the model itself or the demo decoding parameters