So the first critical "unlock technology" is the Transformer, a neural net architecture powerful enough to become a general-purpose computer. I've written more about this here: 1) https://
x.com/karpathy/statu
s/1582807367988654081
… and 2)
AI
-
Transformer: The Critical Unlock Technology for General-Purpose AI
By
–
-
Transformers’ In-Context Learning: Emergent Ability at Scale
By
–
The non-obvious crux of the shift is an empirical finding, emergent only at scale, and well-articulated in the GPT-3 paper (
https://
arxiv.org/abs/2005.14165). Basically, Transformers demonstrate the ability of "in-context" learning. At run-time, in the activations. No weight updates. -
Neural Language Models: 20 Years of Autoregressive Architecture Evolution
By
–
E.g. ~20 years ago Bengio et al 2003 (pdf: https://
jmlr.org/papers/volume3
/bengio03a/bengio03a.pdf
…) trained a neural language model. The state of the art GPT+friends of today are the exact same (autoregressive) model, except the neural net architecture is upgraded from an MLP to a Transformer. -
Neural Language Models: From Overlooked Niche to AI Breakthrough
By
–
An interesting historical note is that neural language models have actually been around for a very long time but noone really cared anywhere near today's extent. LMs were thought of as specific applications, not as mainline research unlocking new general AI paths and capabilities
-
Perfect commits and feature building best practices guide
By
–
I talk about that a fair bit in these posts:
– https://
simonwillison.net/2022/Oct/29/th
e-perfect-commit/
…
– https://
simonwillison.net/2022/Jan/12/ho
w-i-build-a-feature/
…
– https://
simonwillison.net/2018/Jul/28/do
cumentation-unit-tests/
… -
Diffusion Models Collection: Redshift, Van Gogh and More
By
–
Diffusion has been generating tons of noise! Find Redshift Diffusion from @Nitrosocke
, Van Gogh Diffusion, and more on Replicate: https://
replicate.com/collections/di
ffusion-models
… -
Earth as Evolution’s Data Center with Biological Processing Units
By
–
The Earth is evolution's data center, and it has trillions of BPUs (biological processing units).
-
RAPIDS Platform Accelerates Data Science and AI Efficiency
By
–
Would you like to increase efficiency, streamline development, & achieve more cost-effective data operations in your #AI solution? Join 12/8 and learn how the free @RAPIDSai platform for accelerated data science can help. https://
nvda.ws/3tXGYip #NVIDIAInception #Startups -
Tesla’s Superior Software-Intensive Car Manufacturing Capabilities
By
–
Tesla knows how to make software-intensive cars better than anyone in the world, for one.