And Google has Google AI, etc. My point is that it’s useful to have an “independent” AI lab in addition to the “corporate” one. (Feel free to disagree, of course. Jeff Dean is not a huge fan of DeepMind, either.)
RESEARCH
-
Transformer Inference Optimization: Reducing Computational Costs
By
–
Large Transformers are powerful but expensive to train & use. The extremely high inference cost is a big bottleneck for adopting them for solving real-world tasks at scale. Check out my new post on some ideas on inference optimization for Transformers:
-
GPT-3.5 passes multiple-choice sections of Bar Exam for Evidence and Torts
By
–
GPT-3.5 took the Bar Exam "Passing range" scores on multiple-choice section for Evidence & Torts categories. Full write-up in Monday's newsletter Research paper: https://
arxiv.org/pdf/2212.14402
.pdf
… -
Comparing Modern GPT Models to OpenAI’s Early GPT-2 Capabilities
By
–
It’s about as good as OpenAI’s baby GPT-2 from ~4 years ago. (Their paper at that time had models from 124M to 1.3B). Today’s bleeding edge GPTs reach scale (in model size and data size) that requires significant infrastructure and further finetuning to align them (RLHF etc).
-
Evolution from search engines to answer engines and beyond
By
–
Tired: search engine
Wired: answer engine
Inspired: ???
🙂 -
AI community shares diverse visions for the future
By
–
1/2 Thanks everyone that replied to this. Lots of great ideas here! The diversity of our hopes for AI is a sign of how large our field is — we collectively want AI to go lots of places.
-
Neural Networks: Zero to Hero Series – Building Networks from Scratch
By
–
(This will be part of my ongoing series Neural Networks: Zero to Hero https://
karpathy.ai/zero-to-hero.h
tml
… , on building neural networks, from scratch, in code. I have tweeted some of these videos individually already) -
GPT-2 Pre-training: Hardware Requirements and Token Processing Estimates
By
–
Rough example, a decent GPT-2 (124M) pre-training reproduction would be 1 node of 8x A100 40GB for 32 hours, processing 8 GPU * 16 batch size * 1024 block size * 500K iters = ~65B tokens. I suspect this wall clock can still be improved ~2-3X+ without getting too exotic.
-
Future Plans for GPT-2 Implementation and Educational Content
By
–
I'd like to continue to make it faster, reproduce the other GPT-2 models, then scale up pre-training to bigger models/datasets, then improve the docs for finetuning (the practical use case). Also working on video lecture where I will build it from scratch, hoping out in ~2 weeks.
-
nanoGPT: Simplest Repository for Training Medium-Sized GPTs
By
–
Didn't tweet nanoGPT yet (quietly getting it to good shape) but it's trending on HN so here it is 🙂 : https://
github.com/karpathy/nanoG
PT
…
Aspires to be simplest, fastest repo for training/finetuning medium-sized GPTs. So far confirmed it reproduced GPT-2 (124M). 2 simple files of ~300 lines