It’s about as good as OpenAI’s baby GPT-2 from ~4 years ago. (Their paper at that time had models from 124M to 1.3B). Today’s bleeding edge GPTs reach scale (in model size and data size) that requires significant infrastructure and further finetuning to align them (RLHF etc).
RESEARCH
-
Evolution from search engines to answer engines and beyond
By
–
Tired: search engine
Wired: answer engine
Inspired: ???
🙂 -
AI community shares diverse visions for the future
By
–
1/2 Thanks everyone that replied to this. Lots of great ideas here! The diversity of our hopes for AI is a sign of how large our field is — we collectively want AI to go lots of places.
-
Neural Networks: Zero to Hero Series – Building Networks from Scratch
By
–
(This will be part of my ongoing series Neural Networks: Zero to Hero https://
karpathy.ai/zero-to-hero.h
tml
… , on building neural networks, from scratch, in code. I have tweeted some of these videos individually already) -
GPT-2 Pre-training: Hardware Requirements and Token Processing Estimates
By
–
Rough example, a decent GPT-2 (124M) pre-training reproduction would be 1 node of 8x A100 40GB for 32 hours, processing 8 GPU * 16 batch size * 1024 block size * 500K iters = ~65B tokens. I suspect this wall clock can still be improved ~2-3X+ without getting too exotic.
-
Future Plans for GPT-2 Implementation and Educational Content
By
–
I'd like to continue to make it faster, reproduce the other GPT-2 models, then scale up pre-training to bigger models/datasets, then improve the docs for finetuning (the practical use case). Also working on video lecture where I will build it from scratch, hoping out in ~2 weeks.
-
nanoGPT: Simplest Repository for Training Medium-Sized GPTs
By
–
Didn't tweet nanoGPT yet (quietly getting it to good shape) but it's trending on HN so here it is 🙂 : https://
github.com/karpathy/nanoG
PT
…
Aspires to be simplest, fastest repo for training/finetuning medium-sized GPTs. So far confirmed it reproduced GPT-2 (124M). 2 simple files of ~300 lines -
Long-term work on arxiv-sanity project iterations
By
–
for sure! would love to know a bit more under the hood. I've working on this problem for a _long_ time, arxiv-sanity versions 1,2,3,4,5 and all 😀
-
Deep Learning and StarCraft 2 Type Checking Discussion
By
–
Drastic, deep learning, StarCraft 2 – type checks … lot of @ylecun too
-
Meta AI Presents CICERO: First Human-Level Diplomacy Game AI
By
–
Meta AI presents CICERO — the first AI to achieve human-level performance in Diplomacy, a strategy game which requires building trust, negotiating and cooperating with multiple players. Learn more about #CICERObyMetaAI: http://
bit.ly/3GBwLzx