Also, it's not the amount of data on which GPT-3 is trained on its the number of parameters. Both are different things!
LLMS
-
Transformer Inference Optimization: Reducing Computational Costs
By
–
Large Transformers are powerful but expensive to train & use. The extremely high inference cost is a big bottleneck for adopting them for solving real-world tasks at scale. Check out my new post on some ideas on inference optimization for Transformers:
-
GPT-3.5 passes multiple-choice sections of Bar Exam for Evidence and Torts
By
–
GPT-3.5 took the Bar Exam "Passing range" scores on multiple-choice section for Evidence & Torts categories. Full write-up in Monday's newsletter Research paper: https://
arxiv.org/pdf/2212.14402
.pdf
… -
Comparing Modern GPT Models to OpenAI’s Early GPT-2 Capabilities
By
–
It’s about as good as OpenAI’s baby GPT-2 from ~4 years ago. (Their paper at that time had models from 124M to 1.3B). Today’s bleeding edge GPTs reach scale (in model size and data size) that requires significant infrastructure and further finetuning to align them (RLHF etc).
-
Evolution from search engines to answer engines and beyond
By
–
Tired: search engine
Wired: answer engine
Inspired: ???
🙂 -
LLMs Need External Tools Like Humans Need Interaction
By
–
LLMs are like a person doing everything just in their head. People wouldn’t get very far like that alone. LLMs wouldn’t either.
-
Clarification on GPT-2 Model Size: 124M vs 1.3B Parameters
By
–
Careful this is the 124M model. The biggest GPT-2 was 1.3B
-
OpenAI ChatGPT System Prompt Revealed After January 2023
By
–
OpenAI #ChatGPT's secret sauce is no more a secret. After Jan 9, 2023 update! Here is the source prompt of #ChatGPT
-
GPT-2 Pre-training: Hardware Requirements and Token Processing Estimates
By
–
Rough example, a decent GPT-2 (124M) pre-training reproduction would be 1 node of 8x A100 40GB for 32 hours, processing 8 GPU * 16 batch size * 1024 block size * 500K iters = ~65B tokens. I suspect this wall clock can still be improved ~2-3X+ without getting too exotic.
-
Future Plans for GPT-2 Implementation and Educational Content
By
–
I'd like to continue to make it faster, reproduce the other GPT-2 models, then scale up pre-training to bigger models/datasets, then improve the docs for finetuning (the practical use case). Also working on video lecture where I will build it from scratch, hoping out in ~2 weeks.