The analogy between GPTs of today to the CPUs of early days of computing are interesting. GPT is a funny kind of programmable text computer. Have to think through it more but e.g.: ## Memory
GPT-4 RAM is ~log2(50K vocab size)*(32K context length)/(8 bits/byte) ~= 64kB,
@karpathy
-
GPT as Programmable Text Computer: Memory Analysis Analogy
By
–
-
Language Models Training: Autoregressive vs Diffusion Approaches
By
–
Common Q: Can you train language model w diffusion?
Favorite A: read this post (the whole blog is excellent) (Roughly speaking state of the art generative AI is either trained autoregressively or with diffusion. The underlying neural net usually a Transformer.) -
ChatGPT Overexplaining Tweets Like Breaking Down Jokes
By
–
bleh ChatGPT doing the equivalent of explaining jokes but for my tweets 😀
-
GPT-4 Rewrites Rick Astley’s Never Gonna Give You Up
By
–
Bot will never give you up Bot will never let you down Bot will never run around and desert you Bot will never make you cry Bot will never say goodbye Bot will never tell a lie and hurt you
(lyrics re-written by gpt-4 ty) -
Confidence in LLM Development and Future Perspectives
By
–
Not exactly, there are and will be others who fab LLMs. But I have increasing confidence in the last paragraph of this post https://
karpathy.github.io/2022/03/14/lec
un1989/
… -
Deep Learning Skills Shift: From Training to Prompt Engineering
By
–
Expectation: I need more deep learning engineers to train better models
Reality: You need prompt engineers and LLM Ops (not sure what to call it (?), post-LLM above-API infra, langchain & friends)
– training is centralizing into megamodels – not fully played out yet but trending -
State of the Art Image Generation Results from 5 Years Ago
By
–
Around 5 years ago we were very proud of these state of the art results in image generation, trained on 32×32 "images" of CIFAR-10. You can kind of make out little wheel shapes, car/plane parts, and organic structures and textures. Pretty cool right
-
Von Neumann’s Hypothetical Transformer Architecture Hyperparameters
By
–
I wonder if von Neumann had a large d_model, n_layer, head_size or block_size, or kv cache. All of these hyperparams might manifest slightly different.
-
AutoGPTs Will Organize into Specialized Autonomous Organizations
By
–
All of that is just one agent/thread. People coalesce into organizations so they can specialize and parallelize work towards shared goals. Imo this is likely to happen to AutoGPTs and for the same reasons, strung into AutoOrgs, with AutoCEO, AutoCFO, AutoICs, etc.