(More likely though, each block refines the information over time in the Transformer forward pass, enriching it with the information gathered from previous tokens during Attention.)
@karpathy
-
Residual Stream Projections and Information Preservation in Neural Networks
By
–
Of course it has access, the projections from each block into the residual stream can be learned to be zero and so preserve any information that is needed.
-
Modern AI Capabilities Compared to Early Computing Hardware Specs
By
–
Difficult to not feel like it is the equivalent of something along the lines of "CPU with a clock rate of more than 10^7 Hz and 20MiB RAM".
-
Toaster storytelling vision for morning routines
By
–
We cannot rest until the toaster tells a tiny story while waiting for the bread in the morning 😀
-
Clarifying Graph Vector Communication: Misleading vs Wrong Description
By
–
Ok I wouldn't say it was "wrong", just "misleading" :). The idea of having a vector stored at each node in the graph, and the vectors communicating via weighted sums over directed edges I think is all well and sound all by itself, at this level of description. It's misleading
-
Brain Computing Power Compared to NVIDIA A100 GPU
By
–
The naive napkin math would go something like 1 brain ~= 1e11 neurons * 1e4 synapses * 1e1 fires/s = 1e16 FLOPS (i.e. 10 petaflops) NVIDIA A100 = 312e12 peak FLOPS, in-practice achievable utilization may be let’s say 50%, i.e. 156e12. Dividing you get 1 brain ~= 1e16 / 156e12 =
-
LLMs’ Killer App is Scarlett Johansson, Not Math
By
–
The killer app of LLMs is Scarlett Johansson. You all thought it was math or something
-
OpenAI releases multimodal model with text audio vision capabilities
By
–
They are releasing a combined text-audio-vision model that processes all three modalities in one single neural network, which can then do real-time voice translation as a special case afterthought, if you ask it to.
— Andrej Karpathy (@karpathy) 13 mai 2024
(fixed it for you) https://t.co/0y36OId88hThey are releasing a combined text-audio-vision model that processes all three modalities in one single neural network, which can then do real-time voice translation as a special case afterthought, if you ask it to. (fixed it for you)
-
Leading AI Safety in Major Paradigm-Shifting Deployments Today
By
–
Just one of the very few people both in charge of and in thick of the practical AI safety of today in the biggest, paradigm shifting deployments of AI today…
-
Learning CUDA: Resources Gap Between Theory and Practice
By
–
I read this book and then I was surprised that I still understood so little of the kernels that started to appear as llm.c contributions, beating mine. It's a pretty good 101 intro. Learning CUDA is like that horse meme, all the learning resources you can find on the left, then