yeah 😀 it's not as bad as it sounds, e.g. using the example in the notebook, training an SVM on ~10K 1536D embeddings is ~1 second. Sometimes it's possible to precompute. And sometimes it's just not worth it, all depends on setting / application.
@karpathy
-
Exemplar SVM: Simple but Counterintuitive in High Dimensions
By
–
Yep exactly! 🙂 The first time I saw the Exemplar SVM idea. It's so simple but also a bit counter-intuitive, I think because low dimensional intuition fails us. A classifier with a single example? In low dimensions it sounds weird. In high dimensions it works great.
-
SVMs Outperform k-Nearest Neighbor for Embedding Lookups
By
–
Random note on k-Nearest Neighbor lookups on embeddings: in my experience much better results can be obtained by training SVMs instead. Not too widely known. Short example: https://
github.com/karpathy/rando
mfun/blob/master/knn_vs_svm.ipynb
… Works because SVM ranking considers the unique aspects of your query w.r.t. data. -
AutoGPT Agents Create Emergent Open Worlds for Indie Games
By
–
Love it – much fertile soil for indie games populated with AutoGPTs, puts "Open World" to shame. Simulates a society with agents, emergent social dynamics.
Paper: https://
arxiv.org/abs/2304.03442
Demo: https://
reverie.herokuapp.com/arXiv_Demo/#
Authors: @joon_s_pk @msbernst @percyliang @merrierm et al. -
Baby GPT as Finite State Markov Chain Visualization
By
–
This is a baby GPT with two tokens 0/1 and context length of 3, viewing it as a finite state markov chain. It was trained on the sequence "111101111011110" for 50 iterations. The parameters and the architecture of the Transformer modifies the probabilities on the arrows. E.g. we
-
Temperature Zero for Deterministic LLM Inference with Argmax Selection
By
–
trivially – just use temperature = 0 at inference, picking argmax token at each step. that they are necessarily stochastic is a common misconception.
-
GPT as Programmable Text Computer: Memory Analysis Analogy
By
–
The analogy between GPTs of today to the CPUs of early days of computing are interesting. GPT is a funny kind of programmable text computer. Have to think through it more but e.g.: ## Memory
GPT-4 RAM is ~log2(50K vocab size)*(32K context length)/(8 bits/byte) ~= 64kB, -
Language Models Training: Autoregressive vs Diffusion Approaches
By
–
Common Q: Can you train language model w diffusion?
Favorite A: read this post (the whole blog is excellent) (Roughly speaking state of the art generative AI is either trained autoregressively or with diffusion. The underlying neural net usually a Transformer.) -
ChatGPT Overexplaining Tweets Like Breaking Down Jokes
By
–
bleh ChatGPT doing the equivalent of explaining jokes but for my tweets 😀
-
GPT-4 Rewrites Rick Astley’s Never Gonna Give You Up
By
–
Bot will never give you up Bot will never let you down Bot will never run around and desert you Bot will never make you cry Bot will never say goodbye Bot will never tell a lie and hurt you
(lyrics re-written by gpt-4 ty)