You can kind of do this already by a bit of prompting, but probably you're right that if you target this as a finetune it might come out better.
@karpathy
-
Estimating GPT Grades for Written Content Quality
By
–
Anyone else find themselves estimating the "GPT grade" of things you hear/read? When something is poorly written or generic, it's "GPT-2 grade" content. When something is lit, you can complement it as being "GPT-7 grade" etc. This reminds me of a fun side project I had saved for
-
Token Encoding and Decoding: Asymmetric Complexity in LLMs
By
–
decoding (tokens -> string) is just lookup table and string concat. encoding (string -> tokens) is a pain. For sentencepiece I *think* llama2.c has a simple implementation that probably works but I'm not 100% sure: https://
github.com/karpathy/llama
2.c/blob/master/run.c#L452
… For tiktoken-style, the problem is the -
Deep dive into tokenization vulnerabilities across multiple language models
By
–
Nice new read on tokenization!
You've heard about the SolidGoldMagikarp token, which breaks GPT-2 because it was present in the training set of the Tokenizer, but not the LLM later. This paper digs in in a lot more depth and detail, on a lot more models, discovering a less -
Implementation at Scale Matters More Than Brilliant Ideas
By
–
Also let's not forget the ability to actually implement your ideas in an efficient, at-scale manner, means you can demonstrate that they work on benchmarks people care about. You'll probably see a lot less interest if you can only prove your brilliant ideas on MNIST. Sometimes
-
Fighting Physics and NVIDIA Compiler Stack for Kernel Optimization
By
–
Great read! My experience is that you’re fighting physics but also the nvidia compiler and the stack overall, and even after pulling *a lot* of tricks we still can’t achieve more than ~80-90% mem bw on many kernels that you’d naively think should be ~100. And the rabbit hole
-
matRBM: Early Open Source Project for Restricted Boltzmann Machines
By
–
It was the worst. But it did have a few really awesome UI features, a built-in debugger, persistent memory, etc. This is one of my first open source projects ever: https://
code.google.com/archive/p/matr
bm/
…
matRBM, a library to train Restricted Boltzmann Machines in Matlab 🙂 Main training loop: -
Evolution of Programming Languages in Machine Learning Research Career
By
–
In roughly all of my experience (Geoff/Ruslan RBM work at UofT, Nando lab at UBC, Andrew Ng lab at Stanford, my 2011 Google internship in baby Google Brain, and ~all computer vision work I was familiar with) it was all only Matlab. I’ve never used Theano but I used Torch in
-
CUDA and C++ Origins Behind AlexNet’s Deep Learning Revolution
By
–
# CUDA/C++ origins of Deep Learning Fun fact many people might have heard about the ImageNet / AlexNet moment of 2012, and the deep learning revolution it started. https://
en.wikipedia.org/wiki/AlexNet What's maybe a bit less known is that the code backing this winning submission to the -
GPT-2 Training Code vs PyTorch’s Universal Capability
By
–
It's worth noting that this code specifically trains GPT-2.
PyTorch trains anything under the sun.