The thread above was, of course, written in collaboration with one of our powerful Large Language Models. Long live Chain Rules!
RESEARCH
-
Token Probability in Language Models Explained
By
–
By smaller events here, we refer to the probability of a token, given past tokens, p(c|ab). In probabilistic language modeling, a “token” is a single unit of text, like a word or part of a word. Modern language models consider a vocabulary size of ~100K tokens.
-
Chain Rule Reduces Token Generation Complexity Exponentially
By
–
To generate a sequence of 1000 tokens requires an insane 100K^1000 = 10^5000 choices. That’s a lot more than the estimated number of atoms in the universe, 10^82! With the chain rule the number of possible choices is "only" 100K * 1000 = 100M, a much more manageable number.
-
Chain Rules: Probability and Calculus in Machine Learning
By
–
Chain Rule(s) rules! Appreciation thread of one of the most interesting coincidences in machine learning. Two rules, both named "Chain Rule", happen to be absolutely critical to recent advances in ML & AI. A on the Chain Rule of Probability & the Chain Rule of Calculus
-
Chain Rule of Probability Powers Large Language Models
By
–
The Chain Rule of Probability is a powerful tool behind recent advances in Large Language Models. By multiplying together the probabilities of many smaller events, we can compute the probability of a complex event made up of those smaller events.
p(abc) = p(c|ab) * p(b|a) * p(a) -
AI Progress Accelerates: Balancing Innovation with Responsible Deployment
By
–
Most amazing fact about AI is that even though it’s starting to feel impressive, a year from now we’ll look back fondly on the AI that exists today as quaint & antiquated. Equal cause for excitement and deliberative caution — important to get the tech and its deployment right.
-
Ghost Cites and ChatGPT: Academic Citation Crisis Worsens
By
–
This is a classic story about an article that was never written yet became widely cited. "Ghost cites" were always problem in academia. But it's about to get much, much worse. This is how Chat GPT summarizes this famously non-existent article:
-
Current AI Limitations: Human Performance Remains the Benchmark
By
–
So does humans, it’s the best that AI can do as of now!!
-
Andrej Karpathy: Learning AI Concepts from Scratch
By
–
4. Andrej Karparthy: There are a lot of innovations happening in the field of AI. But what if you can understand the concepts behind that from Scratch? Andrej helps you with that. https://
youtube.com/@AndrejKarpathy -
3Blue1Brown: Essential ML Channel with Great Animations
By
–
3. 3 Blue 1 Brown: If you are into ML, you should have known this Channel. But if you are not go check this out. Top-level content with easy-to-understand Animations. https://
youtube.com/channel/UCYO_j
ab_esuFRV4b17AJtAw
…