@oriolvinyalsml

–

13 February 2023 19h20

Language modeling is compression gone wrong.

13 February 2023

Backpropagation: Essential for Training Deep Neural Networks

By

–

12 February 2023 18h17

This is very important in Deep Learning: it allows calculating derivatives of highly complex neural networks, composed of stacked layers of simple functions. Gradient descent is used to update the neural net weights, allowing it to learn from data and perform complex tasks.

LLM Collaboration and Chain Rules Innovation

By

–

12 February 2023 18h17

The thread above was, of course, written in collaboration with one of our powerful Large Language Models. Long live Chain Rules!

Chain Rule Reduces Token Generation Complexity Exponentially

By

–

12 February 2023 18h17

To generate a sequence of 1000 tokens requires an insane 100K^1000 = 10^5000 choices. That’s a lot more than the estimated number of atoms in the universe, 10^82! With the chain rule the number of possible choices is "only" 100K * 1000 = 100M, a much more manageable number.

Token Probability in Language Models Explained

By

–

12 February 2023 18h17

By smaller events here, we refer to the probability of a token, given past tokens, p(c|ab). In probabilistic language modeling, a “token” is a single unit of text, like a word or part of a word. Modern language models consider a vocabulary size of ~100K tokens.

Chain Rules: Probability and Calculus in Machine Learning

By

–

12 February 2023 18h17

Chain Rule(s) rules! Appreciation thread of one of the most interesting coincidences in machine learning. Two rules, both named "Chain Rule", happen to be absolutely critical to recent advances in ML & AI. A on the Chain Rule of Probability & the Chain Rule of Calculus

Chain Rule of Probability Powers Large Language Models

By

–

12 February 2023 18h17

The Chain Rule of Probability is a powerful tool behind recent advances in Large Language Models. By multiplying together the probabilities of many smaller events, we can compute the probability of a complex event made up of those smaller events.
p(abc) = p(c|ab) * p(b|a) * p(a)