I guess there is a tradeoff between how each activation is used, whether it is doing something that's optimal for a current or future token. i'm using information and store both pretty loosely. 🙂
@jxmnop
-
Transformers Token Processing: Information Storage Nuances
By
–
yes! there is a lot of nuance. My preferred way to put it is "transformers don't pre-store information for future tokens at the expense of the current token" (very much)
-
Paper on Pre-caching and Thinking Ahead in Language Models
By
–
oh and here's a link to the paper. includes a lot of theorems, experiments on a synthetic problem to explicitly test pre-caching (one sort of "thinking ahead") and experiments on language.
-
Language Models Predict Next-Next Tokens From Hidden States
By
–
and confusingly, we can predict the next-next token from hidden states at a given timestep. it just turns out that this is seems to be mostly a property of language — for example if you say "New" then it's somewhat likely that "City" is two tokens away
-
Joint Mathematical Research in AI by Wu and Levine
By
–
this was joint work between myself and two mathematicians, which was really fun. and there's a lot of theory in the paper that I didn't touch on here. work was led by Wilson Wu, an incredibly talented math PhD at Boulder, and @lionellevine at Cornell. stay tuned for updates 🙂
-
Transformers Planning Ahead Through Hidden States and Caching
By
–
it's not that they *can't* plan ahead. as we note, transformers are actually encouraged to make information useful for future states, since the loss at a given position is computed using hidden states at previous positions and in a fake problem where they have to cache
-
Transformers predict next tokens not lookahead like humans
By
–
I think our most crucial finding is that although humans think far ahead while speaking (especially while doing complex reasoning problems) it turns out that transformer language models…. don't seem to do that. they just predict the next token.
-
Language Models Planning Future Tokens Research
By
–
New Research: a lot of talk today about "what happens" inside a language model, since they spend the exact same amount of compute on each token, regardless of difficulty. we touch on this question on our new theory paper, Do Language Models Plan for Future Tokens?
-
Draft Model Size Considerations in AI Development
By
–
good point but the draft model is relatively small
-
Speculative Decoding: 3-4x Faster Model Inference Performance
By
–
doesn't speculative decoding basically solve this problem already? (I think we can already run models 3-4x faster in practice by using a draft model and doing speculative decoding)