and confusingly, we can predict the next-next token from hidden states at a given timestep. it just turns out that this is seems to be mostly a property of language — for example if you say "New" then it's somewhat likely that "City" is two tokens away
LLMS
-
Transformers predict next tokens not lookahead like humans
By
–
I think our most crucial finding is that although humans think far ahead while speaking (especially while doing complex reasoning problems) it turns out that transformer language models…. don't seem to do that. they just predict the next token.
-
Transformers Planning Ahead Through Hidden States and Caching
By
–
it's not that they *can't* plan ahead. as we note, transformers are actually encouraged to make information useful for future states, since the loss at a given position is computed using hidden states at previous positions and in a fake problem where they have to cache
-
Language Models Planning Future Tokens Research
By
–
New Research: a lot of talk today about "what happens" inside a language model, since they spend the exact same amount of compute on each token, regardless of difficulty. we touch on this question on our new theory paper, Do Language Models Plan for Future Tokens?
-
Claude-LLM-Trainer Gains Momentum Among AI Enthusiasts
By
–
Thanks for writing about Claude-LLM-Trainer, Ben! Excited for more people to experience it!
-
Cohere Command R+ Advances LLM Capabilities with 128k Context
By
–
The new @cohere Command R+ model looks like a serious contender. – 128k context length
– best-in-class function calling (beats GPT-4-Turbo)
– fantastic multilingual support Excited to try this model and see what it enables! -
Cohere releases Command R+ LLM with 128k context, multi-language support, and tool use
By
–
New Command R+ from Cohere — 128k context, open weights for non-commercial use, commercial API priced similar to Claude 3 Sonnet Tokenizer is designed to be efficient in 10 languages so definitely consider for non-English text. Multi-hop tool use sounds interesting too
-
DBRX Instruct AI Model Now Available on Poe Platform
By
–
DBRX Instruct is hosted by @FireworksAI_HQ
. You can try it today at https://
poe.com/dbrx-instruct-
fw
… and across all Poe apps. (2/2) -
DBRX Instruct Open Source Model Launches on Poe Platform
By
–
New on Poe: DBRX Instruct! This leading-edge open source model excels at a wide range of tasks, including language understanding, programming, logic, and math, and has been fine-tuned to specialize in few-turn interactions. (1/2)
-
LLM Tool Adds Command R and Command R Plus Plugin Support
By
–
My LLM command-line tool for interacting with large language models now has a plugin for Command R and Command R Plus