It’s whatever the byte-pair encoding algorithm says which depends on the pre-train statistics. There are many shorter ones too — the top 130 longest tokens are all banners and whitespace of various lengths.
@goodside
-
BPE token vocabulary depends on frequency in pre-train corpus
By
–
That’s right — the token vocabulary is determined using BPE so long strings only get their own tokens when they occur often in the pre-train corpus.
-
Long tokens exist because they’re common; ChatGPT knows their contents and length.
By
–
Not that surprising — the only reason long tokens exist is because they're common. If you ask about long tokens (in ChatGPT Classic so it doesn't use code) it knows what punctuation/words they contain, can guess their approximate length, etc.
-
Pulled full list from tiktoken and sorted in Python
By
–
I pulled the full list from tiktoken and sorted in Python.
-
Line of dashes in comment: a GPT-4 token
By
–
//—————————————————————————————————————- is a single GPT-4 token.
-
Codellama-70b-instruct success rate, prompt variations
By
–
FWIW I wasn’t able to reproduce, using http://
labs.perplexity.ai with codellama-70b-instruct. I guess it depends on some detail of system prompt / format. I got 10/10 success using both prompts from the replies here and my own. -
Self-rewarding Language Model Learns and Improves Itself
By
–
> be me self-rewarding LM
> write my own prompts
> write my own responses
> give myself feedback
> be me smarter self-rewarding LM -
Defining Jailbreaks vs. Prompt Injections in AI Interactions
By
–
Maybe one way to define the distinction: Jailbreaks violate the fixed policies of the vendor, whereas prompt injections violate the dynamic policies of a "superuser" who controls the outer prompt.
-
GPT-4 Text Handling and Information Hiding Techniques
By
–
Part of the magic is that GPT-4 can read text written this way but in any other context you would need a decoder on the other side. And if you allow a decoder there are lots of ways to hide information in non-printing chars like ZWSP etc.
-
LLMs’ Value in Simplifying Technical Tasks like Matplotlib Usage
By
–
If LLMs achieve nothing but letting you use matplotlib without docs it will all have been worth it.