That’s right — the token vocabulary is determined using BPE so long strings only get their own tokens when they occur often in the pre-train corpus.
LLMS
-
Long tokens exist because they’re common; ChatGPT knows their contents and length.
By
–
Not that surprising — the only reason long tokens exist is because they're common. If you ask about long tokens (in ChatGPT Classic so it doesn't use code) it knows what punctuation/words they contain, can guess their approximate length, etc.
-
Line of dashes in comment: a GPT-4 token
By
–
//—————————————————————————————————————- is a single GPT-4 token.
-
Google to Rename Bard to Gemini
By
–
Google is also planning to rename Bard into Gemini soon (according to leaks). Gonna be interesting
-
Ludwig Celebrates 10k Milestone with LLM Hackathon Winners
By
–
Congrats to our #LLM Hackathon Winners! To celebrate Ludwig's 10k we ran a mini virtual hackathon. Projects include: Automating data quality checks in Python #QA for Indian Tax Code #Classification for Customer Service
and more! https://
pbase.ai/3HGSQMo -
Google releases update for Bard AI
By
–
There still some countries where it is not available yet But finally, google was left behind for so long. It is the first Bard update since December last year
-
Comparing AI tool capabilities and image formatting
By
–
Quite unfortunate that we cannot make the wide and not square Wondering if it is better what Copilot is offering
-
Building Large Language Models for Quantitative Finance Applications
By
–
In the #generativeAI landscape, large language models (LLMs) stand out as game-changers. In this webinar, learn the nuances of building LLMs, with a focus on how they can be used in #quantitativefinance. Register now. #financialservices https://
nvda.ws/3SHMoLl -
Building Long-Context RAG with Nomic Embeddings and Mistral
By
–
Build Long-context RAG from scratch: Nomic Embeddings + Mistral The context window of open source LLMs and embedding models has been relatively small vs proprietary models. But, methods to expand context window (RoPE, self-extend) are quickly changing this. Today, @nomic_ai has
-
Pi LLM Stands Out for Daily Chat and Smart Suggestions
By
–
This is what makes @pi stand out from other LLMs for me. I use it for daily chat, general information and brainstorming. It's friendly and it goes beyond just answering what you said, it gives you extra suggestion and he also remembers what you said before.
→ View original post on X — @inflectionai, 2024-02-01 19:56 UTC