Sadly it doesn't support streaming tokens yet – I nearly got that working but ran into an API bug that I couldn't work around
CODE
-
API Streaming Bug Prevents Incomplete Token Display
By
–
I almost managed to get streaming to work against your API after some reverse-engineering, but then ran into a show-stopper bug where incomplete tokens were returned in a way that means I can't stream without accidentally displaying them
-
New LLM Plugin llm-reka for Reka AI Models API
By
–
New LLM plugin: llm-reka – for running prompts against the @RekaAILabs family of models via their API https://
github.com/simonw/llm-reka pipx install llm
llm install llm-reka
llm keys set reka
# paste key here
llm -m reka-core '5 fun facts about walruses' -
API Streaming Support Request for AI Model Responses
By
–
Any plans to add streaming to your API? It looks like right now it only supports returning the entire response as a single chunk of JSON
-
LLM Randomness: Breaking Out to NumPy for Random Choice
By
–
Whenever I want it to do the equivalent of random.choice, I break out to a tool that calls np.random. I'd love if I could just tell it to be random.
-
Exploring Tokenizer Vocabulary: Tool Call Token Structure
By
–
Here's how to start exploring the vocabulary of the tokenizer – interesting to see they have tokens like [TOOL_CALLS], [AVAILABLE_TOOLS], [/AVAILABLE_TOOLS], [TOOL_RESULTS] and [/TOOL_RESULTS] in there
-
Cost estimation and context optimization for advanced RAG queries
By
–
I want to be able to give users a cost estimate before they run larger prompts, but I'm also interested in knowing how much content I can cram into the context for advanced RAG queries
-
Anthropic Needs Public Tokenizer API Like Google Gemini
By
–
Anthropic don't release any form of their tokenizer at all – not even an API endpoint that lets you count tokens before prompting with them (Google Gemini has that) It's a small thing but it's the feature I most want from them at the moment
-
Mixtral Tokenizer Solves OpenAI’s JSON Token Counting Problem
By
–
OpenAI's tokenizer is available (as the tiktoken Python library) but you're on your own when it comes to guessing how many tokens will be used by the additional JSON syntax used for previous messages and tool definitions Mixtral's tokenizer here solves that problem
-
Document Upload Transparency and AI Processing Visibility
By
–
I want to know exactly what happens to documents I upload to it – how they are split and chunked, how text is extracted from PDFs, what gets embedded and stored, and what gets retrieved and included in prompts (Ideally I'd like a debug interface that lets me see it for myself)