I almost managed to get streaming to work against your API after some reverse-engineering, but then ran into a show-stopper bug where incomplete tokens were returned in a way that means I can't stream without accidentally displaying them
@simonw
-
New LLM Plugin llm-reka for Reka AI Models API
By
–
New LLM plugin: llm-reka – for running prompts against the @RekaAILabs family of models via their API https://
github.com/simonw/llm-reka pipx install llm
llm install llm-reka
llm keys set reka
# paste key here
llm -m reka-core '5 fun facts about walruses' -
API Streaming Support Request for AI Model Responses
By
–
Any plans to add streaming to your API? It looks like right now it only supports returning the entire response as a single chunk of JSON
-
Quick Prompt to Assess Model Capability Level
By
–
Anyone got a good short go-to prompt for running against a model that instantly gives an indicator of how capable that model is? I'm trying to figure out a neat way of determining if a model feels gpt-3.5-turbo or gpt-4 level with the minimum amount of work (and API token spend)
-
RAG and Token Optimization with Expanded Context Windows
By
–
I was playing with maximizing RAG content last year, but that was when models still had tiny context lengths – 4,000 or 8,000 wasn't a lot to play with Token optimization like that is less interesting now we have 100,000+ tokens to play with even with the less expensive models
-
Exploring Tokenizer Vocabulary: Tool Call Token Structure
By
–
Here's how to start exploring the vocabulary of the tokenizer – interesting to see they have tokens like [TOOL_CALLS], [AVAILABLE_TOOLS], [/AVAILABLE_TOOLS], [TOOL_RESULTS] and [/TOOL_RESULTS] in there
-
Cost estimation and context optimization for advanced RAG queries
By
–
I want to be able to give users a cost estimate before they run larger prompts, but I'm also interested in knowing how much content I can cram into the context for advanced RAG queries
-
Anthropic Needs Public Tokenizer API Like Google Gemini
By
–
Anthropic don't release any form of their tokenizer at all – not even an API endpoint that lets you count tokens before prompting with them (Google Gemini has that) It's a small thing but it's the feature I most want from them at the moment
-
Mixtral Tokenizer Solves OpenAI’s JSON Token Counting Problem
By
–
OpenAI's tokenizer is available (as the tiktoken Python library) but you're on your own when it comes to guessing how many tokens will be used by the additional JSON syntax used for previous messages and tool definitions Mixtral's tokenizer here solves that problem
-
Document Upload Transparency and AI Processing Visibility
By
–
I want to know exactly what happens to documents I upload to it – how they are split and chunked, how text is extracted from PDFs, what gets embedded and stored, and what gets retrieved and included in prompts (Ideally I'd like a debug interface that lets me see it for myself)