BTW, Since Jamba supports a 256K context with high throughput, we also stumbled upon an issue where the fused_moe kernel didn’t work well in long contexts. Others seems to have had this too, according to some other open issues
LLMS
-
Jamba Model Now Supported in vLLM for Efficient Serving
By
–
Jamba support is now live on vLLM Due to its novel hybrid SSM-Transformer arch, Jamba didn’t work out-of-the-box in vLLM. our own @MorZusman worked together with @vllm_project to integrate Jamba for an efficient serving in vLLM
-
Jamba Modeling and Mamba Cache Management Updates
By
–
Things in this PR:
– Jamba modeling file, supporting Jamba variations
– Mamba cache management (also benefit other Mamba-based models)
– Send more request related properties to the forward pass, for model-specific implementations (including things needed for speculative decoding) -
Agentic RAG with Transformers Agents Improves Retrieval Performance
By
–
New cookbook! I show to to make agentic RAG using Transformers Agents. Compared to vanilla RAG, agentic RAG can: Reformulate the query Critique the retrived content to re-retrieve if needed Score increase of 8.5%! (Llama-3-70B-judge)
-
Character AI introduces Suggested Reply feature for chat assistance
By
–
Character AI is working on the Suggested Reply feature. It takes the 3 latest messages from the chat context and generates suggestions for a quick selection. I haven't played enough with @character_ai yet but will this be like a chat on autopilot?
-
LLM assistants usage frequency over past 6 months
By
–
Over the past 6 months, how often have you been using LLM assistants compared to the previous 6 month period?
-
ICL versus Fine-tuning: Cost-Latency Trade-offs in LLM Optimization
By
–
I agree actually, if a problem can be solved with ICL (in terms of cost, latency, etc. and not just accuracy), then it's perfect. But I want to point out that it also makes fine-tuning even more relevant in a lot of use cases.
-
Ctrl-G Framework: Logical Control for LLM Generation
By
–
7/ Adaptable Logical Control for LLMs – presents the Ctrl-G framework to facilitate control of LLM generations that reliably follow logical constraints; it combines LLMs and Hidden Markow Models to enable following logical constraints (represented as deterministic finite
-
Synthetic Data Effects on LLM Bias and Internal Attributes
By
–
8/ LLM See, LLM Do – closely investigates the effects and effectiveness of synthetic data and how it shapes a model’s internal biases, calibration, attributes, and preferences; finds that LLMs are sensitive towards certain attributes even when the synthetic data prompts appear