10/ Mitigating Hallucination via Generation Constraint – proposes a new training-free approach to mitigate hallucination in LLMs; they scaled the readout vector that constrains generation in a memory-augmented LLM decoder; recent works claim that LLMs with explicit memory
@dair_ai
-
Model Collapse: Training AI on Synthetic Data Risks
By
–
9/ Model Collapse on Synthetic Data – investigates the effects of training models on recursively generated data; finds that training on model-generated content can cause irreversible defects where the original content distribution disappears; shows that the effect, referred to as
-
Text-to-SQL Survey: LLM Techniques and Benchmarks
By
–
7/ Text-to-SQL Survey – provides a survey on employing LLMs for Text-to-SQL tasks, including prompt engineering techniques, fine-tuning methods, benchmarks, and more.
-
MINT-1T Open-Sources 1 Trillion Token Multimodal Dataset
By
–
8/ MINT-1T – open-sources a large-scale multimodal interleaved dataset consisting of 1 trillion tokens which has 3.4 billion images; it also includes new sources such as PDFs and ArXiv papers.
-
Teaching LLM Agents to Self-Improve Through Iterative Feedback
By
–
6/ Teaching LLM Agents to Self-Improve – claims it is possible to iteratively fine-tune LLMs with the ability to improve their own response over multiple turns with additional environment feedback; the LLM learns to detect and correct its previous mistakes in subsequent
-
LazyLLM Dynamic Token Pruning Accelerates LLM Inference by 2.34x
By
–
5/ LazyLLM – introduces a novel dynamic token pruning method for efficient long-context LLM inference; it can accelerate the prefilling stage of a Llama 2 7B model by 2.34x and maintain high accuracy; it selectively computes the KV for tokens that are important for the next token
-
AlphaProof Solves IMO Mathematics Problems with Formal Reasoning
By
–
2/ AlphaProof & Alpha Geometry 2 – solved 4 out of 6 problems in this year’s IMO which is the equivalent of a silver-medal score; AlphaProof consists of a Gemini model that automatically translates natural language problem statements into formal statements (i.e., formalizer
-
RAG vs Long-Context LLMs: Self-Route Strategy
By
–
3/ RAG vs. Long-Context LLMs – compares RAG and long-context LLMs and finds that long-context LLMs outperform RAG on average performance while RAG is significantly less expensive; proposes Self-Route, leveraging self-reflection to route queries to RAG or LC; reports that
-
OpenDevin Platform Enables Generalist Software Agents
By
–
4/ OpenDevin – presents a platform to develop generalist agents that interact with the world through software; features include 1) an interaction mechanism for interaction between agents, interfaces, and environments, 2) an environment including a sandboxed operating system and
-
Learning at Test Time: Linear Complexity Sequence Modeling Innovation
By
–
10/ Learning at Test Time – proposes new sequence modeling layers with linear complexity and an expressive hidden state; defines a hidden state as an ML model itself capable of updating even on test sequence; by a linear model and a two-layer MLP based hidden state is found to