7/ Aya23 – a family of multilingual language models that can serve up to 23 languages; it intentionally focuses on fewer languages and allocates more capacity to these languages; shows that it can outperform other massive multimodal models on those specific languages.
@dair_ai
-
Attention as RNN: Parallel Training with Constant Memory Inference
By
–
6/ Attention as an RNN – presents a new attention mechanism that can be trained in parallel (like Transformers) and be updated efficiently with new tokens requiring constant memory usage for inferences (like RNNs).
-
GNN-RAG: Combining LLMs with Graph Neural Networks
By
–
5/ GNN-RAG – combines the language understanding abilities of LLMs with the reasoning abilities of GNNs in a RAG style; the GNN extracts useful and relevant graph information while the LLM takes the information and leverages its capabilities to perform question answering over
-
Abacus Embeddings Achieves 99% Accuracy on Addition Problems
By
–
3/ Abacus Embeddings – achieves 99% accuracy on 100-digit addition problems by training on only 20-digit numbers with a single GPU; the main challenge this work addresses is the inability of transformers to track the exact position of digits.
-
Vision-Language Modeling: Introduction and Effective Training Methods
By
–
4/ Introduction to Vision-Language Modeling – presents an introduction to vision-language models along with key details of how they work and how to effectively train these models.
-
Contextual Position Encoding: New Method for Transformer Models
By
–
1/ Contextual Position Encoding – proposes a new position encoding method, CoPE, to enable the position to be conditioned on context by incrementing position only on certain tokens…
-
Top ML Papers of the Week: SimPO, GNN-RAG, and More
By
–
The Top ML Papers of the Week (May 27 – June 2): – SimPO
– GNN-RAG
– Attention as an RNN
– Abacus Embeddings
– Symbolic Chain-of-Thought
– Contextual Position Encoding
… -
DeepSeek-Prover: AI Model Generates Lean 4 Mathematical Proofs
By
–
9/ DeepSeek-Prover – introduces an approach to generate Lean 4 proof data from high-school and undergraduate-level mathematical competition problems; it uses the synthetic data, comprising of 8 million formal statements and proofs, to fine-tune a DeepSeekMath 7B model…
-
Efficient Multimodal LLMs: Survey of Structures and Applications
By
–
10/ Efficient Multimodal LLMs – provides a comprehensive and systematic survey of the current state of efficient multimodal large language models; discusses efficient structures and strategies, applications, limitations, and promising future directions.
-
INDUS: Comprehensive LLM Suite for Scientific Earth and Planetary Research
By
–
8/ Scientific Applications of LLMs – presents INDUS, a comprehensive suite of LLMs for Earth science, biology, physics, planetary sciences, and more; includes an encoder model, embedding model, and small distilled models.