1/ The Reversal Curse – finds that LLMs trained on sentences of the form “A is B” will not automatically generalize to the reverse direction “B is A”; shows the effect across model sizes and model families.
@dair_ai
-
Top ML Papers of the Week: LLMs and Vision Transformers
By
–
Top ML Papers of the Week (Sep 25 – Oct 1): – MentalLlaMa
– Boolformer
– The Reversal Curse in LLMs
– Long-Context Scaling with LLMs
– Graph Neural Prompting with LLMs
– Vision Transformers Need Registers
… -
Galactica: Specialized LLM Achieves SOTA Scientific NLP Results
By
–
Ever wondered how far specialized large language models (LLMs) can go in scientific NLP? Meet Galactica. At the time of release, this LLM attained new SOTA results in PubMedQA and MedMCQA with a 106B-token scientific corpus and excels in tasks like LaTeX equations and reasoning.
-
OWL: Specialized LLM for IT Operations Management
By
–
9/ LLMs for IT Operations – proposes OWL, an LLM for IT operations tuned using a self-instruct strategy based on IT-related tasks; it discusses how to collect a quality instruction dataset and how to put together a benchmark.
-
KOSMOS-2.5: Multimodal AI for Document Text Generation
By
–
10/ KOSMOS-2.5 – a multimodal model for machine reading of text-intensive images capable of document-level text generation and image-to-markdown text generation.
-
Language Modeling as Compression: LLM Prediction Capabilities
By
–
7/ Language Modeling is Compression – evaluates the compression capabilities of LLMs; investigates how and why compression and prediction are equivalent; shows that LLMs are powerful general-purpose compressors due to their in-context learning abilities.
-
Compositional Foundation Models for Long-Horizon Goals
By
–
8/ Compositional Foundation Models – proposes foundation models that leverage multiple expert foundation models trained on language, vision, and action data to solve long-horizon goals.
-
LMSYS-Chat-1M: 1M Conversations Dataset with 25 LLMs
By
–
6/ LMSYS-Chat-1M – a large-scale dataset containing 1 million real-world conversations with 25 state-of-the-art LLM; it is collected from 210K unique IP addresses on the Vincuna demo and Chatbot Arena website.
-
LLMs for Structured Data Generation: Fine-tuning Approach
By
–
5/ LLMs for Generating Structured Data – studies the use of LLMs for generating complex structured data; proposes a structure-aware fine-tuning method, applied to Llama-7B, which significantly outperform other model like GPT-3.5/4 and Vicuna-13B.
-
Contrastive Decoding Improves LLM Reasoning Performance
By
–
3/ Contrastive Decoding Improves Reasoning in Large Language Models – shows that contrastive decoding leads Llama-65B to outperform Llama 2 and other models on commonsense reasoning and reasoning benchmarks.