5/ Fine-Tuning Language Models with Just Forward Passes – proposes a memory-efficient zeroth-order optimizer and a corresponding SGD algorithm to finetune large LMs with the same memory footprint as inference.
@dair_ai
-
Thought Cloning: Learning to Think While Acting
By
–
4/ Thought Cloning – introduces an imitation learning framework to learn to think while acting; the idea is not only to clone the behaviors of human demonstrators but also the thoughts humans have when performing behaviors.
-
PaLM’s Multilingual Capabilities: Translation Pairs Across 44 Languages
By
–
10/ Searching for Needles in a Haystack – shows that PaLM is exposed to over 30 million translation pairs across at least 44 languages; shows that incidental bilingualism connects to the translation capabilities of PaLM.
-
Symbol Tuning Boosts Language Models In-Context Learning Performance
By
–
9/ Symbol Tuning – an approach to finetune LMs on in-context input-label pairs where natural language labels are replaced by arbitrary symbols; boosts performance on unseen in-context learning tasks and algorithmic reasoning tasks.
-
CodeT5+ Achieves State-of-the-Art Code Understanding and Generation
By
–
8/ CodeT5+ – supports a wide range of code understanding and generation tasks and different training methods to improve efficacy and computing efficiency; achieves SoTA on tasks like code completion, math programming, and text-to-code retrieval tasks.
-
DoReMi: Domain-Weighted Resampling for Efficient Model Training
By
–
7/ DoReMi – trains a small proxy model over domains to produce domain weights without knowledge of downstream tasks; it resamples a dataset with the domain weights which allows using a 280M proxy model to train an 8B model (30x larger) more efficiently.
-
TinyStories: Efficient Language Models for Story Generation
By
–
6/ TinyStories – uses a synthetic dataset of short stories to train and evaluate LMs that are much smaller than SoTA models but can produce fluent and consistent stories with several paragraphs, and demonstrate reasoning capabilities.
-
Drag Your GAN: Interactive Point-Based Image Control Method
By
–
1/ Drag Your GAN – an approach for controlling GANs that allows dragging points of the image to precisely reach target points in a user-interactive manner.https://t.co/Nbhtle5VRG
— DAIR.AI (@dair_ai) 21 mai 20231/ Drag Your GAN – an approach for controlling GANs that allows dragging points of the image to precisely reach target points in a user-interactive manner.
-
Top ML Papers of the Week: DragGAN, CodeT5+, Med-PaLM 2
By
–
Top ML Papers of the Week (May 15 – 21): – DragGAN
– CodeT5+
– StructGPT
– Med-PaLM 2
– Symbol Tuning
– Evidence of Meaning in LLMs
… -
MultiModal-GPT: Vision Language Model for Multi-Round Dialogue
By
–
10/ MultiModal-GPT – a vision and language model for multi-round dialogue with humans; the model is fine-tuned from OpenFlamingo, with LoRA added in the cross-attention and self-attention parts of the language model.