LlamaFusion Extends pretrained text-only LLMs with multimodal capabilities while preserving language performance. Problem:
Training multimodal models from scratch is costly and risks degrading pretrained language abilities. Method:
Adds image-specific modules to pretrained
@askalphaxiv
-
LlamaFusion Extends LLMs with Multimodal Capabilities
By
–
-
LMAgent: Multimodal Large-Scale Multi-User Simulation Framework
By
–
LMAgent: A Large-Scale Multimodal Agents Society for Multi-user Simulation A scalable framework for simulating dynamic, multimodal multi-user behavior using large-scale multimodal LLMs. Problem: Existing LLM-based multi-agent systems simulate only text-based interactions
-
Best-of-N Jailbreaking Method Bypasses Major AI Models
By
–
Best-of-N Jailbreaking Introducing a black-box method that jailbreaks AI models across text, vision, and audio by applying simple prompt augmentations. BoN achieves high success rates, such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet, with 10,000 augmented prompts, and also
-
Scaling Speech-Text Pre-Training with Synthetic Interleaved Data
By
–
Scaling Speech-Text Pre-Training with Synthetic Interleaved Data A method for scaling speech language models (SpeechLMs) by using synthetic speech-text interleaved data, bypassing the need for parallel speech-text datasets. Problem: Limited unsupervised speech and parallel
-
NVILA: Efficient Frontier Visual Language Models
By
–
NVILA: Efficient Frontier Visual Language Models A family of efficient and accurate visual language models that optimize processing high-resolution images and long videos. Problem: While VLMs have advanced in accuracy, their efficiency remains a significant challenge,
-
PaliGemma 2: Versatile Vision-Language Models for Transfer
By
–
PaliGemma 2: A Family of Versatile VLMs for Transfer An upgraded vision-language model family optimized for transfer tasks across various domains and resolutions. Problem: Previous VLMs lacked versatility in model size, resolution, and task-specific transfer capabilities,
-
AI Governance Blueprint: Maximizing Benefits While Managing Risks
By
–
Shaping AI's Impact on Billions of Lives A blueprint for leveraging AI to maximize societal benefits while minimizing risks through guided milestones and collaboration. Problem: AI’s transformative potential risks being guided solely by commercial interests, leading to
-
Motion Prompting Framework for Precise Video Generation Control
By
–
Motion Prompting for Generation A flexible motion control framework for generating videos with precise spatio-temporal dynamics through motion prompts. Problem: Existing video generation models lack nuanced control over dynamic actions and temporal compositions,
-
Florence-VL: Enhanced Multimodal Vision-Language Model
By
–
Florence-VL A multimodal large language model (MLLM) that integrates enriched visual representations from Florence-2 for improved vision-language alignment. Problem: Existing vision-language models (VL) are limited by less versatile visual representations and the need for
-
EfficientTAMs: Lightweight Video Object Segmentation Models
By
–
Efficient Track Anything (EfficientTAMs) Lightweight models for video object segmentation and tracking, optimized for speed and smaller model sizes. Problem: SAM 2's high computational complexity limits real-world applications, especially on mobile devices. Method: