They first used the INQUIRE dataset to test if VLMs could narrow a pool of five million images to the top 100 most relevant results (AKA ranking). For straightforward search queries like "a reef w/manmade structures & debris," relatively large models like "SigLIP" found matching
MULTIMODAL AI
-
Multimodal LLMs Struggle with Reranking Task Performance
By
–
But how well could multimodal models rerank those 100 results? In these tests, huge LLMs trained on more curated data like GPT-4o struggled. Its precision score was only 59.6%, the highest achieved by any model.
-
INQUIRE Dataset: Domain-Specific Image Search Challenges
By
–
What makes the INQUIRE dataset so challenging? The answer: its domain specificity. INQUIRE includes search queries based on discussions w/ecologists, biologists, oceanographers, & other experts about the types of images they’d look for, including animals’ unique physical
-
Larger VLMs Risk Producing Biased Results for Nature Researchers
By
–
In these evaluations, the researchers found that larger, more advanced VLMs, which are trained on far more data, can sometimes get ecologists & other nature researchers the results they want to see. pic.twitter.com/B0uIYYj8M3
— MIT CSAIL (@MIT_CSAIL) 17 décembre 2024In these evaluations, the researchers found that larger, more advanced VLMs, which are trained on far more data, can sometimes get ecologists & other nature researchers the results they want to see.
-
AI Models Struggle With Expert Knowledge in Visual Analysis Tasks
By
–
The models performed reasonably well on straightforward queries about visual content, like identifying debris on a reef, but struggled significantly w/queries requiring expert knowledge, like IDing specific biological conditions or behaviors.
-
Multimodal VLMs for Biodiversity Image Retrieval Research
By
–
Could multimodal vision language models (VLMs) help biodiversity researchers retrieve images for their studies? MIT CSAIL, @ucl
, @inaturalist
, @EdinburghUni
, & @UMassAmherst researchers designed a performance test to find out. Each VLM’s task: Locate & reorganize the most -
Video Data Contains More Intermediate Reasoning Than Published Text
By
–
Maybe it's because videos data contain a moderate amount of intermediate reasoning steps. For a natural/unedited video, everything is raw, live, and corrections are versioned while texts(on the web) tends to be the final version people publishes.
-
Advanced Voice Mode Integration with Search Functionality
By
–
3. Integration of Advanced Voice Mode and Search Functionality Perhaps the most significant aspect of this announcement is the integration of advanced voice mode with the search functionality. Previously, real-time APIs could perform search tasks via function calls, but
-
Apollo-3B multimodal AI model now available for testing
By
–
Play with the model directly over here: https://
huggingface.co/spaces/Apollo-
LMMs/Apollo-3B
… -
Meta Apollo Multimodal Models: 7B SoTA Video Understanding
By
–
Let's gooo! @AIatMeta released Apollo Multimodal Models Apache 2.0 licensed – 7B SoTA & beats 30B+ checkpoints🔥
— Vaibhav (VB) Srivastav (@reach_vb) 16 décembre 2024
Key insights:
> 1.5B, 3B and 7B model checkpoints
> Can comprehend up-to 1 hour of video 🤯
> Temporal reasoning & complex video question-answering
> Multi-turn… pic.twitter.com/v8bjDCp6qdLet's gooo! @AIatMeta released Apollo Multimodal Models Apache 2.0 licensed – 7B SoTA & beats 30B+ checkpoints Key insights: > 1.5B, 3B and 7B model checkpoints
> Can comprehend up-to 1 hour of video > Temporal reasoning & complex video question-answering
> Multi-turn