Thanks! Attention modulation (like you did) is I think why autoregressive loss based models can localize objects better. Nice work too!
@_yutaroyamada
-
EMNLP Presentation Tomorrow at East Foyer
By
–
I will be presenting this work at #EMNLP tomorrow, 12/9, from 11:00 to 13:30 at East Foyer. 5/5
-
Concept Association Bias in Vision-Language Models: Contrastive vs Autoregressive
By
–
We call this Concept Association Bias (CAB). We’ve found that models trained using contrastive loss (e.g. parts of BLIP and BLIP-2) also have CAB. However, models trained with autoregressive loss (e.g. OFA and BLIP-2-FlanT5) don't exhibit this bias. 4/5
-
CLIP’s Strategy for Handling Unmentioned Visual Concepts in Images
By
–
Thus, when an image includes two concepts (e.g. a lemon and an eggplant) while the text prompt only mentions one concept (e.g. lemon), CLIP attempts to account for the unmentioned concept (like the eggplant) by saying 'purple', a color commonly associated with eggplants.
3/5 -
CLIP Training: Maximizing Image-Text Embedding Similarity
By
–
This is because CLIP is trained using contrastive loss, where the goal is to maximize the similarity between the embeddings of images and text.
2/5 -
Why CLIP Misidentifies Lemon Color: Purple Instead of Yellow
By
–
When you ask CLIP the color of the lemon in the image below, CLIP responds with ‘purple' instead of 'yellow' (and vice versa). Why does this happen?
1/5 -
LLM Text Adversarial Problems Easier Than Vision Recognition
By
–
I do agree that trying to solve this within LLM will be hard. But the problem seems easier to solve in practice than vision, unless people come up with a way to make an adversarial string that reads like natural language
-
Adversarial Examples in Vision vs Language Models Vulnerability
By
–
Adversarial examples have been hard to solve in vision mainly because it was imperceptible to humans and added adv noise is hard to remove. Looking at the demo, it seems relatively easy to build a filter to remove adversarial suffix before feeding the query into LLM…
-
Dexterous Manipulation in Real Life Applications
By
–
dexterous manipulation in real life 👀 https://t.co/zOvBfvoeLi
— Yutaro Yamada (@_yutaroyamada) 8 janvier 2023dexterous manipulation in real life