Robicheaux: CLIP encoders lack fine-grained visual features due to caption-matching objective
Peter Robicheaux · Best of 2024 in Vision [LS Live @ NeurIPS] · Dec 22, 2024 · at 22:21
Peter Robicheaux of Roboflow explains why multimodal foundation models suffer from perceptual blindness on fine visual details.
0:00 / 0:17exact quote · 17.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Models that have been initialized with Clip as their vision encoder, they don't have fine-grained details and the features extracted using Clip because Clip sort of doesn't need to find these fine grade details to do its job correctly, which is just to match captions and images, right?”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Peter Robicheaux
Robicheaux: 2B PaliGemma 2 Beats ChatGPT on MMVP with 47.3%
“The big result, and one of the reasons that I was really excited about this paper, is that they blow everything else away on MMVP. I mean, 47.3, sure, that's nowhere near human accuracy, which again is 94%, but for, you know, two billion language, two billion …”
Robicheaux: AIMv2 Avoids Performance Saturation as Scale Increases
“And we can see that this is finally a model that doesn't saturate. It's even at the highest parameter count, it's, it appears to be Well, at the highest parameter account, it appears to be improving in performance with more and more samples seen”
Robicheaux: Convolutional models fail to benefit from pre-training compared to transformers
“Essentially, I think it's kind of been shown now that convolution models, like, just don't benefit from pre-training and just don't, like, have the level of intelligence to transform models.”
Robicheaux: Adding DINOv2 features degrades multimodal language modeling performance
“As you increase the number of Dynav two features, your model does worse and worse and worse on the actual language modeling task, and that's because Dynav two features were trained completely from a self-supervised manner and completely in image space. It know…”