Dec 22, 2024 · 55m · latent-space
Best of 2024 in Vision [LS Live @ NeurIPS]
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Presented live at NeurIPS, this technical retrospective highlights the defining computer vision breakthroughs of 2024, analyzing generative video architectures, the ascendance of real-time Detection Transformers over YOLO, methods for resolving vision-language model perceptual blindness, and compact edge models like Moondream.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.3 Guest teaching 0.3 Guest disagreement 0.1 The hosts pushing back 0.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 20:30 LLMs literally cannot see
Peter takes a strong contrarian stance against general VLM hype by proving top commercial models fail on simple visual clock tasks.
Hardest push from the hosts ▶ 39:10 Audience pushes on generalist model failureAn audience participant directly questions why major frontier lab models remain inferior to real-time DETRs at object detection.
Biggest teaching moment ▶ 40:06 Isaac unpacks detection architecture limitsIsaac educates the room on the domain-specific nuances of object detection architectures and historical pre-training deficits in convolutional models.
The host holds their own ▶ 54:44 Host highlights vision as first among equalsHost Alessio synthesizes the presentations by emphasizing vision's preeminent importance across multimodal AI paradigms.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Overview of Major Computer Vision Trends in 2024 | 0 | 0 | 0 | 0 | Isaac presents a solo technical lecture on video generation models and Sora replications without host dialogue. Host-side engagement and pushback are absent during this monologue. | |
| SAM 2 and Temporal Memory for Promptable Video Segmentation | 0 | 0 | 0 | 0 | Isaac continues his presentation detailing SAM 2 architecture, hierarchical encoders, and temporal memory banks. There is no host involvement or disagreement. | |
| The Ascendance of DETRs Over YOLO in Real-Time Object Detection | 0 | 0 | 0 | 0 | Isaac presents technical benchmarks comparing real-time DETRs to YOLO architectures. The segment is entirely a presentation monologue without host participation. | |
| Investigating VLM Blindness and Fine-Grained Perception with MMVP | 0 | 0 | 1 | 0 | Peter begins his talk with the provocative thesis that VLMs cannot see fine details, explaining the MMVP benchmark. It is an instructional presentation delivered to an audience without conversational pushback. | |
| Architectures for Spatial and Semantic Granularity: Florence-2, PaliGemma 2, and AIMv2 | 0 | 0 | 0 | 0 | Peter walks through Florence-2, PaliGemma 2, and AIMv2 benchmark results in an educational format. Host metrics remain zero due to the lecture setting. | |
| Audience Q&A on Foundation Models Versus Specialist Object Detectors | 1 | 2 | 0 | 0 | The host opens the floor to audience questions, prompting Isaac and Peter to explain why generalist VLMs lag behind specialist object detectors. The guests collaboratively break down domain-specific architectures and pre-training dynamics. | |
| Moondream: Compact Vision-Language Models and Sub-Billion Edge Pruning | 0 | 0 | 0 | 0 | Vik presents Moondream's architecture and sub-billion edge pruning methodology. The segment proceeds as a solo technical demo and presentation. | |
| Grounded Chain-of-Thought for Analog Gauge Reading and Visual Reasoning | 1 | 0 | 0 | 0 | Vik explains grounded chain-of-thought for gauge reading before the host offers a brief concluding thought on vision modalities. The dynamic is entirely collaborative and educational. |
Statements from this episode (23)
Robinson: DETRs are replacing YOLO in real-time object detection
“And then also how debtors are starting to take over the real time object detection scene from YOLOs which have been dominant for years.”
Robinson: 2024 DETR improvements are Pareto superior to YOLO
“And then how debtors are the improvements in 2024 to debtors that are making them a Pareto improvement to yellow base models.”
Robinson: High-Performance Diffusion Models Are Shifting to Rectified Flows
“It's also it's also worth noting that most diffusion models today, the very high performance ones are switching away from the classic like DDPM, Denoising Diffusion Probability Modeling Framework to rectified flows.”
Robinson: Rectified Flows Enable Faster Sampling by Approaching Single-Step Inference
“Rectified flows have a very interesting property of that. As they converge, they actually get closer to being able to be sampled with a single step, which means that in practice, you can actually generate high quality samples much faster.”
Robinson: Original DiT Research Showed Compute Scaling Outweighed Hyperparameters
“This is so interesting because the original diffusion transformer paper from Facebook actually showed that, in fact, the specific hyperparameters of the transformer didn't really matter that much. What mattered was that you were just increasing the amount of c…”
Robinson: SAM has saved Roboflow users 75 years of labeling time
“SAM for us has saved our users 75 years of labeling time.”
Robinson: SAM 2's hierarchical encoder delivers 6x faster inference than ViT
“SAM replaced that with a hierarchical encoder, which gets approximately the same results, but leads to a six times faster inference, which is excellent, especially considering how in a trend of 23 was replacing the VIT with more efficient backbones.”
Robinson: YOLO architectures have hit a performance plateau
“So, for years, yellows have been the dominant way of doing real time object detection, and we can see here that they've essentially stagnated. The performance between 10 and 11 is not meaningfully different. At least, you know, in, in this type of high level c…”
Robinson: D-FINE models achieve +4.6 AP over YOLO at equivalent latency
“So, we can look here and see the yellow series has this plateau and then these RT debtor, LW debtor and define have meaningfully changed that plateau so that in fact the best defined models are plus 4.6 AP on Coco at the same latency.”
Robinson: Factoring in NMS Latency Shows DETRs Outperform YOLO
“Once you include the NMS in the latency calculation, you see that, in fact, these debtors are outperforming, at least at this time, the yellows that existed.”
Robicheaux: CLIP encoders lack fine-grained visual features due to caption-matching objective
“Models that have been initialized with Clip as their vision encoder, they don't have fine-grained details and the features extracted using Clip because Clip sort of doesn't need to find these fine grade details to do its job correctly, which is just to match c…”
Robicheaux: Adding DINOv2 features degrades multimodal language modeling performance
“As you increase the number of Dynav two features, your model does worse and worse and worse on the actual language modeling task, and that's because Dynav two features were trained completely from a self-supervised manner and completely in image space. It know…”
Robicheaux: Florence-2 Achieves 60% mAP on COCO
“They get 60%, 60% map on Cocoa, which is, like, approaching state of the art, and they train with... You're good. And they train with a much more much more efficiently.”
Robicheaux: PaliGemma 1 Pre-Training Saturates at 300M Examples
“One of my critiques, I guess, of polygeoma one, at least, is that you find that performance saturates as a pre-trained model after only three hundred million examples seen.”
Robicheaux: 2B PaliGemma 2 Beats ChatGPT on MMVP with 47.3%
“The big result, and one of the reasons that I was really excited about this paper, is that they blow everything else away on MMVP. I mean, 47.3, sure, that's nowhere near human accuracy, which again is 94%, but for, you know, two billion language, two billion …”
Robicheaux: AIMv2 Avoids Performance Saturation as Scale Increases
“And we can see that this is finally a model that doesn't saturate. It's even at the highest parameter count, it's, it appears to be Well, at the highest parameter account, it appears to be improving in performance with more and more samples seen”
Robinson: YOLO and real-time detectors historically gained little from pre-training
“And the other thing is, until recently, the real time object detectors didn't even really benefit from pre-training. Like, you see the Yolos that are, like, essentially saturated showing very little difference with pretraining improvements with using pretraine…”
Robicheaux: Convolutional models fail to benefit from pre-training compared to transformers
“Essentially, I think it's kind of been shown now that convolution models, like, just don't benefit from pre-training and just don't, like, have the level of intelligence to transform models.”
Korupati: Moondream built a 0.5B model by pruning its 2B model
“The way we built our model was to start with the two billion parameter model and
Prune it while doing continual training to retain performance.”
Korupati: Developers Can Prototype on 2B Models and Prune for Deployment
“I think the thing that's really exciting about this is it makes it possible for developers to build using the two B per model and
Just explore, build their application, and then once they're ready to deploy, figure out what exactly they need out of the model a…”
Korupati: VLMs Fail at Gauges Due to E-Commerce Training Biases
“In the case of gauges, most gauges images aren't gauges in the wild. They're product Detail images like these where it's always set to zero. It's paired with an alt text that says something like GIVTO pressure sensor PSI zero to 30 or something. And so, the mo…”
Korupati: Visual Chain-of-Thought Will Probably Generalize Across Tasks
“The real question is, is it going to generalize? Probably, like, there's some signs from text models that when you train on a broad number of tasks, it does generalize, and I'm seeing some signs with our model as well.”
Korupati: Vision-Language Models Are Lagging Behind LLMs in Reasoning
“LLMs are showing enormous progress in reasoning, especially with the latest set of models that we've seen, but we're not really seeing, I have a feeling that VLMs are lagging behind, as we can see with these tasks that should be very simple for a human to do t…”