Vision Language Models

“A lot of our developer within Infrax and for VLM are like retreating from using Fable five because you have a two hour job and you trigger the red line, which is false positive. And then you have to lose all of your work. And so a lot of our developer are usin…”

“Getting to the point of I can make a token out of this model is not that hard because generally the open source inference engines, your VLMs, SGLangs of the world oftentimes even receive weights ahead of time maintainers do, or the people making the model merg…”

“We actually have a vision language model controlling a Windows 95 machine. Because that's the only way to automate it.”

“And what this two work really show is that if you start from a vision language model that is really powerful, and you kind of use robotic data to adapt this model to speak robot language, if you will then you see a lot of transfer from the kind of knowledge th…”

“I mean, I think it's fair to say that, you know, vision understanding sort of stalled out, right? You got to object recognition, and then progress just wasn't being made, right? If you look at any of these vision language models, it's the language that's doing…”

“I think most AI chips will not be consumed by people programming anything for it. They will download an open source inference engine, they will download an open source model, and then they will put it on the, and it's really simple to download VLM and, like, m…”

“All of them will have a very good UX for download model, run model on VLM by The middle of the year, I think, right? Certainly AMD is already there by the end of this quarter.”

“The challenge is that if we, if you think that language model rewards are hackable vision language model rewards or, you know, like other sensory signal rewards are infinitely more hackable.”

“At this point we have Six different vision models in a VLM”

“And what we start to realize is that all of these different data sources contribute to each other. They give you just like a bigger picture of what the world is like and better understanding. And it just turns out that robot actions is just like yet another la…”

“Base cognitive functions are just becoming available as an API. So like vision is just going to be available. We shouldn't work on a vision problem, like go fine tune like a Yolo or whatever. VLM is your favorite.”

“And in terms of the architecture, we're using transformers and we are using pre-trained models, pre-trained vision language models, and that allows you to leverage all of the rich information on the internet.”

“We had a research result a couple years ago where we showed that if you leverage vision language models, then you could actually get the robot to do tasks that require concepts that were never in the robot's training data, but were in the internet.”

“In the case of gauges, most gauges images aren't gauges in the wild. They're product Detail images like these where it's always set to zero. It's paired with an alt text that says something like GIVTO pressure sensor PSI zero to 30 or something. And so, the mo…”

“The real question is, is it going to generalize? Probably, like, there's some signs from text models that when you train on a broad number of tasks, it does generalize, and I'm seeing some signs with our model as well.”

“LLMs are showing enormous progress in reasoning, especially with the latest set of models that we've seen, but we're not really seeing, I have a feeling that VLMs are lagging behind, as we can see with these tasks that should be very simple for a human to do t…”

“The power of transformers, the power of realism is mind-blowing, right? So with just a little bit of effort, you get something on the road, and it works. You can, you know, drive, I don't know, 1000 of miles, and we just, it will blow your mind. But then is th…”

“Most VLMs are distillations of proprietary closed source models, right? So if you need to generate synthetic data, like most open weight models rely heavily on synthetic data from private models.”

“For example, all this proprietary stuff sucks at clocks, so nothing that's a distillation will be good at clocks. Nothing can point if you just distill from this. If they can't point, your VLM won't point, so we show how to get good data.”

“My intuition at least is that I suspect it's really, the heavy lifting might be actually more towards the Q and A side rather than captioning side. I view captioning as a bootstrap, but at the end of the day, we want the model to generalize what they want to d…”

“So that, that's what we've been very focused on that Waymo most recently is taking kind of the AI backbone and all of the AI, the Waymo AI that is over the years we've built up that is really proficient at this task of autonomous driving and combining it with …”