The Exchanges

Q I cannot end this podcast without talking about the architecture because this is your effectively the, the sort of research level, architecture level innovation that enabled what I've been calling object permanence for Sam and it's memory attention. What was the inspiration going into it? And, you know, what did you find?

A Yeah, so at a high level, the way we think about extending SAM to video is that an image is just a special case of a video that just has one frame. With that idea in mind, we can extend the SAM architecture to be able to support segmentation across videos. So this is a quick video that shows how this works. So SAM architecture, we have the image encoder, we have a prompt encoder, we have a mask decoder. You can click. On an image, and that basically is a prompt. We use that prompt along with the image embedding to make a mask prediction for that image. Going to SAM-II, we can also apply SAM-II to images because we can, you know, as I said, treat an image as a video with a single frame. And so when we are in the SAM-II architecture, we introduce this new memory mechanism that consists of Three main components. There's memory attention, there's a memory encoder, and then there's a memory bank. And when we apply SAM to images, these are effectively not used, and the architecture just collapses down to the original SAM architecture. But when we do apply this to video, the memory components become really useful because they provide the context of the target object from other frames. And so this could be from the past frames, it can be from There's two types of memory. So there's like the condition, conditional frames or the prompted frames, which are basically the frames at which a …

AI assessment note: “an image is just a special case of a video that just has one frame.”

Q So as an audience surrogate who knows less than the two of you, I'm going to ask a really dumb question maybe, but is everyone using stock, a segment, anything? Are they fine tuning for the medical domain? Like how on earth could it work for the medical field without fine tuning, right? Like, is that a thing?

A So I mean, I can give a quick perspective from the research side. So one of the things, design decisions we made in SAM was to not have class labels. And so all the data is annotated in a class agnostic way. So anything that has a boundary, we consider to be an object. So for example, in any image, there's lots of small objects. We might not know what the name of them are, but you can draw a boundary around it. So you can imagine that we have eleven million images in the SA-I-B dataset. We annotated all the objects. There's many, many small objects. And so if you think about cells, there are also kind of small objects. There's probably things in the training data that looked like it, but we didn't have to label it. And so that means that even when you use SAM for applications that it wasn't really trained for, because we didn't Restricted to a certain set of categories, you can actually use it out of the box without custom adaptation. But having said that, there's probably certain domains where you need some expertise in order to be able to segment something properly. And for those use cases, having some extra fine tuning data would probably help. And we've sort of seen that there's some papers that have come out that do this. And, you know, we'd love to hear Joseph, how people are collecting data with SAM and fine tuning for their use cases.

AI assessment note: “because we didn't Restricted to a certain set of categories, you can actually use it out of the box”

Q as well, which I really love. And I think it's innovative if you want to Share anything about that. I think that the top question is like, how do you decide the size of data set? You know, what were you constrained by? People were asking about scaling laws. You had some ablations, but as a research manager for this whole thing, like how do you decide what you need?

A Yeah, I mean, it's a great question. I think it's, as with all papers, you write them at the end of the project, so we can put these nice plots at the end. But going into it, I think, you know, the data engine design really follows sort of the model design, how we thought about the task, how we thought of the model capabilities. You can really see it's reflected in the different phases of the data engine. We started with just SAM. We apply SAM per frame. That's like the most basic way of extending SAM to video. Then the most obvious thing to do is to take the output masks from SAM and then provide it as input into a video object segmentation model that takes the mask as the first frame input. And that's exactly what we did. We had SAM plus a version of SAM-II that only had mask as input. And then in the last phase, we got rid of SAM entirely and just had this one unified model that can do both image and video segmentation, and I can do everything in just one model. And we found that, you know, going from each phase, it both improved the efficiency and it improved the data quality. And in particular, when you get rid of this two-part model, one of the advantages is that when you make refinement clicks, so You prompt the model in one frame to select an object. Then you propagate those predictions to all the other frames of the video to track the object. But if the model makes a m…

AI assessment note: “the data engine design really follows sort of the model design”

Q I'm just interested in leaving breadcrumbs for other researchers. Anyone interested in this kind of architecture? Like, are there papers that you would refer people to that or influential in your thinking or, you know, have, have other interesting alternative approaches?

A I think there's other ways in which you can do tracking in video. You might, you might not even need the full mask. I think there's some other works that just track like points on objects. It really, really depends on what your application is. Like if you don't care about the entire mask, you could just track a bounding box. You could just track a point on an object. And so having the high fidelity mask might not actually be necessary for certain use cases. From that perspective, you might not need the full capabilities. Of SAM or SAM-II. There's many different approaches to tracking. I think I would encourage people to think about, like, what actually they need for their use case, and then try to find something that, that fits versus, yeah, maybe SAM-II is too much. You know, maybe you don't even need the full mask.

AI assessment note: “I think there's some other works that just track like points on objects.”