Screenshot

“Images are like, They often give you so much context and you don't have to type so much. So I feel like this is like one of the hacks where you can prompt faster, just like make a screenshot so that agents are really good at figuring out what you want.”

“The first way is this, the simplest way and the newest way, but it doesn't work very well, which is you take a screenshot of the website and then you click on a particular pixel value on the website and like models are not very good at that at the moment. Like…”

“And then the third way is kind of a hybrid where you present the screenshot, but you also present like a textual summary of the output. And that's the one that I think will probably work best.”

“And one place where, interestingly, segment anything may be less performant than other models is handling screenshots.”