The Exchanges

Q the perfect manifestation of how technology always is like much better in one direction when it's invented, but there's always like these things that need to be iterated on. And so are there other things worth knowing about whether it's These negative prompts, whether it's these glitches that are still in the matrix. What would you call out from your, again, many hours of being deep in, in these tools?

A I think it depends on the model. One example was when DALI came out. It's not very good at understanding that it's drawing things in a square. So if, if, if you're drawing a person, it's often going to have like its feet and its head cut off because it's, it's seeing those in portrait photos. But one thing you could do with DALI is you can actually upload like an image To, like, do variations of, and if you upload an image that's just like a little white border, um, then it knows that nothing can go there, and that, that kind of encourages it, forces it to kind of think inside the box, if you will. But then, of course, you have now tools like Midjourney, who've been, like, iterating on their text-to-image model, like, a lot more aggressively than OpenAI, who Understandably, I think maybe have some other things in the, in the, in the cooker, you know, which have now grown that into the model itself. So when you type things in, it knows it's a square and actually it would sometimes do quite clever things in order to fit it in that space. So if you ask for kind of like a group selfie of three people, you know, on something like Dally, that's going to be cut off at the end because you're used to seeing someone taking like a disposable camera photo. Whereas It's clever enough to know that one of them kind of needs to be standing behind the other or like leaning in from the side. So …

AI assessment note: “One example was when DALI came out. It's not very good at understanding”

Q there Certain learnings, maybe the eighty-twenty approach of becoming a good prompt engineer in terms of things that you think are, are really valuable to understand. Maybe it's the prompt length. Maybe it's using certain modifiers within your prompt. Maybe it's just like a, a framework for thinking about prompting. Is there anything that's surface that you think would be really valuable to someone who's just starting out with prompting?

A Oh yeah. Like I think if you've never used one before, like the best Way to explain how they work is to always like describe something as if it already exists. Imagine that it's an image in some kind of downloadable clip art library, or it's, it's on, it's on a photography gallery, and you know, someone's written underneath. Oh, this is a fine example of a modern, uh, photography shot. And those are the kind of descriptions that you're trying to kind of mimic to tell these kind of tools. What you're looking for. And it also gives it like a natural sense of why these tools are bad at some things and, and the kind of prompts that don't really work because you never, you know, if there's like a, let's say some, uh, like an archive image of some women celebrating on the steps of a church in 1972, it will have that kind of caption, but they never go, the woman on the left is wearing a yellow hat. The woman on the right is wearing, you know, they just don't go into that because you can see it. So ironically, they often describe generally what the image is about, but not like how you would draw it step by step. And that's why these tools are less good at saying like, I want this thing over here and then that thing next to it and then something on top and that thing should be much bigger because that's in the real life. That's not how images are described in language.

AI assessment note: “the best Way to explain how they work is to always like describe something as if it already exists.”

Q the perfect manifestation of how technology always is like much better in one direction when it's invented, but there's always like these things that need to be iterated on. And so are there other things worth knowing about whether it's These negative prompts, whether it's these glitches that are still in the matrix. What would you call out from your, again, many hours of being deep in, in these tools?

A I think it depends on the model. One example was when DALI came out. It's not very good at understanding that it's drawing things in a square. So if, if, if you're drawing a person, it's often going to have like its feet and its head cut off because it's, it's seeing those in portrait photos. But one thing you could do with DALI is you can actually upload like an image To, like, do variations of, and if you upload an image that's just like a little white border, um, then it knows that nothing can go there, and that, that kind of encourages it, forces it to kind of think inside the box, if you will. But then, of course, you have now tools like Midjourney, who've been, like, iterating on their text-to-image model, like, a lot more aggressively than OpenAI, who Understandably, I think maybe have some other things in the, in the, in the cooker, you know, which have now grown that into the model itself. So when you type things in, it knows it's a square and actually it would sometimes do quite clever things in order to fit it in that space. So if you ask for kind of like a group selfie of three people, you know, on something like Dally, that's going to be cut off at the end because you're used to seeing someone taking like a disposable camera photo. Whereas It's clever enough to know that one of them kind of needs to be standing behind the other or like leaning in from the side. So …

AI assessment note: “One example was when DALI came out. It's not very good at understanding”

Q there Certain learnings, maybe the eighty-twenty approach of becoming a good prompt engineer in terms of things that you think are, are really valuable to understand. Maybe it's the prompt length. Maybe it's using certain modifiers within your prompt. Maybe it's just like a, a framework for thinking about prompting. Is there anything that's surface that you think would be really valuable to someone who's just starting out with prompting?

A Oh yeah. Like I think if you've never used one before, like the best Way to explain how they work is to always like describe something as if it already exists. Imagine that it's an image in some kind of downloadable clip art library, or it's, it's on, it's on a photography gallery, and you know, someone's written underneath. Oh, this is a fine example of a modern, uh, photography shot. And those are the kind of descriptions that you're trying to kind of mimic to tell these kind of tools. What you're looking for. And it also gives it like a natural sense of why these tools are bad at some things and, and the kind of prompts that don't really work because you never, you know, if there's like a, let's say some, uh, like an archive image of some women celebrating on the steps of a church in 1972, it will have that kind of caption, but they never go, the woman on the left is wearing a yellow hat. The woman on the right is wearing, you know, they just don't go into that because you can see it. So ironically, they often describe generally what the image is about, but not like how you would draw it step by step. And that's why these tools are less good at saying like, I want this thing over here and then that thing next to it and then something on top and that thing should be much bigger because that's in the real life. That's not how images are described in language.

AI assessment note: “describe something as if it already exists”

Q want and didn't picture in your head sometimes requires another tool. And so have you found, I've heard some people are using Facetune or different AIs to take it to The final level, or I guess you could also use in painting and out painting a little more discreetly. So how have you found the relationship of maybe one tool to the suite of other tools that exist out there?

A I think there's lots of exciting crossovers. Actually, I kind of think it's a big opportunity for, like, the photoshops of this world, because those are tools that presuppose you have some kind of original image to be able to, like, manipulate, whereas now there's a huge amount of, like, raw but maybe not perfect material for people to work with. There's lots of things also that I've been trying to, like, do in prompting that are actually, like, more easily achieved in other tools, so you can, you know, I spent ages trying to get this kind of vintage film look, but if you're like an Instagram influencer, you know, which I'm sure you are, um, who isn't? Um, but there's loads of, there's loads of iPhone apps, right, that are out there just to like give all your photos that kind of like dreamy You know, vintage film look.

AI assessment note: “I've been trying to, like, do in prompting that are actually, like, more easily achieved in other tools”

Q some. Controversy around using specific artists work to train your new images, but let's look forward to today. I feel like as we talked about six months later, these tools have evolved a lot. Are there any different ways that you can prompt today or kind of leverage these tools that didn't exist six months ago that are really important and maybe extending the way that you can use them?

A A hundred percent. So The main one, and these things are like changing all the time, right? So, but now there's increasingly tools where you can prompt like with an image. Again, that's almost like an entire new field of exploration, because it's not combining the image with the other, with like your words in the way you would expect something like Photoshop to do it. Like it's not collaging them together. So the results can be really surprising. Really unexpected, probably quite difficult to control, but then you potentially have interesting opportunities like, okay, I can make a load of kind of abstract kind of stuff using like my brand colors or something that's important to me, photos of me, who knows? And then, yeah, and then I'm going to use that and kind of multiply that baseline, that kind of visual base with custom other prompts, and then everything will have this kind of lightness. And then, of course, like the big thing that happened since the days of the prompt book and so on was, of course, that huge spike in interest in, um, selfies, right? Like the lenses and the profile picture.ai, and there were like a dozen of them, which was just prompting with your face, basically, um, prompting with 20 selfies and being like, yeah, I want to see more of this guy because it's me, obviously. And then within the image to image Space. You've now got other startups that are doin…

AI assessment note: “now there's increasingly tools where you can prompt like with an image.”

Q want and didn't picture in your head sometimes requires another tool. And so have you found, I've heard some people are using Facetune or different AIs to take it to The final level, or I guess you could also use in painting and out painting a little more discreetly. So how have you found the relationship of maybe one tool to the suite of other tools that exist out there?

A I think there's lots of exciting crossovers. Actually, I kind of think it's a big opportunity for, like, the photoshops of this world, because those are tools that presuppose you have some kind of original image to be able to, like, manipulate, whereas now there's a huge amount of, like, raw but maybe not perfect material for people to work with. There's lots of things also that I've been trying to, like, do in prompting that are actually, like, more easily achieved in other tools, so you can, you know, I spent ages trying to get this kind of vintage film look, but if you're like an Instagram influencer, you know, which I'm sure you are, um, who isn't? Um, but there's loads of, there's loads of iPhone apps, right, that are out there just to like give all your photos that kind of like dreamy You know, vintage film look.

AI assessment note: “things also that I've been trying to, like, do in prompting that are actually, like, more easily achieved in other tools”

Q Let's take a step back. When we originally reached out to you, it was around six months ago, and you had just written something called your prompt book. Why don't you give everyone a little bit of an idea of what that prompt book was, what it is now, and also what prompted you to want to write it in the first place?

A This was in the initial heyday of DALI too, which was like OpenAI's text to image Model. They rolled it out to a few test people at a time. They were super cautious about how it might be misused, how it kind of could end up kind of having a backlash, all these kinds of things, which then only increased the sense of people wanting to get their hands on this thing, because at the time this was pre things you might think of now as stable diffusion. Mid journey kind of predated those by some small margin and seems way ahead of anything people have kind of tried using before. If you've used a text to image AI by now, you know, it's basically a text box and it all comes down to what you type in. It doesn't have, you know, buttons and all the kind of controls you might expect when you log into something like Photoshop. So the question then becomes like a lot of people, one's mind goes blank or you don't actually know the name Or the words of like what you're trying to type in, right? So if you've actually been to art school or you're up on your art history or your design language, then you probably got a head start on everyone else. But on places like Twitter and Reddit, there are people posting like these amazing images, but because of the nature of social media, it's all lost. So I started trying to like collect these cool examples and these cool kind of terms people were using to c…

AI assessment note: “I started trying to like collect these cool examples and these cool kind of terms”

Q some. Controversy around using specific artists work to train your new images, but let's look forward to today. I feel like as we talked about six months later, these tools have evolved a lot. Are there any different ways that you can prompt today or kind of leverage these tools that didn't exist six months ago that are really important and maybe extending the way that you can use them?

A A hundred percent. So The main one, and these things are like changing all the time, right? So, but now there's increasingly tools where you can prompt like with an image. Again, that's almost like an entire new field of exploration, because it's not combining the image with the other, with like your words in the way you would expect something like Photoshop to do it. Like it's not collaging them together. So the results can be really surprising. Really unexpected, probably quite difficult to control, but then you potentially have interesting opportunities like, okay, I can make a load of kind of abstract kind of stuff using like my brand colors or something that's important to me, photos of me, who knows? And then, yeah, and then I'm going to use that and kind of multiply that baseline, that kind of visual base with custom other prompts, and then everything will have this kind of lightness. And then, of course, like the big thing that happened since the days of the prompt book and so on was, of course, that huge spike in interest in, um, selfies, right? Like the lenses and the profile picture.ai, and there were like a dozen of them, which was just prompting with your face, basically, um, prompting with 20 selfies and being like, yeah, I want to see more of this guy because it's me, obviously. And then within the image to image Space. You've now got other startups that are doin…

AI assessment note: “now there's increasingly tools where you can prompt like with an image.”

Q experience with these tools, I'm curious if you see a parallel skill set, uh, where you can kind of compare prompt engineering to, is it similar to learning to code? Is it similar to being able to storytell effectively? Is it similar to being able to process numbers in an Excel sheet? Like, is there a parallel skill set where it reminds you of You know, something you've done before.

A I think there was an era, I don't know if we're still in it, where there was a certain category of person who could consider themselves like good at Googling stuff. Do you know that there's this thing where you can type in like file type this and find this thing. There's this big debate over whether like, especially in text to image, you know, is it really, is there really like any artistry to it? And I'm familiar that I'm not so sure cause I'm, I'm no artist, but there's definitely something it's always about, it's like discovering an image that's already out there. You've just got to find the words that summon it forth as if you're kind of navigating like an infinite Pinterest of things that haven't quite existed until you manifested them.

AI assessment note: “a certain category of person who could consider themselves like good at Googling stuff”

Q between Excel and Google Sheets, where if you know how to use Excel, you really can drop right into Google Sheets, and it's relatively straightforward. You might have to switch up your shortcuts a little bit or learn one little thing here and there, but for the most part, you can, again, drop from one to the other. Or would you liken them more to learning to speak different languages?

A It's not that different. I think the principles are like very similar, and then the nuances are slightly different. So I think now if you went from like, like Dali to Midjourney, it would be like amazing. And then if you went back in the other direction, you'd be like, it doesn't do what I want, but that's because Midjourney is doing so much of the heavy lifting to help you make something really good. If you are using the tools to create some like very specific effect, Kind of like imagine that I guess, yeah, like a very complicated Excel formula that would not have the exact same output in the other tool because they're trained on like a different set of images. Stable diffusion, I think some five billion and then like a smaller set of like twelve million for the what does nice look like fine tuning that's happened on the top and how they've optimized it. You know, they, they, they have made different like creative decisions there. It's maybe like driving a different car. Like if you, if you like floor the accelerator and different and then Do this in various different cars. Some are gonna take off. Some are gonna trundle along.

AI assessment note: “It's not that different. I think the principles are like very similar”

Q to write and read. These are just kind of fundamental elemental skills that are in everyone's skill sets. They're taught in schools. Where do you sit with that in terms of how you see this progressing? Like, is it worthwhile? You could also position the question as to become an excellent top one percent prompt engineer, or is it like, oh, everyone should kind of have this in their toolbox.

A Well, that depends. I think on the one hand, there's obviously every incentive for the people that make these foundational tools to make prompt engineering, for instance, not A thing because they want everyone to be able to do it. Right. Again, if you look at the most recent model of mid journey, like version four, if you like stuff that would not have been even possible six months ago, you can literally do the thing where you type in, I can't remember your example, but like, I remember because I posted one, someone was arguing about it and I was like, look at this space stuff. I just typed in space duck and it's like this amazing astronaut duck. And he's like, there's no way you just type that in. So I went back and checked and I was like, no, I lied. I actually typed in a really cool space duck. But at the same time, like, with any material, like, artistic or otherwise, if you push things to the boundary, there's always going to be someone that, like, explores everything that's possible, or, like, just iterates, iterates, iterates on something, you know, they're obviously going to explore further on the map of what's possible than someone that isn't, so I don't think it will become, like, this necessary skill that everyone needs to have, but I do think it will become, you know, like, some people are expert wood whittlers or, you know, really good at Animating hair or whatever…

AI assessment note: “I don't think it will become, like, this necessary skill that everyone needs to have”

Q to write and read. These are just kind of fundamental elemental skills that are in everyone's skill sets. They're taught in schools. Where do you sit with that in terms of how you see this progressing? Like, is it worthwhile? You could also position the question as to become an excellent top one percent prompt engineer, or is it like, oh, everyone should kind of have this in their toolbox.

A Well, that depends. I think on the one hand, there's obviously every incentive for the people that make these foundational tools to make prompt engineering, for instance, not A thing because they want everyone to be able to do it. Right. Again, if you look at the most recent model of mid journey, like version four, if you like stuff that would not have been even possible six months ago, you can literally do the thing where you type in, I can't remember your example, but like, I remember because I posted one, someone was arguing about it and I was like, look at this space stuff. I just typed in space duck and it's like this amazing astronaut duck. And he's like, there's no way you just type that in. So I went back and checked and I was like, no, I lied. I actually typed in a really cool space duck. But at the same time, like, with any material, like, artistic or otherwise, if you push things to the boundary, there's always going to be someone that, like, explores everything that's possible, or, like, just iterates, iterates, iterates on something, you know, they're obviously going to explore further on the map of what's possible than someone that isn't, so I don't think it will become, like, this necessary skill that everyone needs to have, but I do think it will become, you know, like, some people are expert wood whittlers or, you know, really good at Animating hair or whatever…

AI assessment note: “I don't think it will become, like, this necessary skill that everyone needs to have”

Q experience with these tools, I'm curious if you see a parallel skill set, uh, where you can kind of compare prompt engineering to, is it similar to learning to code? Is it similar to being able to storytell effectively? Is it similar to being able to process numbers in an Excel sheet? Like, is there a parallel skill set where it reminds you of You know, something you've done before.

A I think there was an era, I don't know if we're still in it, where there was a certain category of person who could consider themselves like good at Googling stuff. Do you know that there's this thing where you can type in like file type this and find this thing. There's this big debate over whether like, especially in text to image, you know, is it really, is there really like any artistry to it? And I'm familiar that I'm not so sure cause I'm, I'm no artist, but there's definitely something it's always about, it's like discovering an image that's already out there. You've just got to find the words that summon it forth as if you're kind of navigating like an infinite Pinterest of things that haven't quite existed until you manifested them.

AI assessment note: “a certain category of person who could consider themselves like good at Googling stuff”

Q when you first wrote your prompt book, you had a requested feature list for Dolly too. And I feel like potentially some of those have been implemented. I haven't cross-referenced them, but are there things that are on your new list of, hey, these tools are great, but they're missing X, Y, Z, or they're lacking in these areas? This would be top of my list to see improved on.

A You know, I think we're going to see like more models come out. I mean, the fact that stable diffusion is kind of open source means that lots of other things are going to be built on top of that. And I think it's going to be really exciting to see some of the directions that people take that in either kind of on a individual sort of prosumer level, people building their own models to create their own stuff. More likely some bigger organizations, you know, training it for like specific purposes. The whole challenge and the whole opportunity, I think at the moment was like, how do you go beyond the text box? How do you go beyond this, like just blank rectangle to create something that is more user friendly, that's more inspiring. That's more how people think, because on the one hand, if you're, if you're not an artist, the ability to describe things with words is definitely a big step forward. But if you think about the next layer, is it still quite hard to describe things with words? Designers, yeah, when they go and work, when they do work for clients, like it's one of their pet peeves because clients are like, they don't like it, but they can't explain why or what they want different. They're like, oh, I want it to be more, do you know what I mean? Like more, and they're like, I don't know, I don't know what that means. Which is basically the position these, you know, AI model…

AI assessment note: “how do you go beyond the text box? How do you go beyond this”

Q between Excel and Google Sheets, where if you know how to use Excel, you really can drop right into Google Sheets, and it's relatively straightforward. You might have to switch up your shortcuts a little bit or learn one little thing here and there, but for the most part, you can, again, drop from one to the other. Or would you liken them more to learning to speak different languages?

A It's not that different. I think the principles are like very similar, and then the nuances are slightly different. So I think now if you went from like, like Dali to Midjourney, it would be like amazing. And then if you went back in the other direction, you'd be like, it doesn't do what I want, but that's because Midjourney is doing so much of the heavy lifting to help you make something really good. If you are using the tools to create some like very specific effect, Kind of like imagine that I guess, yeah, like a very complicated Excel formula that would not have the exact same output in the other tool because they're trained on like a different set of images. Stable diffusion, I think some five billion and then like a smaller set of like twelve million for the what does nice look like fine tuning that's happened on the top and how they've optimized it. You know, they, they, they have made different like creative decisions there. It's maybe like driving a different car. Like if you, if you like floor the accelerator and different and then Do this in various different cars. Some are gonna take off. Some are gonna trundle along.

AI assessment note: “It's not that different. I think the principles are like very similar”

Q interesting about these AIs is that they are a little bit of a black box. You can't understand Every little piece that went from your input to your output. And so you can't like fine tune it in the same ways as maybe some other skills that we've learned in the past. And so how have you learned to actually become a better prompt engineer given that black box nature?

A I mean, I think another aspect is you and I both type in the same thing. It's not going to make the same. The same picture because it kind of starts from this random cloud of noise and your cloud of noise is different to mine. And then it's slowly turning these clouds more and more into something that looks like a orangutan in a tuxedo. So that's really frustrating when you're like testing things because was it good or did you just get lucky or alternatively, if you're not seeing what you're expected, should you just hit it again and again? Like, and then when you see someone else has made something really cool, did they do something really clever or did they just, you know, is it like a Persistence.

AI assessment note: “So that's really frustrating when you're like testing things”

Q when you first wrote your prompt book, you had a requested feature list for Dolly too. And I feel like potentially some of those have been implemented. I haven't cross-referenced them, but are there things that are on your new list of, hey, these tools are great, but they're missing X, Y, Z, or they're lacking in these areas? This would be top of my list to see improved on.

A You know, I think we're going to see like more models come out. I mean, the fact that stable diffusion is kind of open source means that lots of other things are going to be built on top of that. And I think it's going to be really exciting to see some of the directions that people take that in either kind of on a individual sort of prosumer level, people building their own models to create their own stuff. More likely some bigger organizations, you know, training it for like specific purposes. The whole challenge and the whole opportunity, I think at the moment was like, how do you go beyond the text box? How do you go beyond this, like just blank rectangle to create something that is more user friendly, that's more inspiring. That's more how people think, because on the one hand, if you're, if you're not an artist, the ability to describe things with words is definitely a big step forward. But if you think about the next layer, is it still quite hard to describe things with words? Designers, yeah, when they go and work, when they do work for clients, like it's one of their pet peeves because clients are like, they don't like it, but they can't explain why or what they want different. They're like, oh, I want it to be more, do you know what I mean? Like more, and they're like, I don't know, I don't know what that means. Which is basically the position these, you know, AI model…

AI assessment note: “how do you go beyond the text box? How do you go beyond this, like just blank rectangle”

Q interesting about these AIs is that they are a little bit of a black box. You can't understand Every little piece that went from your input to your output. And so you can't like fine tune it in the same ways as maybe some other skills that we've learned in the past. And so how have you learned to actually become a better prompt engineer given that black box nature?

A I mean, I think another aspect is you and I both type in the same thing. It's not going to make the same. The same picture because it kind of starts from this random cloud of noise and your cloud of noise is different to mine. And then it's slowly turning these clouds more and more into something that looks like a orangutan in a tuxedo. So that's really frustrating when you're like testing things because was it good or did you just get lucky or alternatively, if you're not seeing what you're expected, should you just hit it again and again? Like, and then when you see someone else has made something really cool, did they do something really clever or did they just, you know, is it like a Persistence.

AI assessment note: “I think another aspect is you and I both type in the same thing.”

Q blog post sharing images, whether it's creating, you know, the next sneaker design that you end up selling. Are there areas that you've seen really emerge from this where people are using these tools today and applying them to, again, what someone might call a practical use case and In addition to maybe what you've seen so far, are there other areas where you're excited to see this be applied?

A Sure. It's interesting though, isn't it? Because I think especially given the tenor Of the conversation, um, around these tools and the ethical and legal aspects they're in. I suspect that when, uh, to an extent when you see these things used, especially in prominent contexts, they might not be advertised as such. Much is like green screen, right? Like when green screen is used in films, you shouldn't be like, that is an amazing use of green screen. You should just be like, oh my God, like he's dangling off a thing. Oh, this must, this must have cost millions. So I think, you know, when we see AI tools used in lots of contexts, not that it's just covered up, but you know, they might also be just a narrow part of the creative process. I raised this point online, I think that you were making, which is like, well, where's this all going? Like, will it ever make images good enough? And like, do, will other people like want to look at them? Because it's not like we have this huge history of like logging in to social media and looking at just like abstract pictures, like, oh, a horse on a, um, on a surfboard. I mean, things tend to have like a grounding in reality, right? Like that's what makes them viral or interesting. It's like a real thing that happened. But then someone was like, no, like maybe this, it won't be that it's going to make Content so good that it's like better than …

AI assessment note: “when you see these things used, especially in prominent contexts, they might not be advertised”