Today we’re going to flip the script and find out how AI image generators see us and the world around us. To do this, we’re going to hack a bit of code called CLIP interrogator by @pharmapsychotic
In many ways the AI describes the world as we would. But it can’t help but throw in editorial comments and a snide barb or two.
Clip Interrogator
[Music] Hi I’m Jen with making the photo today We’re going to flip the script a little And find out how AI sees us and the World around us to do this we’re going To hack a bit of code called clip Interrogator by Pharma psychotic Let’s see what’s in the mind of an AI Image generator clip interrogator is a Free program so throw anything you want At it as long as you can take the heat I Use clip interrogator in the video Reversing AI image prompts when I was Trying to recreate the image teatra Opera spaciel by Jason Allen give clip Interrogator an AI image and it tells You what the prompt might have been this Is what clip interrogator was designed To do clip interrogator gave me some Suggestions and I was able to reproduce Elements of the target image it didn’t Let me exactly reproduce the image but It did give me some artists and text Descriptions to try I don’t know how Close clip interrogator got to the Original prompt so let’s try it with a Prompt that I know this is an elephant In a tiny wicker basket that I created On Dali let’s see how close clip Interrogator gets the AI got the idea of An elephant in a hot air balloon and the Movie Up seems to apply but I don’t know This artist okay I can see the Similarities especially in the color and
Testing Differences
The Whimsy of the image one of the great Features of clip interrogator is that it Suggests artists using an artist’s name Is an easy way to communicate a style to AI but when I feed the prompt back into Dolly I get a very different kind of Image the elephant is in the basket but The images were almost monochrome I’m Wondering if it’s the phrase ambient Occlusion yep that seems to be it Dolly’s picking up on the phrase ambient Occlusion a lighting effect and waiting It stronger in the image running the Prompt through mid-journey makes the Overall style closer to the mark But mid Journey seems to blank on the elephant In the hot air balloon clip interrogator Is a tool for crafting AI image prompts But is that all it can do let’s feed a Photo into the AI and see how it Describes the real world I don’t have Any elephants in hot air balloons but I Do have plenty of hot air balloon photos Okay the first thing is that it didn’t Recognize this as a photo but I have Been told that my photography style is Painterly so that’s probably okay it got The subject just fine there’s some Gibberish in the prompt the description Romantic simple path traced is a bit Weird or maybe it’s just poetic but I’m Beginning to suspect that the AI is just Throwing up random artists I looked up This artist and I see no connection at
Objects
All I sent this prompt through an AI Generator to see if it understood the Description of course it’s not an exact Rendering of the photo but at least it’s In the right Ballpark and mid-journey Came up with an even better rendering Okay maybe I’ll take back what I said Maybe the AI does know what it’s talking About let’s try a more mainstream Landscape photo this one taken in the Colorado Rockies initially clip Interrogator returned an error but That’s probably because I still had my Watermark on the photo the AI returned a Bare Bones sort of description and it Didn’t even ID this as a photo I’m not Sure what a carbide lamp has to do with Anything think but okay I was more Concerned with who this Darren Bader guy Was I’m not seeing the resemblance to His digital images myself let’s see how The AI describes Landscapes from the Master himself Ansel Adams the AI Spotted Ansel Adams immediately no Problem I’m not sure Bruce Davidson has Much to do with anything and the aspect Ratio is wrong but otherwise a fair Enough description so far so good on the Nature photos AI seems to see the world Much as we do but take the artist that The clip interrogator throws out with a Grain or a ton of salt this is a Pity Because this is an area where many of us Need the most help and you can see
How AI Learns
Evidence of where the AI is getting its Learning by mentioning Shutterstock and Unsplash I ran a bunch of images from Wiki commas through the AI and for the Most part it had no problem iding the Subject and often it id’d the photo as a Wiki Commons photo probably accessing Metadata we can’t see but I did manage To stump up interrogator a couple of Times it just processed and processed And processed I guess this is why catch Bugs still work let’s be a little mean And give the AI something more difficult Even for humans a classic optical Illusion what does the AIC okay I’m not Going to catch it out quite that easily It recognized this image as an optical Illusion it mentioned a woman and an old Woman but the AI didn’t just describe What it sees the AI made a value Judgment it’s not just a cartoon but a Silly cartoon It also says this is what Stable diffusion looks like that’s a bit Deep let’s see if stable diffusion Agrees nope stable diffusion had other Ideas but we’re starting to get a look At what the AI really thinks about what It sees lately people have been feeding Clip interrogators selfies and getting Roasted I have my headshot handy let’s See how the AI sees me okay I thought I Was hiding my gray hair better but it is A profile pic and I am smiling I’m not Sure where they all natural raw comes
From or the artist but if I feed the Prompt back into Dali I do get a Headshot the AI was pretty good Describing street photography as well Lots of the description is correct and It identified that the woman was wearing A sari even if it didn’t identify the Place as India but like humans the AI Can get the wrong end of the stick and Misinterpret what it’s looking at this Iconic photo by Henri Cartier brassonne For instance nope it’s not looking at Trains or explosions and the AI didn’t Recognize this iconic photo of Marilyn Monroe but it did mention a different Marilyn whose art is as colorful and Impressionistic as it comes very unlike This image the AI recognized the photo As funny but also an exploitable image The AI could be right the AI may not Have recognized Marilyn Monroe but it Had no problem with The Rock he sure has The AI fooled though age 35 the rock is At least 50. either that or the AI is a Suck up from the descriptions I know the AI is looking at image sites like Unslash let’s pull a couple of selfies Up and see what the AI sees Yep this is A woman taking a selfie but leave it to The AI to come up with some Tick-Tock Slang sassy Baka translates roughly to Suspicious full it’s an insult but is The AI wrong the AI sees this image as Generally sweet and wholesome the only
Barb is the insult wrapped up in the Word gelatinous it you have to mention That never ask an AI if you look fat in This outfit it has no social filter I Threw a bunch more photos from unsplash At the AI the AI seems to describe the World much as we do but it did have a Funny way of tossing in an insult or Editorial comment or two along the way And who taught the AI to swear If you’re brave enough submit images of Yourself into clip interrogator and see How the AI sees you if this video was Helpful remember to like the video and Subscribe to the channel this is Jen at Making the photo let’s make something Amazing together [Music]