How to use Weights
For Stable Diffusion
With the AI Art
Deforum Diffusion
Notebook
This is a video that shows how the weights work in prompts
and how you can use them to micro manage your still images and animations.
You can use them to get rid of unwanted elements and also to
balance out your prompt and give certain elements of the prompt more
input than others.
Hello everybody and welcome to my Tutorial here on prompt weights and this Is going to be a pretty in-depth Tutorial or guide whatever you want to Call it just because I feel that prompt Weights I think a lot of people don’t Really use them to their full extent but They are very extremely useful for kind Of fine-tuning your prompts now I did do A tutorial on prompt weights for disco Diffusion but for stable diffusion and The deformed Network here notebook there Is a slightly different format I believe So what I have here first thing we’re Going to do is we’re going to turn on Everything is well I don’t know if it’s Default but let me just go over some of My settings so I have animation mode on None and I am just right now using the 1.4 stable diffusion model I haven’t got The 1.5 yet but I will probably do a Guide on that soon as well there is a 1.5 out now we can use that has been Released and I’ve also been really Getting into making my own kind of Customized models and I’m going to Release one to everybody here very soon I’m working on my next one will be Werewolves or space or something but Let’s get back to this one okay so Prompt weights here To do this we first need to enable Prompt weights which will be down here In our in our run settings
And so I have this checked and then this Has to be turned on if this isn’t turned On it will get really ugly let me just Kind of show you really quick Let me go ahead and just set these all To one So I have a blue square one a red circle One a yellow triangle one And what this will do is it’ll give them All equal amounts of input into the Image generation although I think this Diffusion I think almost all diffusion Is kind of partial to the red color for Some reason red seems to overpower Everything else and that even goes way Back to disco diffusion if you’d run a Video a lot of times it would kind of Turn totally red so we’ll probably get Mostly red okay so now you’ll see this Is just really ugly this is a mess now This is we can fix this by doing this we Just click normalize prompt weights And now we should get something more Coherent so I don’t know if there’s a Way to leave this unchecked and use it I Just always check it so if you’re using Prompt ways just always leave that Checked and then I will show you what This one does And I always have that on too just Because it kind of shows you what it’s Doing okay and there is our video you’ll See a blue square one a red circle one a Yellow triangle one so it’s giving all
Those Parts of the prompt equal say in the Image generation and like I said red Does seem to overpower everything for Some reason Let me go ahead and run this one more Time I’m going to show you how we can Use this this is really good especially If you haven’t used weights before I Would turn this on and it will kind of Show you How it’s Distributing that prompt So if you notice a blue square has a Weight of 0.33 a red circle 0.33 a Yellow triangle 0.33 so these numbers That we’re inning are really just Relative to each other so this would be The exact same thing as if I put two in All of them it would still Be distributed each one would have one Third of the total value so I’ll go Ahead and do that And we should get an image that is Identical or very similar I’m using the Same seed So this will be the same in math terms This will be the exact same as entering Them all as one if you notice now it’s 0.33.33.33 So really these prompt weights they are Just in Rel in relation to each other When you put those prompt weights notice This is the same image so a lot of People I think mostly use the prompt
Weights because they want to get rid of You know maybe trees or hands like you Could put hands minus one and it does Work for that a bit but I really think The best most accurate use of it is just To kind of balance out what you already Have in your prompt so let’s go ahead Now I’m going to go ahead now and put Change these values a little bit this is Just kind of an introduction to show you What the prom voids are so now I just Have a blue square one and a zero on the Other ones And we’ll go ahead and run this okay and Like I said this is going to be kind of A deep dive so if you want to go grab a Snack you know or something this might Be a long video I’m not sure okay so now We have a blue square And that’s quite different from my other Ones and let’s go ahead I’m just going To do the same thing here So we’re just going to kind of go over The basic stuff here first and then we Will get into the fun more complicated Stuff as the video goes on so now we Have a red circle with a value of one And the other two with a value of zero And then we’ll do the yellow triangle Last And then like like for this you wouldn’t Really enter a prompt like this normally Because if you didn’t want a blue square Yellow triangle in it you just wouldn’t
Put them in the prompt so I’m just kind Of showing you how this works though so Now we have there is our red circle And now we will do our yellow triangle Just to complete this part Okay and if you notice now it’s showing Us the weight so now it’s showing us Yellow triangle is a weight of one so The yellow triangle is one hundred Percent of the prompt it’s using it is Ignoring the other ones And there we go yellow triangle so now What we can do also We can go ahead and crank this up to Five and I’ll go ahead and put a one on Each of these Actually just to make it Easier for me to do the math here I’m Going to put eight And a one and a one so eighty percent of The image will be yellow triangle ten Percent square and 10 red circle so all These values will always add up to a Total of one hundred percent of the Prompt So it doesn’t matter what numbers you Put in there what it’s going to do is Just balance them against each other And give each weight a percentage of the Total prompt so with this we have a Total of nine or excuse me ten for a Value we’ve got 8 9 10. And so the blue square is going to get 10 the red circle is going to be 10
Percent And then the other eighty percent is Going to go towards the yellow triangle And there we go And now let’s split it up between The Blue Square And the yellow triangle will give both Of those A4 and we’ll give the red Circle A2 So this should be mostly yellow triangle And blue square with just a little bit Of a red circle But again like I said it does seem like The red color kind of overpower so there We got our weight 0.4 The blue square red circle point two and Yellow triangle 0.4 Okay and there we go so our Yeah a red circle we’re not seeing much Circles in there we’re seeing the blue And the yellow but we still do see a lot Of red but that is how the weights work So let me go ahead now and we’ll get to Using this in a more functional way okay Now I want to point something out too if You’re familiar with um Disco diffusion If you had a equation like this where it Says a force by Lake one trees minus one Um what would happen is you’d get an Error because 1 minus 1 is 0 but this Apparently you can do that you can have The total equals zero in this What will happen is it will just keep The one and it should just not generate
The trees yeah so the um so all the way To 100 of the weight is just going Towards the forest by leg so you don’t Have to worry about the numbers adding Up to one here And also want to show like um some People want to use these prompts to like Get rid of an item so it it can do that But it’s a lot more useful I think it’s More accurate at just balancing out What’s already in your prompt so let’s Go ahead now and try to do trees minus 10. let’s say we don’t want any trees in Our image or hands you know if you want To make something you can put a negative Weight in there and it should stop that From rendering so I don’t know if that’s If that Forest is going to counteract That that’s what I kind of wanted to Test here Because normally a forest is going to Have trees but we’ve got trees at -10 Now it’s just going to do the same thing So it’s just gonna yeah it’s just gonna So this is why that’s really not a Really good use for it Yes so we’re still getting trees so that That part of that now forest does have Trees in it but that is Um probably one of the issues there with Trying to use it to just get rid of Something So it doesn’t work as well with that so Let me show you a little more practical
Use so then like I said it can do that Let’s um Let’s just do this let’s do a mountain Lake we’ll take out Forest And maybe this can be a way we can get Rid of trees So let’s try the same one I just Replaced for us with mountain S usually have trees so we’ll see And the minus 10 that I did is not any Different than just minus one Because you can see it’s just it’s Either way it’s okay so there we don’t Have any trees there let’s go ahead and I do have the same seed here so let’s go Ahead and I’m just going to put zero instead of a Negative weight so let’s see if there’s Any difference Between 0 and negative one I don’t think So but we’ll find out here Okay well there it did work it did work It got rid of the trees in the one so it Can work but it’s it’s just a little Spotty but there we see that it did Actually work in this case so you can Use it as a Negative way to get rid of elements that Might normally be in your prompt but if You notice when I had a lake by a forest Since forests have trees that overrode The trees minus one so if you do have Anything in there that might Um generate the same thing really
Specifically like a mountain doesn’t Have to have trees but a lot of them do So I guess that’s the difference here But let’s go ahead and do this one more Time So now it should have a mountain with no Trees because it is giving that negative One to the trees there where when we had Zero it just basically ignored it and Just didn’t um it didn’t have that in The prompt the the part about the um Weight minus one of the trees if you’ll Notice it just left it out so that is The difference so you can use it to Remove unwanted elements So we’re going to do this one more time Here with the trees at zero and mountain Lake at one and this time if you’ll Notice it doesn’t have that weight minus One for the trees the only thing it’s Doing now is a mountain lake So there is a difference between zero And negative one for the prompt weight And there we go now we’ve got trees Again okay and for the next part here I’m going to show you how to use it in a Bit more of a practical method and we’ll Keep getting more functional as this Tutorial goes on so what I have here is I have a cottage by a lake That has a weight of one then I have by Thomas Kinkade with the weight of one Then I have two other artists here by Sonia Delaney zero and Hugh Farah zero
Now I tried to pick three artists with Very different styles of Thomas Kinkade Kind of does bright really colorful Um pictures of cottages and things and You’ll you’ll probably recognize his Style with really bright colors Um Sonia Delaney I actually haven’t used Her but she’s just an abstract artists So her art was mostly just shapes and Things and then Hugh Ferris had black And white pictures of buildings so I Tried to pick three very different Artists here so right now in this first One I have Thomas Kinkade with one and The other artists with zero and we’ll go Ahead and run this And there it’s showing us the weights so It’s just doing Cottage By Lake 50 Thomas Kincaid 50 percent And we’ll kind of do this with all three Just so we can see the difference and There is a Thomas Kinkade picture notice The bright colors The Shining lights And now we’ll go ahead and We will put the next artist in there And this is Sonia Delaney this is the Sort of abstract artist With colored shapes And I’m hoping this doesn’t even really Give us a coherent Cottage I’d like I Said I wanted three really different Kind of contrasting Styles we’ll see What this looks like okay that actually Looks kind of nice okay but it’s it’s
Definitely a different style we can tell It’s a lot kind of uh Not as much detail I guess we can see Okay now let’s do this last one Hopefully this one will be black and White Okay and now we have by Hugh Ferris and Again this isn’t how we would use it Normally if we wanted a zero we would Just not put them in there but I’m just Kind of doing this to illustrate how we Can use these weights to kind of Partially get an artist effect in here So we’ll do that next I’ll show you that Next right now I just want to show you The three Styles hopefully these Contrast enough okay there we go that One’s definitely black and white it’s Still kind of realistic Yeah it actually looks pretty realistic But it is black and white okay so what You can do here let’s say we want mostly Thomas Kincaid image And then we just want a little bit of Sonia and a little bit of hue so I’m Putting Thomas Kincaid four And then we can also crank up our Cottage by a lake to give it a little More prominence there to a three so we Got there three seven eight nine let’s Go ahead and let’s just put the cottage By like with four this will give us some Nice even numbers So forty percent of the prompt is going
To be Cottage by a lake 40 percent of the prompt is going to be By Thomas Kinkade and then 10 percent to Sonia and 10 percent to Hue And we can see that down there there’s Our 0.4 for cottage by Lake Point four For by Thomas Kincaid and then 10 to the Other ones Okay and there we go we can see that is Mostly still the Thomas Kinkade looking Image And now let’s go ahead and just switch These values around a bit We’ll give Sony the majority of it this Time And so this is a way you can just put in Like if you don’t fully want the effect Of part of your prompt like Um I’ll I’ll do it next but other ones Can be modifiers like volumetric Lighting and that kind of thing Okay and now we’ve got much more of that And then let’s go ahead and do the same Thing here To hube and then he should be the Predominant force in this image So it’s really just a percentage just Know when you’re doing these with the Weights the numbers don’t really have Any value except to each other it’s just A relation between the other things in Your problem so it doesn’t matter if you Put You know a four there or 400 100 100 400
Is still going to break it down into a Number of one and each number is just Going to give it that percentage of that Whole number and there we go so now that One is mostly the black and white so That is how the prompts work Now let me show you one other thing here Let me show you what happens if we don’t Put a weight here so I’m going to put a 2 by Thomas Kinkade there and I’m going To go ahead and crank this one up to six And I’m not going to put a weight by Sonia Delani at all Now if you notice What it did is it gave Sonia Delaney and Hugh Ferris a weight of 0.5 and Tom Kincaid just to 0.17 So what it did and if you notice I put a 6 there though I didn’t put a Well that’s the total weight actually Let me do this another way that’s Get this so the math is a little easier To do there we go two two okay Okay so I’ve got Cottage By Lakewood to Thomas Kincaid with two and then Sonya Delaney I didn’t put anything there I Just put a comma and then by Hugh Ferris For six Okay but if you’ll notice it took um the Sonia Delaney and just kind of mixed her In there with Hugh Ferris for 0.6 so Right now it’s breaking That from This prompt over to the next one is just
Giving that to this one So let’s do this let’s do I’m going to put another artist in there HR Geiger who’s also black and white And I believe this will do the same Thing I believe then this point this six It’ll assign it to everything to the two I just want to double check that yeah So if you if you just want to save some Time and you want the same weights in Something you can speed it up a bit by Doing that now we’ve got a very Different looking image again so now let Me get into we’re going to get into some Even more functional stuff in ways we Can just give us more power more ways to Micromanage our image okay so what I’ve Done here is I’ve added some pretty Common modifiers here that you’ll see a Lot so I’ve got a cottage by a lake art Station deviantART volumetric lighting Matte painting concept art Unreal Engine By Thomas Kinkade and right now I have Everything with just the equal weight a Weight of one so I could just do this Without the weights this would basically Be like not having any weights because It’s just going to degenerate everything Equally If you know it’s just giving a 0.12 to Everything Okay and here’s our final image there With all those having equal weights so What you can do you know let’s say okay
I like this image but maybe I want it to Look Um more 3D I want to have more of that Unreal Engine effect so you can take That part of the prompt here And we’ll Crank that part up to four Okay now we’ve got a much more kind of Almost just a video game looking thing There because that Unreal Engine so this Is a really way you can just like get in There and just micromanage everything so What I’ll usually do What a good thing to do here is you want To take the main parts of your prompt Which is a cottage by a lake that’s our Main one and then we want the artist to Have quite a bit of say too you want to Make it a Thomas Kincaid picture so We’ll give both of those a four and then We’ll leave these other modifiers here Like art station deviantART etc etc We’re just going to give those a weight Of one So you’ll notice even without changing The prompt just by using the weights we Can get a host of different looking Images here so right now the Thomas Kinkade is 0.29 the cottage by a lake so That’s about 60 percent of the image That is going to be a cottage by leg by Thomas Kinkade then the rest stuff the Modifiers don’t have as much say in the Image generation And let’s do something let me go ahead
And just take out the arch station and DeviantART And I’m replaced with something A little more noticeable I’m going to put Glock house fog and Mist and I’m going to crank this up a Bit Okay so now our fog is 25 of the image So we can use the weights for is just Like I said to micromanage it and Control exactly how much fog you get in Your image So we’ll focus on that for this one Okay there we go and I’m not really Noticing any fog there so let’s go ahead Let’s Crank that up even more Let’s just go crazy let’s put it up to 20. And Glock house is kind of a sort of a Light blue almost Gray Color okay there we go that should be 60 Of the image now that glucose fog and Mist there we go now we’re seeing some More fog so we can use that to just Really control certain elements of our Prompt so this one I’ve got a noise Schedule I just .02 to the whole thing And I’ve turned down the string schedule Those changes or I’ve turned it up from 0.4 up to 0.5 for the changes so we’ll Still lower a bit from the 0.55 but not Quite as much And then I’m going to leave the Cadence Setting where it is let’s see how many
Frames we got here Yep that’ll work okay so now what I’ve Done is I’ve replaced the different Artists with summer flowers winter snow And fall leaves so we’ve got a steady Value of a cottage by a lake of three And by Thomas kincaida three in all our Prompts And in our first prompt we have a summer Value or summer flowers four from Keyframe 0 to 50 and we have zero and Zero for winter snow Then at keyframe 51 we have a negative Prompt summer flowers minus one so it’s Going to erase those flowers and then It’s going to start making fall leaves Of four And then once it’s done there we can Just put a zero so I’ve got summer Flower zero and then our winter snow Goes up to four and our fall leaves We’ve got a minus one so what this Should do is start off our animation With summer flowers then turn into fall And then winter so let’s go ahead and do This I did a lot of these it kind of Animations for disco diffusion you can Also use the same principle to age a Person like we could have a young a Young boy a teenager A man and an old man I’ve done some Videos like that to kind of time lapse Videos that are kind of neat so let’s go Ahead and run this
Okay and I just started and you see Right now we’ve got a cottage by Lake by Thomas Kincaid is 60 percent And then our summer flowers is the 40 Percent So everything’s right on schedule now Okay we’re approaching keyframe 50 now So it should start getting rid of those Summer flowers And start making fall leaves And we actually have a negative prompt On there which will help to kind of get Rid of them as well that’s shown right There negative one And we should start getting some more Fall colors here And it should continue to erase those Flowers Yep there we go so now we’re seeing more Kind of leaves on trees And again we’re keeping the same artist The same prompt is still um Cottage by a Lake by Thomas Kinkade And then at keyframe 100 it’ll start Getting rid of the leaves and start Adding snow Okay we’re approaching keyframe 100 now So that’s one reason why I always really Liked using weights in Disco diffusion For animations because it really kind of Adds a storing a storytelling kind of Element to your videos So here we hit 100 so now we should Start seeing some winter snow
Those fall leaves should start Disappearing yeah winter snow should be Coming in Okay and there we go we are for some Reason still get there is snow going but We are still getting those summer Flowers for some reason so maybe I Should I’ll run this one more and I’ll Go ahead and put a negative weight on Those as well After all that’s the whole purpose of Our weights here is to control Our animation here how we want it Yeah so we are still getting those Summer flowers I’m not sure why it’s at A zero let’s go ahead and do that Let’s just keep this at Negative one through the whole thing Which I didn’t think would be necessary But it looks like it is and Yeah and then actually let’s go ahead And crank that winter snow up to six Let’s really you know what let’s no We’re gonna do ten let’s just make it a Blizzard here at the end let’s make it More dramatic let’s actually even do a Little bit of the same with the fall Leaves here And I think let’s also turn up the Cottage by a lake a bit And Tom’s Kincaid down a bit so we’ll Get our subject a little more in there And this is why I kind of do these Videos like I do where I’m just actually
Recording what I’m doing Because it really helps to kind of just Actually see you know real-time Adjustments and things you can do So I like to leave my mistakes and Errors or Corrections like I’m doing now In so let’s go ahead and do this again I think we can leave all that the same Yep okay let’s do one more try here Let’s see if we can get a little more Dramatic changes okay it’s starting here We still haven’t got a cottage but it is Cottage By lake is cranked up there we Don’t got a lake either that’s kind of Odd Considering that is That should be where the big yeah Cottage By lake is 40 so we should be Seeing that summer flowers is also 40 Percent Well let’s go let’s go modify it again Let’s go ahead and turn that up to five Oh you know what let’s give this a big Chunk of it let’s do six This is after all what the weights are For is to tell it how we want our Image to look Okay it’s starting up now we can even See a little cottage peak in there in The background and then our Lake should Probably disappear a bit because I don’t Have it at um that high through the Whole thing Again I want to kind of emphasize the
Changes or I do have it that high but I’ve got the the fall leaves are bigger Than the summer flowers there to eight And the winter snows that attend so It should really emphasize those season Changes on this one Okay and here comes frame 50 where the Summer flower should start disappearing And we should start seeing fall leaves Yep there we go we can definitely see Those fall colors now okay and we are Approaching keyframe 100 now it should Start turning to Winter Okay and so you can see how this using Weights really kind of adds kind of a Storytelling element like I also could Have put Um frozen lake in there you know or Maybe have some bears come into the Scene things like that have things come And go so the weights are really a Really good way to kind of micromanage Not just your still images but your Animations as well As you can see we used the same prompt Here all we changed in these keyframes Is just the weight of certain parts of The prompt And so that also makes it really easy to You know just copy and paste over and Over again So this is about done we’ll go ahead and Render the video here like I said a lot Of people really like these kind of
Videos when I did them with disco Diffusion let’s go ahead And I think this will be better than That last one we did And should render pretty quickly too Since it’s only 150 frames there we go All right let’s go ahead and look at it Here Here we’ve got summer Now it’s changing into fall And then we’ve got winter so I hope you Like this tutorial and again I will be Focusing on models next I’m actually Making my own models now and I’m having A lot of fun with that I am going to Post a model here for everybody I’ve Been making Um my first two models I made for my Patrons only because I I had a week or Two this month where I wasn’t very Active I had some family stuff I had to Take care of and that is all done now so I am going to continue to making models For my patreon page but I also am going To start posting some public ones for Everybody just say thank you for Subscribing the channel and I just Really like making them so I’ll get into That next and models next but I hope That this weights tutorial kind of Really showed you how the weights work And the format and how to use them Because you can really do some great Things with them it’s really lets you
Like I said just micromanage I know I’ve Used that word a lot but that’s what I Like to do I really like to micromanage My images and videos and weights are a Great tool to do that so thank you for Watching and I will be back soon with More content I am going to get busy Again here with the videos and tutorials You all have a great rest of your day