Features
This is a feature showcase page for Stable Diffusion web UI.
All examples are non-cherrypicked unless specified otherwise.
Stable Diffusion 2.0
Basic models
Models are supported: 768-v-ema.ckpt (model, config) and 512-base-ema.ckpt (model, config). 2.1 checkpoints should also work.
- download the checkpoint (from here: https://huggingface.co/stabilityai/stable-diffusion-2)
- put it into models/Stable-Diffusion directory
- grab the config from SD2.0 repository and put it into same place as the checkpoint, renaming it to have same filename (i.e. if your checkpoint is named
768-v-ema.ckpt
, the config should be named768-v-ema.yaml
) - select the new checkpoint from the UI
Train tab will most likely be broken for the 2.0 models.
If 2.0 or 2.1 is generating black images, enable full precision with --no-half
or try using the --xformers
optimization.
Note: SD 2.0 and 2.1 are more sensitive to FP16 numerical instability (as noted by themselves here) due to their new cross attention module.
On fp16: comment to enable, in webui-user.bat:
@echo off
set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=your command line options
set STABLE_DIFFUSION_COMMIT_HASH="c12d960d1ee4f9134c2516862ef991ec52d3f59e"
set ATTN_PRECISION=fp16
call webui.bat
Depth-guided model
- download the 512-depth-ema.ckpt checkpoint
- place it in models/Stable-diffusion
- grab the config and place it in the same folder as the checkpoint
- rename the config to
512-depth-ema.yaml
- select the new checkpoint from the UI
The depth-guided model will only work in img2img tab.
Outpainting
Outpainting extends the original image and inpaints the created empty space.
Example:
Original | Outpainting | Outpainting again |
---|---|---|
Original image by Anonymous user from 4chan. Thank you, Anonymous user.
You can find the feature in the img2img tab at the bottom, under Script -> Poor man’s outpainting.
Outpainting, unlike normal image generation, seems to profit very much from large step count. A recipe for a good outpainting is a good prompt that matches the picture, sliders for denoising and CFG scale set to max, and step count of 50 to 100 with Euler ancestral or DPM2 ancestral samplers.
81 steps, Euler A | 30 steps, Euler A | 10 steps, Euler A | 80 steps, Euler A |
---|---|---|---|
Inpainting
In img2img tab, draw a mask over a part of the image, and that part will be in-painted.
Options for inpainting:
- draw a mask yourself in the web editor
- erase a part of the picture in an external editor and upload a transparent picture. Any even slightly transparent areas will become part of the mask. Be aware that some editors save completely transparent areas as black by default.
- change mode (to the bottom right of the picture) to “Upload mask” and choose a separate black and white image for the mask (white=inpaint).
Inpainting model
RunwayML has trained an additional model specifically designed for inpainting. This model accepts additional inputs - the initial image without noise plus the mask - and seems to be much better at the job.
Download and extra info for the model is here: https://github.com/runwayml/stable-diffusion#inpainting-with-stable-diffusion
To use the model, you must rename the checkpoint so that its filename ends in inpainting.ckpt
, for example, 1.5-inpainting.ckpt
.
After that just select the checkpoint as you’d usually select any checkpoint and you’re good to go.
Masked content
The masked content field determines content is placed to put into the masked regions before they are inpainted.
mask | fill | original | latent noise | latent nothing |
---|---|---|---|---|
Inpaint at full resolution
Normally, inpainting resizes the image to the target resolution specified in the UI. With Inpaint at full resolution enabled, only the masked region is resized, and after processing it is pasted back to the original picture. This allows you to work with large pictures and allows you to render the inpainted object at a much larger resolution.
Input | Inpaint normal | Inpaint at whole resolution |
---|---|---|
Masking mode
There are two options for masked mode:
- Inpaint masked - the region under the mask is inpainted
- Inpaint not masked - under the mask is unchanged, everything else is inpainted
Alpha mask
Input | Output |
---|---|
Prompt matrix
Separate multiple prompts using the |
character, and the system will produce an image for every combination of them.
For example, if you use a busy city street in a modern city|illustration|cinematic lighting
prompt, there are four combinations possible (first part of the prompt is always kept):
a busy city street in a modern city
a busy city street in a modern city, illustration
a busy city street in a modern city, cinematic lighting
a busy city street in a modern city, illustration, cinematic lighting
Four images will be produced, in this order, all with the same seed and each with a corresponding prompt:
Another example, this time with 5 prompts and 16 variations:
You can find the feature at the bottom, under Script -> Prompt matrix.
Color Sketch
Basic coloring tool for img2img. To use this feature in img2img, enable with --gradio-img2img-tool color-sketch
in commandline args. To use this feature in inpainting mode, enable with --gradio-inpaint-tool color-sketch
. Chromium-based browsers support a dropper tool. (see picture)
Stable Diffusion upscale
Upscale image using RealESRGAN/ESRGAN and then go through tiles of the result, improving them with img2img. It also has an option to let you do the upscaling part yourself in an external program, and just go through tiles with img2img.
Original idea by: https://github.com/jquesnelle/txt2imghd. This is an independent implementation.
To use this feature, select SD upscale from the scripts dropdown selection
(img2img tab).
The input image will be upscaled to twice the original width and height, and UI’s width and height sliders specify the size of individual tiles. Because of overlap, the size of the tile can be very important: 512x512 image needs nine 512x512 tiles (because of overlap), but only four 640x640 tiles.
Recommended parameters for upscaling:
- Sampling method: Euler a
- Denoising strength: 0.2, can go up to 0.4 if you feel adventurous
Original | RealESRGAN | Topaz Gigapixel | SD upscale |
---|---|---|---|
Attention/emphasis
Using ()
in the prompt increases the model’s attention to enclosed words, and []
decreases it. You can combine multiple modifiers:
Cheat sheet:
-
a (word)
- increase attention toword
by a factor of 1.1 -
a ((word))
- increase attention toword
by a factor of 1.21 (= 1.1 * 1.1) -
a [word]
- decrease attention toword
by a factor of 1.1 -
a (word:1.5)
- increase attention toword
by a factor of 1.5 -
a (word:0.25)
- decrease attention toword
by a factor of 4 (= 1 / 0.25) -
a \(word\)
- use literal()
characters in prompt
With ()
, a weight can be specified like this: (text:1.4)
. If the weight is not specified, it is assumed to be 1.1. Specifying weight only works with ()
not with []
.
If you want to use any of the literal ()[]
characters in the prompt, use the backslash to escape them: anime_\(character\)
.
On 2022-09-29, a new implementation was added that supports escape characters and numerical weights. A downside of the new implementation is that the old one was not perfect and sometimes ate characters: “a (((farm))), daytime”, for example, would become “a farm daytime” without the comma. This behavior is not shared by the new implementation which preserves all text correctly, and this means that your saved seeds may produce different pictures. For now, there is an option in settings to use the old implementation.
NAI uses my implementation from before 2022-09-29, except they have 1.05 as the multiplier and use {}
instead of ()
. So the conversion applies:
- their
{word}
= our(word:1.05)
- their `` = our
(word:1.1025)
- their
[word]
= our(word:0.952)
(0.952 = 1/1.05) - their
[[word]]
= our(word:0.907)
(0.907 = 1/1.05/1.05)
Loopback
Selecting the loopback script in img2img allows you to automatically feed output image as input for the next batch. Equivalent to saving output image, and replacing the input image with it. Batch count setting controls how many iterations of this you get.
Usually, when doing this, you would choose one of many images for the next iteration yourself, so the usefulness of this feature may be questionable, but I’ve managed to get some very nice outputs with it that I wasn’t able to get otherwise.
Example: (cherrypicked result)
Original image by Anonymous user from 4chan. Thank you, Anonymous user.
X/Y plot
Creates a grid of images with varying parameters. Select which parameters should be shared by rows and columns using X type and Y type fields, and input those parameters separated by comma into X values/Y values fields. For integer, and floating point numbers, and ranges are supported. Examples:
- Simple ranges:
-
1-5
= 1, 2, 3, 4, 5
-
- Ranges with increment in bracket:
-
1-5 (+2)
= 1, 3, 5 -
10-5 (-3)
= 10, 7 -
1-3 (+0.5)
= 1, 1.5, 2, 2.5, 3
-
- Ranges with the count in square brackets:
-
1-10 [5]
= 1, 3, 5, 7, 10 -
0.0-1.0 [6]
= 0.0, 0.2, 0.4, 0.6, 0.8, 1.0
-
Here are the settings that create the graph above:
Prompt S/R
Prompt S/R is one of more difficult to understand modes of operation for X/Y Plot. S/R stands for search/replace, and that’s what it does - you input a list of words or phrases, it takes the first from the list and treats it as keyword, and replaces all instances of that keyword with other entries from the list.
For example, with prompt a man holding an apple, 8k clean
, and Prompt S/R an apple, a watermelon, a gun
you will get three prompts:
a man holding an apple, 8k clean
a man holding a watermelon, 8k clean
a man holding a gun, 8k clean
The list uses the same syntax as a line in a CSV file, so if you want to include commas into your entries you have to put text in quotes and make sure there is no space between quotes and separating commas:
-
darkness, light, green, heat
- 4 items -darkness
,light
,green
,heat
-
darkness, "light, green", heat
- WRONG - 4 items -darkness
,"light
,green"
,heat
-
darkness,"light, green",heat
- RIGHT - 3 items -darkness
,light, green
,heat
Textual Inversion
Short explanation: place your embeddings into the embeddings
directory, and use the filename in the prompt.
Long explanation: Textual Inversion
Resizing
There are three options for resizing input images in img2img mode:
- Just resize - simply resizes the source image to the target resolution, resulting in an incorrect aspect ratio
- Crop and resize - resize source image preserving aspect ratio so that entirety of target resolution is occupied by it, and crop parts that stick out
- Resize and fill - resize source image preserving aspect ratio so that it entirely fits target resolution, and fill empty space by rows/columns from the source image
Example:
Sampling method selection
Pick out of multiple sampling methods for txt2img:
Seed resize
This function allows you to generate images from known seeds at different resolutions. Normally, when you change resolution, the image changes entirely, even if you keep all other parameters including seed. With seed resizing you specify the resolution of the original image, and the model will very likely produce something looking very similar to it, even at a different resolution. In the example below, the leftmost picture is 512x512, and others are produced with exact same parameters but with larger vertical resolution.
Info | Image |
---|---|
Seed resize not enabled | |
Seed resized from 512x512 |
Ancestral samplers are a little worse at this than the rest.
You can find this feature by clicking the “Extra” checkbox near the seed.
Variations
A Variation strength slider and Variation seed field allow you to specify how much the existing picture should be altered to look like a different one. At maximum strength, you will get pictures with the Variation seed, at minimum - pictures with the original Seed (except for when using ancestral samplers).
You can find this feature by clicking the “Extra” checkbox near the seed.
Styles
Press the “Save prompt as style” button to write your current prompt to styles.csv, the file with a collection of styles. A dropbox to the right of the prompt will allow you to choose any style out of previously saved, and automatically append it to your input. To delete a style, manually delete it from styles.csv and restart the program.
if you use the special string {prompt}
in your style, it will substitute anything currently in the prompt into that position, rather than appending the style to your prompt.
Negative prompt
Allows you to use another prompt of things the model should avoid when generating the picture. This works by using the negative prompt for unconditional conditioning in the sampling process instead of an empty string.
Advanced explanation: Negative prompt
Original | Negative: purple | Negative: tentacles |
---|---|---|
CLIP interrogator
Originally by: https://github.com/pharmapsychotic/clip-interrogator
CLIP interrogator allows you to retrieve the prompt from an image. The prompt won’t allow you to reproduce this exact image (and sometimes it won’t even be close), but it can be a good start.
The first time you run CLIP interrogator it will download a few gigabytes of models.
CLIP interrogator has two parts: one is a BLIP model that creates a text description from the picture.
Other is a CLIP model that will pick few lines relevant to the picture out of a list. By default, there
is only one list - a list of artists (from artists.csv
). You can add more lists by doing the following:
- create
interrogate
directory in the same place as webui - put text files in it with a relevant description on each line
For example of what text files to use, see https://github.com/pharmapsychotic/clip-interrogator/tree/main/data.
In fact, you can just take files from there and use them - just skip artists.txt because you already have a list of
artists in artists.csv
(or use that too, who’s going to stop you). Each file adds one line of text to the final description.
If you add “.top3.” to filename, for example, flavors.top3.txt
, the three most relevant lines from this file will be
added to the prompt (other numbers also work).
There are settings relevant to this feature:
-
Interrogate: keep models in VRAM
- do not unload Interrogate models from memory after using them. For users with a lot of VRAM. -
Interrogate: use artists from artists.csv
- adds artist fromartists.csv
when interrogating. Can be useful to disable when you have your list of artists ininterrogate
directory -
Interrogate: num_beams for BLIP
- parameter that affects how detailed descriptions from BLIP model are (the first part of generated prompt) -
Interrogate: minimum description length
- minimum length for BLIP model’s text -
Interrogate: maximum descripton length
- maximum length for BLIP model’s text -
Interrogate: maximum number of lines in text file
- interrogator will only consider this many first lines in a file. Set to 0, the default is 1500, which is about as much as a 4GB videocard can handle.
Prompt editing
Prompt editing allows you to start sampling one picture, but in the middle swap to something else. The base syntax for this is:
[from:to:when]
Where from
and to
are arbitrary texts, and when
is a number that defines how late in the sampling cycle should the switch be made. The later it is, the less power the model has to draw the to
text in place of from
text. If when
is a number between 0 and 1, it’s a fraction of the number of steps after which to make the switch. If it’s an integer greater than zero, it’s just the step after which to make the switch.
Nesting one prompt editing inside another does work.
Additionally:
-
[to:when]
- addsto
to the prompt after a fixed number of steps (when
) -
[from::when]
- removesfrom
from the prompt after a fixed number of steps (when
)
Example:
a [fantasy:cyberpunk:16] landscape
- At start, the model will be drawing
a fantasy landscape
. - After step 16, it will switch to drawing
a cyberpunk landscape
, continuing from where it stopped with fantasy.
Here’s a more complex example with multiple edits:
fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][ in background:0.25] [shoddy:masterful:0.5]
(sampler has 100 steps)
- at start,
fantasy landscape with a mountain and an oak in foreground shoddy
- after step 25,
fantasy landscape with a lake and an oak in foreground in background shoddy
- after step 50,
fantasy landscape with a lake and an oak in foreground in background masterful
- after step 60,
fantasy landscape with a lake and an oak in background masterful
- after step 75,
fantasy landscape with a lake and a christmas tree in background masterful
The picture at the top was made with the prompt:
`Official portrait of a smiling world war ii general, [male:female:0.99], cheerful, happy, detailed face, 20th century, highly detailed, cinematic lighting, digital art painting by Greg Rutkowski’s
And the number 0.99 is replaced with whatever you see in column labels on the image.
The last column in the picture is [male:female:0.0], which essentially means that you are asking the model to draw a female from the start, without starting with a male general, and that is why it looks so different from others.
Alternating Words
Convenient Syntax for swapping every other step.
[cow|horse] in a field
On step 1, prompt is “cow in a field.” Step 2 is “horse in a field.” Step 3 is “cow in a field” and so on.
See more advanced example below. On step 8, the chain loops back from “man” to “cow.”
[cow|cow|horse|man|siberian tiger|ox|man] in a field
Prompt editing was first implemented by Doggettx in this myspace.com post.
Highres. fix
A convenience option to partially render your image at a lower resolution, upscale it, and then add details at a high resolution. By default, txt2img makes horrible images at very high resolutions, and this makes it possible to avoid using the small picture’s composition. Enabled by checking the “Highres. fix” checkbox on the txt2img page.
Without | With |
---|---|
Composable Diffusion
A method to allow the combination of multiple prompts. combine prompts using an uppercase AND
a cat AND a dog
Supports weights for prompts: a cat :1.2 AND a dog AND a penguin :2.2
The default weight value is 1.
It can be quite useful for combining multiple embeddings to your result: creature_embedding in the woods:0.7 AND arcane_embedding:0.5 AND glitch_embedding:0.2
Using a value lower than 0.1 will barely have an effect. a cat AND a dog:0.03
will produce basically the same output as a cat
This could be handy for generating fine-tuned recursive variations, by continuing to append more prompts to your total. creature_embedding on log AND frog:0.13 AND yellow eyes:0.08
Interrupt
Press the Interrupt button to stop current processing.
4GB videocard support
Optimizations for GPUs with low VRAM. This should make it possible to generate 512x512 images on videocards with 4GB memory.
--lowvram
is a reimplementation of an optimization idea by basujindal.
Model is separated into modules, and only one module is kept in GPU memory; when another module needs to run, the previous
is removed from GPU memory. The nature of this optimization makes the processing run slower – about 10 times slower
compared to normal operation on my RTX 3090.
--medvram
is another optimization that should reduce VRAM usage significantly by not processing conditional and
unconditional denoising in the same batch.
This implementation of optimization does not require any modification to the original Stable Diffusion code.
Face restoration
Lets you improve faces in pictures using either GFPGAN or CodeFormer. There is a checkbox in every tab to use face restoration, and also a separate tab that just allows you to use face restoration on any picture, with a slider that controls how visible the effect is. You can choose between the two methods in settings.
Original | GFPGAN | CodeFormer |
---|---|---|
Saving
Click the Save button under the output section, and generated images will be saved to a directory specified in settings; generation parameters will be appended to a csv file in the same directory.
Loading
Gradio’s loading graphic has a very negative effect on the processing speed of the neural network.
My RTX 3090 makes images about 10% faster when the tab with gradio is not active. By default, the UI
now hides loading progress animation and replaces it with static “Loading…” text, which achieves
the same effect. Use the --no-progressbar-hiding
commandline option to revert this and show loading animations.
Prompt validation
Stable Diffusion has a limit for input text length. If your prompt is too long, you will get a warning in the text output field, showing which parts of your text were truncated and ignored by the model.
Png info
Adds information about generation parameters to PNG as a text chunk. You can view this information later using any software that supports viewing PNG chunk info, for example: https://www.nayuki.io/page/png-file-chunk-inspector
Settings
A tab with settings, allows you to use UI to edit more than half of parameters that previously were commandline. Settings are saved to config.js file. Settings that remain as commandline options are ones that are required at startup.
Filenames format
The Images filename pattern
field in the Settings tab allows customization of generated txt2img and img2img images filenames. This pattern defines the generation parameters you want to include in filenames and their order. The supported tags are:
[steps], [cfg], [prompt], [prompt_no_styles], [prompt_spaces], [width], [height], [styles], [sampler], [seed], [model_hash], [prompt_words], [date], [datetime], [job_timestamp].
This list will evolve though, with new additions. You can get an up-to-date list of supported tags by hovering your mouse over the “Images filename pattern” label in the UI.
Example of a pattern: [seed]-[steps]-[cfg]-[sampler]-[prompt_spaces]
Note about “prompt” tags: [prompt]
will add underscores between the prompt words, while [prompt_spaces]
will keep the prompt intact (easier to copy/paste into the UI again). [prompt_words]
is a simplified and cleaned-up version of your prompt, already used to generate subdirectories names, with only the words of your prompt (no punctuation).
If you leave this field empty, the default pattern will be applied ([seed]-[prompt_spaces]
).
Please note that the tags are actually replaced inside the pattern. It means that you can also add non-tags words to this pattern, to make filenames even more explicit. For example: s=[seed],p=[prompt_spaces]
User scripts
If the program is launched with --allow-code
option, an extra text input field for script code
is available at the bottom of the page, under Scripts -> Custom code. It allows you to input python
code that will do the work with the image.
In code, access parameters from web UI using the p
variable, and provide outputs for web UI
using the display(images, seed, info)
function. All globals from the script are also accessible.
A simple script that would just process the image and output it normally:
import modules.processing
processed = modules.processing.process_images(p)
print("Seed was: " + str(processed.seed))
display(processed.images, processed.seed, processed.info)
UI config
You can change parameters for UI elements:
- radio groups: default selection
- sliders: default value, min, max, step
- checkboxes: checked state
- text and number inputs: default values
The file is ui-config.json in webui dir, and it is created automatically if you don’t have one when the program starts.
Checkboxes that would usually expand a hidden section will not initially do so when set as UI config entries.
Some settings will break processing, like step not divisible by 64 for width and height, and some, like changing the default function on the img2img tab, may break UI. I do not have plans to address those in near future.
ESRGAN
It’s possible to use ESRGAN models on the Extras tab, as well as in SD upscale.
To use ESRGAN models, put them into ESRGAN directory in the same location as webui.py. A file will be loaded as a model if it has .pth extension. Grab models from the Model Database.
Not all models from the database are supported. All 2x models are most likely not supported.
img2img alternative test
Deconstructs an input image using a reverse of the Euler diffuser to create the noise pattern used to construct the input prompt.
As an example, you can use this image. Select the img2img alternative test from the scripts section.
Adjust your settings for the reconstruction process:
- Use a brief description of the scene: “A smiling woman with brown hair.” Describing features you want to change helps. Set this as your starting prompt, and ‘Original Input Prompt’ in the script settings.
- You MUST use the Euler sampling method, as this script is built on it.
- Sampling steps: 50-60. This MUCH match the decode steps value in the script, or you’ll have a bad time. Use 50 for this demo.
- CFG scale: 2 or lower. For this demo, use 1.8. (Hint, you can edit ui-config.json to change “img2img/CFG Scale/step” to .1 instead of .5.
- Denoising strength - this does matter, contrary to what the old docs said. Set it to 1.
- Width/Height - Use the width/height of the input image.
- Seed…you can ignore this. The reverse Euler is generating the noise for the image now.
- Decode cfg scale - Somewhere lower than 1 is the sweet spot. For the demo, use 1.
- Decode steps - as mentioned above, this should match your sampling steps. 50 for the demo, consider increasing to 60 for more detailed images.
Once all of the above are dialed in, you should be able to hit “Generate” and get back a result that is a very close approximation to the original.
After validating that the script is re-generating the source photo with a good degree of accuracy, you can try to change the details of the prompt. Larger variations of the original will likely result in an image with an entirely different composition than the source.
Example outputs using the above settings and prompts below (Red hair/pony not pictured)
“A smiling woman with blue hair.” Works. “A frowning woman with brown hair.” Works. “A frowning woman with red hair.” Works. “A frowning woman with red hair riding a horse.” Seems to replace the woman entirely, and now we have a ginger pony.
user.css
Create a file named user.css
near webui.py
and put custom CSS code into it. For example, this makes the gallery taller:
#txt2img_gallery, #img2img_gallery{
min-height: 768px;
}
A useful tip is you can append /?__theme=dark
to your webui url to enable a built in dark theme
e.g. (http://127.0.0.1:7860/?__theme=dark
)
Alternatively, you can add the --theme=dark
to the set COMMANDLINE_ARGS=
in webui-user.bat
e.g. set COMMANDLINE_ARGS=--theme=dark
notification.mp3
If an audio file named notification.mp3
is present in webui’s root folder, it will be played when the generation process completes.
As a source of inspiration:
- https://pixabay.com/sound-effects/search/ding/?duration=0-30
- https://pixabay.com/sound-effects/search/notification/?duration=0-30
Tweaks
Ignore last layers of CLIP model
This is a slider in settings, and it controls how early the processing of prompt by CLIP network should be stopped.
A more detailed explanation:
CLIP is a very advanced neural network that transforms your prompt text into a numerical representation. Neural networks work very well with this numerical representation and that’s why devs of SD chose CLIP as one of 3 models involved in stable diffusion’s method of producing images. As CLIP is a neural network, it means that it has a lot of layers. Your prompt is digitized in a simple way, and then fed through layers. You get numerical representation of the prompt after the 1st layer, you feed that into the second layer, you feed the result of that into third, etc, until you get to the last layer, and that’s the output of CLIP that is used in stable diffusion. This is the slider value of 1. But you can stop early, and use the output of the next to last layer - that’s slider value of 2. The earlier you stop, the less layers of neural network have worked on the prompt.
Some models were trained with this kind of tweak, so setting this value helps produce better results on those models.