Features

This is a feature showcase page for Stable Diffusion web UI.

All examples are non-cherrypicked unless specified otherwise.

Stable Diffusion 2.0

Basic models

Models are supported: 768-v-ema.ckpt (model, config) and 512-base-ema.ckpt (model, config). 2.1 checkpoints should also work.

• download the checkpoint (from here: https://huggingface.co/stabilityai/stable-diffusion-2)
• put it into models/Stable-Diffusion directory
• grab the config from SD2.0 repository and put it into same place as the checkpoint, renaming it to have same filename (i.e. if your checkpoint is named 768-v-ema.ckpt, the config should be named 768-v-ema.yaml)
• select the new checkpoint from the UI

Train tab will most likely be broken for the 2.0 models.

If 2.0 or 2.1 is generating black images, enable full precision with --no-half or try using the --xformers optimization.

Note: SD 2.0 and 2.1 are more sensitive to FP16 numerical instability (as noted by themselves here) due to their new cross attention module.

On fp16: comment to enable, in webui-user.bat:

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=your command line options
set STABLE_DIFFUSION_COMMIT_HASH="c12d960d1ee4f9134c2516862ef991ec52d3f59e"
set ATTN_PRECISION=fp16

call webui.bat

Depth-guided model

More info. PR. Instructions:

• download the 512-depth-ema.ckpt checkpoint
• place it in models/Stable-diffusion
• grab the config and place it in the same folder as the checkpoint
• rename the config to 512-depth-ema.yaml
• select the new checkpoint from the UI

The depth-guided model will only work in img2img tab.

Outpainting

Outpainting extends the original image and inpaints the created empty space.

Example:

Original	Outpainting	Outpainting again

Original image by Anonymous user from 4chan. Thank you, Anonymous user.

You can find the feature in the img2img tab at the bottom, under Script -> Poor man’s outpainting.

Outpainting, unlike normal image generation, seems to profit very much from large step count. A recipe for a good outpainting is a good prompt that matches the picture, sliders for denoising and CFG scale set to max, and step count of 50 to 100 with Euler ancestral or DPM2 ancestral samplers.

81 steps, Euler A	30 steps, Euler A	10 steps, Euler A	80 steps, Euler A

Inpainting

In img2img tab, draw a mask over a part of the image, and that part will be in-painted.

Options for inpainting:

• draw a mask yourself in the web editor
• erase a part of the picture in an external editor and upload a transparent picture. Any even slightly transparent areas will become part of the mask. Be aware that some editors save completely transparent areas as black by default.
• change mode (to the bottom right of the picture) to “Upload mask” and choose a separate black and white image for the mask (white=inpaint).

Inpainting model

RunwayML has trained an additional model specifically designed for inpainting. This model accepts additional inputs - the initial image without noise plus the mask - and seems to be much better at the job.

Download and extra info for the model is here: https://github.com/runwayml/stable-diffusion#inpainting-with-stable-diffusion

To use the model, you must rename the checkpoint so that its filename ends in inpainting.ckpt, for example, 1.5-inpainting.ckpt.

After that just select the checkpoint as you’d usually select any checkpoint and you’re good to go.

Masked content

The masked content field determines content is placed to put into the masked regions before they are inpainted.

mask	fill	original	latent noise	latent nothing

Inpaint at full resolution

Normally, inpainting resizes the image to the target resolution specified in the UI. With Inpaint at full resolution enabled, only the masked region is resized, and after processing it is pasted back to the original picture. This allows you to work with large pictures and allows you to render the inpainted object at a much larger resolution.

Input	Inpaint normal	Inpaint at whole resolution

Masking mode

There are two options for masked mode:

• Inpaint masked - the region under the mask is inpainted
• Inpaint not masked - under the mask is unchanged, everything else is inpainted

Alpha mask

Input	Output

Prompt matrix

Separate multiple prompts using the | character, and the system will produce an image for every combination of them. For example, if you use a busy city street in a modern city|illustration|cinematic lighting prompt, there are four combinations possible (first part of the prompt is always kept):

• a busy city street in a modern city
• a busy city street in a modern city, illustration
• a busy city street in a modern city, cinematic lighting
• a busy city street in a modern city, illustration, cinematic lighting

Four images will be produced, in this order, all with the same seed and each with a corresponding prompt:

Another example, this time with 5 prompts and 16 variations:

You can find the feature at the bottom, under Script -> Prompt matrix.

Color Sketch

Basic coloring tool for img2img. To use this feature in img2img, enable with --gradio-img2img-tool color-sketch in commandline args. To use this feature in inpainting mode, enable with --gradio-inpaint-tool color-sketch. Chromium-based browsers support a dropper tool. (see picture)

Stable Diffusion upscale

Upscale image using RealESRGAN/ESRGAN and then go through tiles of the result, improving them with img2img. It also has an option to let you do the upscaling part yourself in an external program, and just go through tiles with img2img.

Original idea by: https://github.com/jquesnelle/txt2imghd. This is an independent implementation.

To use this feature, select SD upscale from the scripts dropdown selection (img2img tab).

The input image will be upscaled to twice the original width and height, and UI’s width and height sliders specify the size of individual tiles. Because of overlap, the size of the tile can be very important: 512x512 image needs nine 512x512 tiles (because of overlap), but only four 640x640 tiles.

Recommended parameters for upscaling:

• Sampling method: Euler a
• Denoising strength: 0.2, can go up to 0.4 if you feel adventurous

Original	RealESRGAN	Topaz Gigapixel	SD upscale

Attention/emphasis

Using () in the prompt increases the model’s attention to enclosed words, and [] decreases it. You can combine multiple modifiers:

Cheat sheet:

• a (word) - increase attention to word by a factor of 1.1
• a ((word)) - increase attention to word by a factor of 1.21 (= 1.1 * 1.1)
• a [word] - decrease attention to word by a factor of 1.1
• a (word:1.5) - increase attention to word by a factor of 1.5
• a (word:0.25) - decrease attention to word by a factor of 4 (= 1 / 0.25)
• a \(word\) - use literal () characters in prompt

With (), a weight can be specified like this: (text:1.4). If the weight is not specified, it is assumed to be 1.1. Specifying weight only works with () not with [].

If you want to use any of the literal ()[] characters in the prompt, use the backslash to escape them: anime_\(character\).

On 2022-09-29, a new implementation was added that supports escape characters and numerical weights. A downside of the new implementation is that the old one was not perfect and sometimes ate characters: “a (((farm))), daytime”, for example, would become “a farm daytime” without the comma. This behavior is not shared by the new implementation which preserves all text correctly, and this means that your saved seeds may produce different pictures. For now, there is an option in settings to use the old implementation.

NAI uses my implementation from before 2022-09-29, except they have 1.05 as the multiplier and use {} instead of (). So the conversion applies:

• their {word} = our (word:1.05)
• their `` = our (word:1.1025)
• their [word] = our (word:0.952) (0.952 = 1/1.05)
• their [[word]] = our (word:0.907) (0.907 = 1/1.05/1.05)

Loopback

Selecting the loopback script in img2img allows you to automatically feed output image as input for the next batch. Equivalent to saving output image, and replacing the input image with it. Batch count setting controls how many iterations of this you get.

Usually, when doing this, you would choose one of many images for the next iteration yourself, so the usefulness of this feature may be questionable, but I’ve managed to get some very nice outputs with it that I wasn’t able to get otherwise.

Example: (cherrypicked result)

Original image by Anonymous user from 4chan. Thank you, Anonymous user.

X/Y plot

Creates a grid of images with varying parameters. Select which parameters should be shared by rows and columns using X type and Y type fields, and input those parameters separated by comma into X values/Y values fields. For integer, and floating point numbers, and ranges are supported. Examples:

• Simple ranges:
  • 1-5 = 1, 2, 3, 4, 5
• Ranges with increment in bracket:
  • 1-5 (+2) = 1, 3, 5
  • 10-5 (-3) = 10, 7
  • 1-3 (+0.5) = 1, 1.5, 2, 2.5, 3
• Ranges with the count in square brackets:
  • 1-10 [5] = 1, 3, 5, 7, 10
  • 0.0-1.0 [6] = 0.0, 0.2, 0.4, 0.6, 0.8, 1.0

Here are the settings that create the graph above:

Prompt S/R

Prompt S/R is one of more difficult to understand modes of operation for X/Y Plot. S/R stands for search/replace, and that’s what it does - you input a list of words or phrases, it takes the first from the list and treats it as keyword, and replaces all instances of that keyword with other entries from the list.

For example, with prompt a man holding an apple, 8k clean, and Prompt S/R an apple, a watermelon, a gun you will get three prompts:

• a man holding an apple, 8k clean
• a man holding a watermelon, 8k clean
• a man holding a gun, 8k clean

The list uses the same syntax as a line in a CSV file, so if you want to include commas into your entries you have to put text in quotes and make sure there is no space between quotes and separating commas:

• darkness, light, green, heat - 4 items - darkness, light, green, heat
• darkness, "light, green", heat - WRONG - 4 items - darkness, "light, green", heat
• darkness,"light, green",heat - RIGHT - 3 items - darkness, light, green, heat

Textual Inversion

Short explanation: place your embeddings into the embeddings directory, and use the filename in the prompt.

Long explanation: Textual Inversion

Resizing

There are three options for resizing input images in img2img mode:

• Just resize - simply resizes the source image to the target resolution, resulting in an incorrect aspect ratio
• Crop and resize - resize source image preserving aspect ratio so that entirety of target resolution is occupied by it, and crop parts that stick out
• Resize and fill - resize source image preserving aspect ratio so that it entirely fits target resolution, and fill empty space by rows/columns from the source image

Example:

Sampling method selection

Pick out of multiple sampling methods for txt2img:

Seed resize

This function allows you to generate images from known seeds at different resolutions. Normally, when you change resolution, the image changes entirely, even if you keep all other parameters including seed. With seed resizing you specify the resolution of the original image, and the model will very likely produce something looking very similar to it, even at a different resolution. In the example below, the leftmost picture is 512x512, and others are produced with exact same parameters but with larger vertical resolution.

Info	Image
Seed resize not enabled
Seed resized from 512x512

Ancestral samplers are a little worse at this than the rest.

You can find this feature by clicking the “Extra” checkbox near the seed.

Variations

A Variation strength slider and Variation seed field allow you to specify how much the existing picture should be altered to look like a different one. At maximum strength, you will get pictures with the Variation seed, at minimum - pictures with the original Seed (except for when using ancestral samplers).

You can find this feature by clicking the “Extra” checkbox near the seed.

Styles

Press the “Save prompt as style” button to write your current prompt to styles.csv, the file with a collection of styles. A dropbox to the right of the prompt will allow you to choose any style out of previously saved, and automatically append it to your input. To delete a style, manually delete it from styles.csv and restart the program.

if you use the special string {prompt} in your style, it will substitute anything currently in the prompt into that position, rather than appending the style to your prompt.

Negative prompt

Allows you to use another prompt of things the model should avoid when generating the picture. This works by using the negative prompt for unconditional conditioning in the sampling process instead of an empty string.

Advanced explanation: Negative prompt

Original	Negative: purple	Negative: tentacles

CLIP interrogator

Originally by: https://github.com/pharmapsychotic/clip-interrogator

CLIP interrogator allows you to retrieve the prompt from an image. The prompt won’t allow you to reproduce this exact image (and sometimes it won’t even be close), but it can be a good start.

The first time you run CLIP interrogator it will download a few gigabytes of models.

CLIP interrogator has two parts: one is a BLIP model that creates a text description from the picture. Other is a CLIP model that will pick few lines relevant to the picture out of a list. By default, there is only one list - a list of artists (from artists.csv). You can add more lists by doing the following:

• create interrogate directory in the same place as webui
• put text files in it with a relevant description on each line

For example of what text files to use, see https://github.com/pharmapsychotic/clip-interrogator/tree/main/data. In fact, you can just take files from there and use them - just skip artists.txt because you already have a list of artists in artists.csv (or use that too, who’s going to stop you). Each file adds one line of text to the final description. If you add “.top3.” to filename, for example, flavors.top3.txt, the three most relevant lines from this file will be added to the prompt (other numbers also work).

There are settings relevant to this feature:

• Interrogate: keep models in VRAM - do not unload Interrogate models from memory after using them. For users with a lot of VRAM.
• Interrogate: use artists from artists.csv - adds artist from artists.csv when interrogating. Can be useful to disable when you have your list of artists in interrogate directory
• Interrogate: num_beams for BLIP - parameter that affects how detailed descriptions from BLIP model are (the first part of generated prompt)
• Interrogate: minimum description length - minimum length for BLIP model’s text
• Interrogate: maximum descripton length - maximum length for BLIP model’s text
• Interrogate: maximum number of lines in text file - interrogator will only consider this many first lines in a file. Set to 0, the default is 1500, which is about as much as a 4GB videocard can handle.

Prompt editing

Prompt editing allows you to start sampling one picture, but in the middle swap to something else. The base syntax for this is:

[from:to:when]

Where from and to are arbitrary texts, and when is a number that defines how late in the sampling cycle should the switch be made. The later it is, the less power the model has to draw the to text in place of from text. If when is a number between 0 and 1, it’s a fraction of the number of steps after which to make the switch. If it’s an integer greater than zero, it’s just the step after which to make the switch.

Nesting one prompt editing inside another does work.

Additionally:

• [to:when] - adds to to the prompt after a fixed number of steps (when)
• [from::when] - removes from from the prompt after a fixed number of steps (when)

Example: a [fantasy:cyberpunk:16] landscape

• At start, the model will be drawing a fantasy landscape.
• After step 16, it will switch to drawing a cyberpunk landscape, continuing from where it stopped with fantasy.

Here’s a more complex example with multiple edits: fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][ in background:0.25] [shoddy:masterful:0.5] (sampler has 100 steps)

• at start, fantasy landscape with a mountain and an oak in foreground shoddy
• after step 25, fantasy landscape with a lake and an oak in foreground in background shoddy
• after step 50, fantasy landscape with a lake and an oak in foreground in background masterful
• after step 60, fantasy landscape with a lake and an oak in background masterful
• after step 75, fantasy landscape with a lake and a christmas tree in background masterful

The picture at the top was made with the prompt:

`Official portrait of a smiling world war ii general, [male:female:0.99], cheerful, happy, detailed face, 20th century, highly detailed, cinematic lighting, digital art painting by Greg Rutkowski’s

And the number 0.99 is replaced with whatever you see in column labels on the image.

The last column in the picture is [male:female:0.0], which essentially means that you are asking the model to draw a female from the start, without starting with a male general, and that is why it looks so different from others.

Alternating Words

Convenient Syntax for swapping every other step.

[cow|horse] in a field

On step 1, prompt is “cow in a field.” Step 2 is “horse in a field.” Step 3 is “cow in a field” and so on.

See more advanced example below. On step 8, the chain loops back from “man” to “cow.”

[cow|cow|horse|man|siberian tiger|ox|man] in a field

Prompt editing was first implemented by Doggettx in this myspace.com post.

Highres. fix

A convenience option to partially render your image at a lower resolution, upscale it, and then add details at a high resolution. By default, txt2img makes horrible images at very high resolutions, and this makes it possible to avoid using the small picture’s composition. Enabled by checking the “Highres. fix” checkbox on the txt2img page.

Without	With

Composable Diffusion

A method to allow the combination of multiple prompts. combine prompts using an uppercase AND

a cat AND a dog

Supports weights for prompts: a cat :1.2 AND a dog AND a penguin :2.2 The default weight value is 1. It can be quite useful for combining multiple embeddings to your result: creature_embedding in the woods:0.7 AND arcane_embedding:0.5 AND glitch_embedding:0.2

Using a value lower than 0.1 will barely have an effect. a cat AND a dog:0.03 will produce basically the same output as a cat

This could be handy for generating fine-tuned recursive variations, by continuing to append more prompts to your total. creature_embedding on log AND frog:0.13 AND yellow eyes:0.08

Interrupt

Press the Interrupt button to stop current processing.

4GB videocard support

Optimizations for GPUs with low VRAM. This should make it possible to generate 512x512 images on videocards with 4GB memory.

--lowvram is a reimplementation of an optimization idea by basujindal. Model is separated into modules, and only one module is kept in GPU memory; when another module needs to run, the previous is removed from GPU memory. The nature of this optimization makes the processing run slower – about 10 times slower compared to normal operation on my RTX 3090.

--medvram is another optimization that should reduce VRAM usage significantly by not processing conditional and unconditional denoising in the same batch.

This implementation of optimization does not require any modification to the original Stable Diffusion code.

Face restoration

Lets you improve faces in pictures using either GFPGAN or CodeFormer. There is a checkbox in every tab to use face restoration, and also a separate tab that just allows you to use face restoration on any picture, with a slider that controls how visible the effect is. You can choose between the two methods in settings.

Original	GFPGAN	CodeFormer

Saving

Click the Save button under the output section, and generated images will be saved to a directory specified in settings; generation parameters will be appended to a csv file in the same directory.

Loading

Gradio’s loading graphic has a very negative effect on the processing speed of the neural network. My RTX 3090 makes images about 10% faster when the tab with gradio is not active. By default, the UI now hides loading progress animation and replaces it with static “Loading…” text, which achieves the same effect. Use the --no-progressbar-hiding commandline option to revert this and show loading animations.

Prompt validation

Stable Diffusion has a limit for input text length. If your prompt is too long, you will get a warning in the text output field, showing which parts of your text were truncated and ignored by the model.

Png info

Adds information about generation parameters to PNG as a text chunk. You can view this information later using any software that supports viewing PNG chunk info, for example: https://www.nayuki.io/page/png-file-chunk-inspector

Settings

A tab with settings, allows you to use UI to edit more than half of parameters that previously were commandline. Settings are saved to config.js file. Settings that remain as commandline options are ones that are required at startup.

Filenames format

The Images filename pattern field in the Settings tab allows customization of generated txt2img and img2img images filenames. This pattern defines the generation parameters you want to include in filenames and their order. The supported tags are:

[steps], [cfg], [prompt], [prompt_no_styles], [prompt_spaces], [width], [height], [styles], [sampler], [seed], [model_hash], [prompt_words], [date], [datetime], [job_timestamp].

This list will evolve though, with new additions. You can get an up-to-date list of supported tags by hovering your mouse over the “Images filename pattern” label in the UI.

Example of a pattern: [seed]-[steps]-[cfg]-[sampler]-[prompt_spaces]

Note about “prompt” tags: [prompt] will add underscores between the prompt words, while [prompt_spaces] will keep the prompt intact (easier to copy/paste into the UI again). [prompt_words] is a simplified and cleaned-up version of your prompt, already used to generate subdirectories names, with only the words of your prompt (no punctuation).

If you leave this field empty, the default pattern will be applied ([seed]-[prompt_spaces]).

Please note that the tags are actually replaced inside the pattern. It means that you can also add non-tags words to this pattern, to make filenames even more explicit. For example: s=[seed],p=[prompt_spaces]

User scripts

If the program is launched with --allow-code option, an extra text input field for script code is available at the bottom of the page, under Scripts -> Custom code. It allows you to input python code that will do the work with the image.

In code, access parameters from web UI using the p variable, and provide outputs for web UI using the display(images, seed, info) function. All globals from the script are also accessible.

A simple script that would just process the image and output it normally:

import modules.processing

processed = modules.processing.process_images(p)

print("Seed was: " + str(processed.seed))

display(processed.images, processed.seed, processed.info)

UI config

You can change parameters for UI elements:

• radio groups: default selection
• sliders: default value, min, max, step
• checkboxes: checked state
• text and number inputs: default values

The file is ui-config.json in webui dir, and it is created automatically if you don’t have one when the program starts.

Checkboxes that would usually expand a hidden section will not initially do so when set as UI config entries.

Some settings will break processing, like step not divisible by 64 for width and height, and some, like changing the default function on the img2img tab, may break UI. I do not have plans to address those in near future.

ESRGAN

It’s possible to use ESRGAN models on the Extras tab, as well as in SD upscale.

To use ESRGAN models, put them into ESRGAN directory in the same location as webui.py. A file will be loaded as a model if it has .pth extension. Grab models from the Model Database.

Not all models from the database are supported. All 2x models are most likely not supported.

img2img alternative test

Deconstructs an input image using a reverse of the Euler diffuser to create the noise pattern used to construct the input prompt.

As an example, you can use this image. Select the img2img alternative test from the scripts section.

Adjust your settings for the reconstruction process:

• Use a brief description of the scene: “A smiling woman with brown hair.” Describing features you want to change helps. Set this as your starting prompt, and ‘Original Input Prompt’ in the script settings.
• You MUST use the Euler sampling method, as this script is built on it.
• Sampling steps: 50-60. This MUCH match the decode steps value in the script, or you’ll have a bad time. Use 50 for this demo.
• CFG scale: 2 or lower. For this demo, use 1.8. (Hint, you can edit ui-config.json to change “img2img/CFG Scale/step” to .1 instead of .5.
• Denoising strength - this does matter, contrary to what the old docs said. Set it to 1.
• Width/Height - Use the width/height of the input image.
• Seed…you can ignore this. The reverse Euler is generating the noise for the image now.
• Decode cfg scale - Somewhere lower than 1 is the sweet spot. For the demo, use 1.
• Decode steps - as mentioned above, this should match your sampling steps. 50 for the demo, consider increasing to 60 for more detailed images.

Once all of the above are dialed in, you should be able to hit “Generate” and get back a result that is a very close approximation to the original.

After validating that the script is re-generating the source photo with a good degree of accuracy, you can try to change the details of the prompt. Larger variations of the original will likely result in an image with an entirely different composition than the source.

Example outputs using the above settings and prompts below (Red hair/pony not pictured)

“A smiling woman with blue hair.” Works. “A frowning woman with brown hair.” Works. “A frowning woman with red hair.” Works. “A frowning woman with red hair riding a horse.” Seems to replace the woman entirely, and now we have a ginger pony.

user.css

Create a file named user.css near webui.py and put custom CSS code into it. For example, this makes the gallery taller:

#txt2img_gallery, #img2img_gallery{
    min-height: 768px;
}

A useful tip is you can append /?__theme=dark to your webui url to enable a built in dark theme
e.g. (http://127.0.0.1:7860/?__theme=dark)

Alternatively, you can add the --theme=dark to the set COMMANDLINE_ARGS= in webui-user.bat
e.g. set COMMANDLINE_ARGS=--theme=dark

notification.mp3

If an audio file named notification.mp3 is present in webui’s root folder, it will be played when the generation process completes.

As a source of inspiration:

• https://pixabay.com/sound-effects/search/ding/?duration=0-30
• https://pixabay.com/sound-effects/search/notification/?duration=0-30

Tweaks

Ignore last layers of CLIP model

This is a slider in settings, and it controls how early the processing of prompt by CLIP network should be stopped.

A more detailed explanation:

CLIP is a very advanced neural network that transforms your prompt text into a numerical representation. Neural networks work very well with this numerical representation and that’s why devs of SD chose CLIP as one of 3 models involved in stable diffusion’s method of producing images. As CLIP is a neural network, it means that it has a lot of layers. Your prompt is digitized in a simple way, and then fed through layers. You get numerical representation of the prompt after the 1st layer, you feed that into the second layer, you feed the result of that into third, etc, until you get to the last layer, and that’s the output of CLIP that is used in stable diffusion. This is the slider value of 1. But you can stop early, and use the output of the next to last layer - that’s slider value of 2. The earlier you stop, the less layers of neural network have worked on the prompt.

Some models were trained with this kind of tweak, so setting this value helps produce better results on those models.