Stable Diffusion XL

With Auto 1111 SDK, using Stable Diffusion XL is super easy. We have a dedicated pipeline for it because it requires different flags to be set than the regular Stable Diffusion Pipeline.

1. First Download a dedicated Stable Diffusion XL safetensors checkpoint from Civit AI. You can use Auto 1111 SDK's dedicated civit downloader to do this:

from auto1111sdk import civit_downloader

civit_download("https://civitai.com/models/101055/sd-xl", "sdxl.safetensors")

2. Import the Stable Diffusion XL pipeline and run the generation:

from auto1111sdk import StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline("sdxl.safetensors")

prompt = "closeup portrait photo of beautiful woman, 8k uhd, high quality, cinematic"
negative_prompt = "(deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime), text, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck"

output = pipe.generate_txt2img(num_images = 1, prompt = prompt, height = 1024, width = 768, negative_prompt = negative_prompt, steps = 20)

One of the problems with these safetensor files and the original Automatic 1111 Web UI is that SDXL is only natively supported using float32 data type. This makes the model/generation extremely memory intensive and long, even on powerful systems. To combat this, Auto 1111 SDK offers a way to set a custom float16 VAE and set custom flags. Here's how:

1. Go to this Huggingface model page and download the float 16 VAE file: https://huggingface.co/madebyollin/sdxl-vae-fp16-fix

2. Define the custom pipe with the following flags:

from auto1111sdk import StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline("sdxl.safetensors", "--skip-torch-cuda-test --medvram")

3. Set the custom VAE file you downloaded

pipe.set_vae("sdxl.vae.safetensors")

Now, the generation time has significantly reduced from 1 minute 10 seconds to just 22 seconds!. Furthermore, the memory/VRAM usage has been significantly reduced! The generation parameters set above generated the below output: