Inpainting

Inpainting refers to the task of using AI to replace specific parts of an image, while keeping the rest of the image the same. For example, given this image generated using Realistic Vision and this mask (the shaded part of the mask refers to the part of the original image I want to replace):

I can use Auto 1111 SDK's Inpainting feature to make the woman's hair color something different. For example, green:

---

Inpainting with Auto 1111 SDK doesn't actually require a different pipeline object. You can use the same pipeline object that you used to generate the image to inpaint the image. However, for better results, it is recommended that you use a dedicated inpainting model, which you can find on Civit AI: https://civitai.com/. In this example, I will be using the dreamshaper v8 inpainting pipeline: https://civitai.com/models/4384?modelVersionId=131004. Here's how you can inpaint:

1. Initialize your pipeline. To download the model, you can use Auto 1111 SDK's Civit AI model downloader.

from auto1111sdk import StableDiffusionPipeline
inpainting_pipe = StableDiffusionPipeline("dreamshaper_v8Inpainting.safetensors")

2. Open the image and mask photos as Image objects

from PIL import Image

image = Image.open("photo.png")
mask = Image.open("mask.png")

3. Inpaint.

output = pipe.inpainting_img2img(num_images = 1, inpaint_full_res_padding=32, initial_noise_multiplier=1, inpainting_fill=1, mask = mask, init_images = image, prompt = prompt, negative_prompt = negative_prompt)

The resulting image will depend heavily on your prompt and negative prompt. In this example, my original prompt and negative prompt were:

prompt = "closeup portrait photo of beautiful 26 y.o woman, makeup, 8k uhd, high quality, dramatic, cinematic,"
negative_prompt = "(deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime), text, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck"

For the inpainting parameters, I simply added ((green hair)) to the prompt. The double brackets indicates Auto 1111 SDK to use prompt attention, which you can read more about on our github repository.

---

Parameters

Currently, Auto 1111 SDK supports the following parameters (and the corresponding default values):

init_image: a PIL Image object
mask: a PIL Image object
prompt: str
negative_prompt: str = ''
seed: int = -1
steps: int = 20
cfg_scale: float = 7.5
num_images: int = 1
sampler_name: str = 'Euler'
denoising_strength: float = 0.75
mask_blur: int = 4
inpaint_full_res_padding: int = 32