ControlNet

We now have ControlNet support on Auto1111SDK from the extension. ControlNet can be added to the original Stable Diffusion model to generate images to greatly customize the generation process.

With Auto1111SDK, you can use ControlNet as follows:

1. Initialize a ControlNet model first. This can be done as follows:

from auto1111sdk import ControlNetModel

model = ControlNetModel(model="<THE CONTROLNET MODEL FILE NAME (WITHOUT EXTENSION)>", 
                   image="<PATH TO IMAGE>")

An example of this would be:

model = ControlNet(model="control_v11p_sd15_openpose", 
                   image="stock_mountain.png")

A full list of parameters is shown below:

model: str # The name of the model
image: str # The path to the image
weight: float = 1.0
resize_mode: int = 1
lowvram: bool = False
processor_res: int = 64
threshold_a: int = 64
threshold_b: int = 64
guidance_start: float = 0.0
guidance_end: float = 1.0
control_mode: int = 0
pixel_perfect: bool = False

2. Add the controlnet model to the StableDiffusionPipeline for text-to-image or image-to-image

pipe = StableDiffusionPipeline("dreamshaper.safetensors", controlnet=model)
prompt = "closeup portrait photo of black dog, 8k uhd, high quality, cinematic"
negative_prompt = "(deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime), text, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck"
output = pipe.generate_txt2img(num_images = 1, prompt = prompt, negative_prompt = negative_prompt, steps = 20)

output[0].save("huh.png")