Usage of ControlNet

Before diving into the backend of ControlNet, in this section, we will start using ControlNet to help control image generation.

In the following example, we will first generate an image using SD, take the Canny shape of the object, and then use the Canny shape to generate a new image with the help of ControlNet.

Note

A Canny image refers to an image that has undergone Canny edge detection, which is a popular edge detection algorithm. It was developed by John F. Canny in 1986. [7]

Let’s use SD to generate an image using the following code:

1. Generate a sample image using SD:

   import torch

   from diffusers import StableDiffusionPipeline

   # load model

   text2img_pipe = StableDiffusionPipeline.from_pretrained(

       "stablediffusionapi/deliberate-v2",

        torch_dtype = torch.float16

   ).to("cuda:0")

   # generate sample image

   prompt = """

   high resolution photo,best quality, masterpiece...