About AI Generated Images

For some of the images in my newsletter, I’m experimenting with using a generative AI to create art based on a text prompt. The technique I’m using is known as VQGAN+CLIP. Right now I’m using a public “notebook” to generate the images, created by Katherine Crowson and others.

Essentially, having seen millions of labeled images on the Internet, this model attempts to create its own. It’s considered an adversarial network because one part of the model is trained to generate the images, and another part of the model is trained to discern whether or not the generated image is “real” or not. It’s basically self-critiquing, and tries iteratively to make better and better images in order to fool its critic. This VICE article has a pretty good non-technical overview.

In order to generate different images, you can not only experiment with different text prompts, but also try adding different modifiers. Adding keywords such as an artists name, an adjective, or even a source website name can have a huge effect, since the model has been trained on so many types of images. A redditor made a great image comparison of how different keywords can affect the results.

The following video shows the process of generating an image over 800 iterations based on the text prompt “The ocean is a desert with its life underground and a perfect disguise above deviantart”. I also started with an initial image, just to set the mood and composition. As you can see, the longer it runs, the less it has any resemblance to the starting image.

The next video was created almost exactly the same as the above, the only difference is the prompt: “The ocean is a desert with its life underground and a perfect disguise above Hieronymous Bosch”. It is pretty clear from the results that the AI has seen its fair share of Bosch paintings.

Finally, somehow this is the creepiest of all, the prompt: “The ocean is a desert with its life underground and a perfect disguise above children’s crayon art”. I like the creative touch of actually adding a crayon to the picture.