Steerable AI-powered Art-making Tools

Abstract

Artificial Intelligence (AI) and Machine Learning (ML) algorithms are pushing the boundaries of art-making but are hard to direct to the user's intention. They broaden the kinds of artifacts artists can create and can make the art creation process easier and more accessible. For example, prompt-based image generation models provide novel ways to generate images with prompts. Moreover, as prompt-based image generation does not require manual art-making skills, even non-experts in visual arts can experience producing images. However, AI models are often not fully steerable to the user's intention, due to their limited interface and unpredictable behaviors. In this dissertation, I expand the use of AI models in human art-making by creating effective, efficient, and iterative steering interactions that mix multiple familiar input modalities. In the first part of the dissertation, I study users and existing creativity support tools to identify how we should design steering interactions for AI-powered tools. From these studies, I found that steerability should let users effectively specify what is allowed and not allowed from AI models with efficient and iterative interactions. Moreover, I identified that artists often convey their intentions about which art to create with modalities beyond those they use to create the artifact. For example, in visual art commissions, they often accompany textual notes along with image references.

Based on the first two studies, I introduce the design of AI-powered art-making tools that mix familiar input modalities into steering interactions. With this approach, I created three AI-powered art-making tools. I designed the first tool, Artinter, to support human-human communication around art commissions. This tool shows how mood boards and concept-based interactions can help in the search and generation of reference artifacts. For example, with Artinter, the user can define their concept of ``gloom'' in the mood board and use it as a slider to steer search and generation functions to draw more references that can convey the user's artistic intention. TaleBrush is a human-AI story co-creation system powered by generative language models. It allows users to steer generative language models with a visual sketching of the protagonist’s fortune. For instance, if the user wants to generate story sentences where the character gradually experiences worse fortunes, the user can visually sketch the fortune arc that goes down. With user sketches, TaleBrush will generate appropriate story sentences. Finally, PromptPaint is a text-to-image generation tool that expands steering interactions beyond text prompts by adopting visual interactions that resemble how we use paint mediums (e.g., oil painting). For example, users can flexibly explore the prompt space beyond what they can verbally describe by mixing the prompts in the prompt palette. Through these tools, my research demonstrates how AI-powered art-making tools can be more usable and useful by mixing familiar input modalities into steering interactions.