Generating Procedural Materials from Text or Image Prompts

Yiwei Hu1,2, Paul Guerrero2, Miloš Hašan2, Holly Rushmeier1, Valentin Deschaintre2
1Yale University,    2Adobe Research
SIGGRAPH 2023
Paper Supplementary arXiv

Procedural materials can be represented by directed computational graphs where each node represents a 2D image generator or filtering operator. The nodes are connected by unidirectional edges defining the computation flow. Our generative model can produce multiple procedural material graphs 1) from image prompts, 2) from text prompts, 3) unconditionally and 4) conditioned on partial graphs (AutoCompletion). Generated graph structures are shown in green, existing structures (in AutoCompletion) are in blue.

Abstract

Node graph systems are used ubiquitously for material design in computer graphics. They allow the use of visual programming to achieve desired effects without writing code. As high-level design tools they provide convenience and flexibility, but mastering the creation of node graphs usually requires professional training. We propose an algorithm capable of generating multiple node graphs from different types of prompts, significantly lowering the bar for users to explore a specific design space. Previous work (MatFormer) was limited to unconditional generation of random node graphs, making the generation of an envisioned material challenging. We propose a multi-modal node graph generation neural architecture for high-quality procedural material synthesis which can be conditioned on different inputs (text or image prompts), using a CLIP-based encoder. We also create a substantially augmented material graph dataset, key to improving the generation quality. Finally, we generate high-quality graph samples using a regularized sampling process and improve the matching quality by differentiable optimization for top-ranked samples. We compare our methods to CLIP-based database search baselines (which are themselves novel) and achieve superior or similar performance without requiring massive data storage. We further show that our model can produce a set of material graphs unconditionally, conditioned on images, text prompts or partial graphs, serving as a tool for automatic visual programming completion.

image prompts

Image Prompts. We show real photos as inputs and three of our generated material graphs. For each example, the top row shows non-optimized outputs directly predicted by our models, while the bottom row shows our final output after optimization. See supplemental material for more results.

text prompts

Text Prompts. Our model generates multiple procedural material graphs given various text prompts. See supplemental material for more results.

autocompletion

Autocompletion. As a sequential model, our model can accept partial sequences (partially completed material graphs) and generate the rest of the structures and parameters toward image prompts. Existing structures are blue and our predicted are green.

BibTeX

@inproceedings{hu2023gen, 
author = {Hu, Yiwei and Guerrero, Paul and Hasan, Milos and Rushmeier, Holly and Deschaintre, Valentin},
title = "{Generating Procedural Materials from Text or Image Prompts}",
year = {2023},
booktitle = {ACM SIGGRAPH 2023 Conference Proceedings}}

Acknowledgement: This work was supported in part by NSF Grant No. IIS-2007283.