Semantic Image Synthesis With Spatially-Adaptive Normalization

Taesung Park; Ming-Yu Liu; Ting-Chun Wang; Jun-Yan Zhu

doi:10.1109/cvpr.2019.00244

Abstract

We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the network, forcing the network to memorize the information throughout all the layers. Instead, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned affine transformation. Experiments on several challenging datasets demonstrate the superiority of our method compared to existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows users to easily control the style and content of image synthesis results as well as create multi-modal results. Code is available upon publication.

Keywords

Normalization (sociology)Computer scienceAffine transformationFidelityImage synthesisArtificial intelligenceModalImage (mathematics)Pattern recognition (psychology)Computer vision

Affiliated Institutions

Related Publications

Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis

Chuan Li , Michael Wand

This paper studies a combination of generative Markov random field (MRF) models and discriminatively trained deep convolutional neural networks (dCNNs) for synthesizing 2D image...

2016 695 citations

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Ting-Chun Wang , Ming-Yu Liu , Jun-Yan Zhu +3 more

We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs). Co...

2018 4266 citations

Spectral Normalization for Generative Adversarial Networks

Takeru Miyato , Toshiki Kataoka , Masanori Koyama +1 more

One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called...

2018 arXiv (Cornell University) 1759 citations

Deep Belief Networks using discriminative features for phone recognition

Abdelrahman Mohamed , Tara N. Sainath , George E. Dahl +3 more

Deep Belief Networks (DBNs) are multi-layer generative models. They can be trained to model windows of coefficients extracted from speech and they discover multiple layers of fe...

2011 289 citations

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Nataniel Ruiz , Yuanzhen Li , Varun Jampani +3 more

Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these mode...

2023 2023 IEEE/CVF Conference on Computer ... 1568 citations

Publication Info

Year: 2019
Type: article
Pages: 2332-2341
Citations: 2662
Access: Closed

External Links

Download PDF (Free) View on DOI.org arXiv Semantic Scholar

Social Impact

Altmetric

Semantic Image Synthesis With Spatially-Adaptive Normalization

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

2662

OpenAlex

505

Influential

2034

CrossRef

Cite This

APA Style

                            
                                    Taesung Park, 
                                
                                    Ming-Yu Liu, 
                                
                                    Ting-Chun Wang
                                
                                et al.
                            
                            (2019). 
                            Semantic Image Synthesis With Spatially-Adaptive Normalization. 
                            2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
                            
                            , 2332-2341.
                            https://doi.org/10.1109/cvpr.2019.00244

Identifiers

DOI: 10.1109/cvpr.2019.00244
arXiv: 1903.07291

Data Quality

Data completeness: 84%