AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Tao Xu; Pengchuan Zhang; Qiuyuan Huang; Han Zhang; Zhe Gan; Xiaolei Huang; Xiaodong He

doi:10.1109/cvpr.2018.00143

Abstract

In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation. With a novel attentional generative network, the AttnGAN can synthesize fine-grained details at different sub-regions of the image by paying attentions to the relevant words in the natural language description. In addition, a deep attentional multimodal similarity model is proposed to compute a fine-grained image-text matching loss for training the generator. The proposed AttnGAN significantly outperforms the previous state of the art, boosting the best reported inception score by 14.14% on the CUB dataset and 170.25% on the more challenging COCO dataset. A detailed analysis is also performed by visualizing the attention layers of the AttnGAN. It for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.

Keywords

Computer scienceGenerative grammarBoosting (machine learning)Generator (circuit theory)Generative adversarial networkArtificial intelligenceImage (mathematics)Similarity (geometry)Adversarial systemMatching (statistics)Deep learningNatural language processingImage synthesisWord (group theory)Pattern recognition (psychology)

Affiliated Institutions

Related Publications

DRAW: A Recurrent Neural Network For Image Generation

Karol Gregor , Ivo Danihelka , Alex Graves +2 more

This paper introduces the Deep Recurrent Atten-tive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism tha...

2015 758 citations

A State-of-the-Art Survey on Deep Learning Theory and Architectures

Md Zahangir Alom , Tarek M. Taha , Chris Yakopcic +7 more

In recent years, deep learning has garnered tremendous success in a variety of application domains. This new field of machine learning has been growing rapidly and has been appl...

2019 Electronics 1521 citations

Show and tell: A neural image caption generator

Oriol Vinyals , Alexander Toshev , Samy Bengio +1 more

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper...

2015 6136 citations

Survey of Hallucination in Natural Language Generation

Ziwei Ji , Nayeon Lee , Rita Frieske +7 more

Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies such as Transformer-bas...

2022 ACM Computing Surveys 2392 citations

Generative Adversarial Networks

Ian J. Goodfellow , Jean Pouget-Abadie , Mehdi Mirza +5 more

The Science of Deep Learning emerged from courses taught by the author that have provided thousands of students with training and experience for their academic studies, and prep...

2022 Cambridge University Press eBooks 2575 citations

Publication Info

Year: 2018
Type: article
Citations: 1812
Access: Closed

External Links

Download PDF (Free) View on DOI.org arXiv Semantic Scholar

Social Impact

Altmetric

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

1812

OpenAlex

320

Influential

1285

CrossRef

Cite This

APA Style

                            
                                    Tao Xu, 
                                
                                    Pengchuan Zhang, 
                                
                                    Qiuyuan Huang
                                
                                et al.
                            
                            (2018). 
                            AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. 
                            2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
                            
                            .
                            https://doi.org/10.1109/cvpr.2018.00143

Identifiers

DOI: 10.1109/cvpr.2018.00143
arXiv: 1711.10485

Data Quality

Data completeness: 84%