Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Abstract

Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.

Keywords

Computer scienceBenchmark (surveying)Artificial intelligenceVisualizationGazeObject (grammar)BackpropagationSalientSequence (biology)Object detectionMachine translationImage (mathematics)Artificial neural networkMachine learningComputer visionPattern recognition (psychology)

Affiliated Institutions

Related Publications

Object Detection With Deep Learning: A Review

Zhong‐Qiu Zhao , Peng Zheng , Shou-Tao Xu +1 more

Due to object detection's close relationship with video analysis and image understanding, it has attracted much research attention in recent years. Traditional object detection ...

2019 IEEE Transactions on Neural Networks ... 5019 citations

A Sparse Object Category Model for Efficient Learning and Exhaustive Recognition

Rob Fergus , Pietro Perona , Andrew Zisserman

We present a "parts and structure" model for object category recognition that can be learnt efficiently and in a semi-supervised manner: the model is learnt from example images ...

2005 266 citations

Object Detection with Discriminatively Trained Part-Based Models

Pedro F. Felzenszwalb , Ross Girshick , David McAllester +1 more

We describe an object detection system based on mixtures of multiscale deformable part models. Our system is able to represent highly variable object classes and achieves state-...

2009 IEEE Transactions on Pattern Analysis... 9911 citations

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Sangdoo Yun , Dongyoon Han , Sanghyuk Chun +3 more

Regional dropout strategies have been proposed to enhance performance of convolutional neural network classifiers. They have proved to be effective for guiding the model to atte...

2019 4293 citations

Attention Is All You Need

Ashish Vaswani , Noam Shazeer , Niki Parmar +5 more

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also co...

2025 6466 citations

Publication Info

Year: 2015
Type: article
Volume: 3
Pages: 2048-2057
Citations: 7493
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

7493

OpenAlex

Cite This

APA Style

                            
                                    Kelvin Xu, 
                                
                                    Jimmy Ba, 
                                
                                    Ryan Kiros
                                
                                et al.
                            
                            (2015). 
                            Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. 
                            arXiv (Cornell University)
                            , 3
                            
                            , 2048-2057.
                            https://doi.org/10.48550/arxiv.1502.03044

Identifiers

DOI: 10.48550/arxiv.1502.03044