Show, Attend and Tell: Neural Image Caption Generation with Visual\n Attention

Abstract

Inspired by recent work in machine translation and object detection, we\nintroduce an attention based model that automatically learns to describe the\ncontent of images. We describe how we can train this model in a deterministic\nmanner using standard backpropagation techniques and stochastically by\nmaximizing a variational lower bound. We also show through visualization how\nthe model is able to automatically learn to fix its gaze on salient objects\nwhile generating the corresponding words in the output sequence. We validate\nthe use of attention with state-of-the-art performance on three benchmark\ndatasets: Flickr8k, Flickr30k and MS COCO.\n

Keywords

Computer scienceBenchmark (surveying)Artificial intelligenceGazeVisualizationObject (grammar)SalientSequence (biology)BackpropagationImage (mathematics)Machine learningArtificial neural networkComputer vision

Related Publications

Object Detection With Deep Learning: A Review

Zhong‐Qiu Zhao , Peng Zheng , Shou-Tao Xu +1 more

Due to object detection's close relationship with video analysis and image understanding, it has attracted much research attention in recent years. Traditional object detection ...

2019 IEEE Transactions on Neural Networks ... 5019 citations

A Sparse Object Category Model for Efficient Learning and Exhaustive Recognition

Rob Fergus , Pietro Perona , Andrew Zisserman

We present a "parts and structure" model for object category recognition that can be learnt efficiently and in a semi-supervised manner: the model is learnt from example images ...

2005 266 citations

Object Detection with Discriminatively Trained Part-Based Models

Pedro F. Felzenszwalb , Ross Girshick , David McAllester +1 more

We describe an object detection system based on mixtures of multiscale deformable part models. Our system is able to represent highly variable object classes and achieves state-...

2009 IEEE Transactions on Pattern Analysis... 9911 citations

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Sangdoo Yun , Dongyoon Han , Sanghyuk Chun +3 more

Regional dropout strategies have been proposed to enhance performance of convolutional neural network classifiers. They have proved to be effective for guiding the model to atte...

2019 4293 citations

Is object localization for free? - Weakly-supervised learning with convolutional neural networks

Maxime Oquab , Léon Bottou , Ivan Laptev +1 more

Successful methods for visual object recognition typically rely on training datasets containing lots of richly annotated images. Detailed image annotation, e.g. by object boundi...

2015 915 citations

Publication Info

Year: 2015
Type: preprint
Citations: 1750
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Show, Attend and Tell: Neural Image Caption Generation with Visual\n Attention

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

1750

OpenAlex

Cite This

APA Style

                            
                                    Kelvin Xu, 
                                
                                    Jimmy Ba, 
                                
                                    Ryan Kiros
                                
                                et al.
                            
                            (2015). 
                            Show, Attend and Tell: Neural Image Caption Generation with Visual\n Attention. 
                            arXiv (Cornell University)
                            
                            .
                            https://doi.org/10.48550/arxiv.1502.03044

Identifiers

DOI: 10.48550/arxiv.1502.03044