Abstract
Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on the learnable activation and advanced initialization, we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66% [33]). To our knowledge, our result is the first to surpass the reported human-level performance (5.1%, [26]) on this dataset.
Keywords
OverfittingInitializationComputer scienceScratchArtificial intelligenceRectifier (neural networks)Artificial neural networkContextual image classificationParametric statisticsDeep neural networksMachine learningConvolutional neural networkDeep learningPattern recognition (psychology)Image (mathematics)Recurrent neural networkMathematicsTypes of artificial neural networks
Affiliated Institutions
Related Publications
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, ...
A survey on Image Data Augmentation for Deep Learning
Abstract Deep convolutional neural networks have performed remarkably well on many Computer Vision tasks. However, these networks are heavily reliant on big data to avoid overfi...
Network In Network
Abstract: We propose a novel deep network structure called In Network (NIN) to enhance model discriminability for local patches within the receptive field. The conventional con...
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. T...
Learning Deconvolution Network for Semantic Segmentation
We propose a novel semantic segmentation algorithm by learning a deep deconvolution network. We learn the network on top of the convolutional layers adopted from VGG 16-layer ne...
Publication Info
- Year
- 2015
- Type
- article
- Pages
- 1026-1034
- Citations
- 18160
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
18160
OpenAlex
Cite This
Kaiming He,
Xiangyu Zhang,
Shaoqing Ren
et al.
(2015).
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.
, 1026-1034.
https://doi.org/10.1109/iccv.2015.123
Identifiers
- DOI
- 10.1109/iccv.2015.123