Efficient and accurate approximations of nonlinear convolutional networks

Abstract

This paper aims to accelerate the test-time computation of deep convolutional neural networks (CNNs). Unlike existing methods that are designed for approximating linear filters or linear responses, our method takes the nonlinear units into account. We minimize the reconstruction error of the nonlinear responses, subject to a low-rank constraint which helps to reduce the complexity of filters. We develop an effective solution to this constrained nonlinear optimization problem. An algorithm is also presented for reducing the accumulated error when multiple layers are approximated. A whole-model speedup ratio of 4× is demonstrated on a large network trained for ImageNet, while the top-5 error rate is only increased by 0.9%. Our accelerated model has a comparably fast speed as the "AlexNet" [11], but is 4.7% more accurate.

Keywords

SpeedupComputer scienceConvolutional neural networkNonlinear systemComputationAlgorithmConstraint (computer-aided design)Rank (graph theory)Mathematical optimizationArtificial intelligenceMathematicsParallel computing

Affiliated Institutions

Related Publications

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Xiangyu Zhang , Xinyu Zhou , Mengxiao Lin +1 more

We introduce an extremely computation-efficient CNN architecture named ShuffleNet, which is designed specially for mobile devices with very limited computing power (e.g., 10-150...

2018 8394 citations

Network In Network

Min Lin , Qiang Chen , Shuicheng Yan

Abstract: We propose a novel deep network structure called In Network (NIN) to enhance model discriminability for local patches within the receptive field. The conventional con...

2014 arXiv (Cornell University) 1037 citations

A new optimizer using particle swarm theory

R.C. Eberhart , James Kennedy

The optimization of nonlinear functions using particle swarm methodology is described. Implementations of two paradigms are discussed and compared, including a recently develope...

2002 14646 citations

UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation

Zongwei Zhou , Md Mahfuzur Rahman Siddiquee , Nima Tajbakhsh +1 more

The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations...

2019 IEEE Transactions on Medical Imaging 3567 citations

Fully convolutional networks for semantic segmentation

Jonathan Long , Evan Shelhamer , Trevor Darrell

Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, ex...

2015 35498 citations

Publication Info

Year: 2015
Type: preprint
Citations: 289
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Efficient and accurate approximations of nonlinear convolutional networks

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

289

OpenAlex

Cite This

APA Style

                            
                                    Xiangyu Zhang, 
                                
                                    Jianhua Zou, 
                                
                                    Ming Xiang
                                
                                et al.
                            
                            (2015). 
                            Efficient and accurate approximations of nonlinear convolutional networks. 
                            
                            .
                            https://doi.org/10.1109/cvpr.2015.7298809

Identifiers

DOI: 10.1109/cvpr.2015.7298809