Abstract
High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such “autoencoder” networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.
Keywords
AutoencoderCurse of dimensionalityInitializationGradient descentArtificial neural networkComputer sciencePrincipal component analysisArtificial intelligencePattern recognition (psychology)Layer (electronics)High dimensionalPrincipal (computer security)AlgorithmMaterials scienceNanotechnology
Affiliated Institutions
Related Publications
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the ...
Learning State Space Trajectories in Recurrent Neural Networks
Many neural network learning procedures compute gradients of the errors on the output layer of units after they have settled to their final values. We describe a procedure for f...
Exploring Strategies for Training Deep Neural Networks
Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently...
Understanding the difficulty of training deep feedforward neural networks
Whereas before 2006 it appears that deep multilayer neural networks were not successfully trained, since then several algorithms have been shown to successfully train them, with...
ADADELTA: An Adaptive Learning Rate Method
We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has mi...
Publication Info
- Year
- 2006
- Type
- article
- Volume
- 313
- Issue
- 5786
- Pages
- 504-507
- Citations
- 20153
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
20153
OpenAlex
Cite This
Geoffrey E. Hinton,
Ruslan Salakhutdinov
(2006).
Reducing the Dimensionality of Data with Neural Networks.
Science
, 313
(5786)
, 504-507.
https://doi.org/10.1126/science.1127647
Identifiers
- DOI
- 10.1126/science.1127647