Abstract
We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.
Keywords
MNIST databaseStochastic gradient descentComputer scienceOverhead (engineering)Artificial intelligenceGradient descentHyperparameterDimension (graph theory)Word error rateTask (project management)Machine learningSelection (genetic algorithm)Scale (ratio)Pattern recognition (psychology)Deep learningArtificial neural networkMathematics
Related Publications
Large-Scale Machine Learning with Stochastic Gradient Descent
During the last decade, the data sizes have grown faster than the speed of processors. In this context, the capabilities of statistical machine learning methods is limited by th...
Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization
Hyperparameters are important for machine learning algorithms since they directly control the behaviors of training algorithms and have a significant effect on the performance o...
Optimization for training neural nets
Various techniques of optimizing criterion functions to train neural-net classifiers are investigated. These techniques include three standard deterministic techniques (variable...
SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent
The SGD-QN algorithm is a stochastic gradient descent algorithm that makes careful use of second-order information and splits the parameter update into independently scheduled c...
Large Scale Distributed Deep Networks
Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance. In this paper, we consider t...
Publication Info
- Year
- 2012
- Type
- preprint
- Citations
- 5515
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
5515
OpenAlex
Cite This
Matthew D. Zeiler
(2012).
ADADELTA: An Adaptive Learning Rate Method.
arXiv (Cornell University)
.
https://doi.org/10.48550/arxiv.1212.5701
Identifiers
- DOI
- 10.48550/arxiv.1212.5701