Abstract
On-line first-order backpropagation is sufficiently fast and effective for many large-scale classification problems but for very high precision mappings, batch processing may be the method of choice. This paper reviews first- and second-order optimization methods for learning in feedforward neural networks. The viewpoint is that of optimization: many methods can be cast in the language of optimization techniques, allowing the transfer to neural nets of detailed results about computational complexity and safety procedures to ensure convergence and to avoid numerical problems. The review is not intended to deliver detailed prescriptions for the most appropriate methods in specific applications, but to illustrate the main characteristics of the different methods and their mutual relations.
Keywords
Computer scienceBackpropagationArtificial neural networkConvergence (economics)Gradient descentArtificial intelligenceFeedforward neural networkMathematical optimizationFeed forwardAlgorithmMathematics
Affiliated Institutions
Related Publications
An analysis of noise in recurrent neural networks: convergence and generalization
Concerns the effect of noise on the performance of feedforward neural nets. We introduce and analyze various methods of injecting synaptic noise into dynamically driven recurren...
Training Recurrent Networks by Evolino
In recent years, gradient-based LSTM recurrent neural networks (RNNs) solved many previously RNN-unlearnable tasks. Sometimes, however, gradient information is of little use for...
Optimization for training neural nets
Various techniques of optimizing criterion functions to train neural-net classifiers are investigated. These techniques include three standard deterministic techniques (variable...
A direct adaptive method for faster backpropagation learning: the RPROP algorithm
A learning algorithm for multilayer feedforward networks, RPROP (resilient propagation), is proposed. To overcome the inherent disadvantages of pure gradient-descent, RPROP perf...
Generalization of Back propagation to Recurrent and Higher Order Neural Networks
A general method for deriving backpropagation algorithms for networks with recurrent and higher order networks is introduced. The propagation of activation in these networks is ...
Publication Info
- Year
- 1992
- Type
- article
- Volume
- 4
- Issue
- 2
- Pages
- 141-166
- Citations
- 1187
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
1187
OpenAlex
Cite This
Roberto Battiti
(1992).
First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method.
Neural Computation
, 4
(2)
, 141-166.
https://doi.org/10.1162/neco.1992.4.2.141
Identifiers
- DOI
- 10.1162/neco.1992.4.2.141