Abstract
A general method for deriving backpropagation algorithms for networks with recurrent and higher order networks is introduced. The propagation of activation in these networks is determined by dissipative differential equations. The error signal is backpropagated by integrating an associated differential equation. The method is introduced by applying it to the recurrent generalization of the feedforward backpropagation network. The method is extended to the case of higher order networks and to a constrained dynamical system for training a content addressable memory. The essential feature of the adaptive algorithms is that adaptive equation has a simple outer product form. Preliminary experiments suggest that learning can occur very rapidly in networks with recurrent connections. The continuous formalism makes the new approach more suitable for implementation in VLSI.
Keywords
BackpropagationComputer scienceDissipative systemArtificial neural networkGeneralizationFeed forwardFeedforward neural networkRecurrent neural networkDifferential equationAlgorithmArtificial intelligenceTheoretical computer scienceMathematicsControl engineeringMathematical analysisEngineering
Affiliated Institutions
Related Publications
Long Short-Term Memory
Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We brief...
Deep Belief Networks using discriminative features for phone recognition
Deep Belief Networks (DBNs) are multi-layer generative models. They can be trained to model windows of coefficients extracted from speech and they discover multiple layers of fe...
Network In Network
Abstract: We propose a novel deep network structure called In Network (NIN) to enhance model discriminability for local patches within the receptive field. The conventional con...
An application of recurrent nets to phone probability estimation
This paper presents an application of recurrent networks for phone probability estimation in large vocabulary speech recognition. The need for efficient exploitation of context ...
A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Recurrent neural networks (RNNs) have been widely adopted in research areas concerned with sequential data, such as text, audio, and video. However, RNNs consisting of sigma cel...
Publication Info
- Year
- 1987
- Type
- article
- Pages
- 602-611
- Citations
- 123
- Access
- Closed
External Links
Citation Metrics
123
OpenAlex
Cite This
Fernando J. Pineda
(1987).
Generalization of Back propagation to Recurrent and Higher Order Neural Networks.
Neural Information Processing Systems
, 602-611.