Training Recurrent Networks by Evolino

Abstract

In recent years, gradient-based LSTM recurrent neural networks (RNNs) solved many previously RNN-unlearnable tasks. Sometimes, however, gradient information is of little use for training RNNs, due to numerous local minima. For such cases, we present a novel method: EVOlution of systems with LINear Outputs (Evolino). Evolino evolves weights to the nonlinear, hidden nodes of RNNs while computing optimal linear mappings from hidden state to output, using methods such as pseudo-inverse-based linear regression. If we instead use quadratic programming to maximize the margin, we obtain the first evolutionary recurrent support vector machines. We show that Evolino-based LSTM can solve tasks that Echo State nets (Jaeger, 2004a) cannot and achieves higher accuracy in certain continuous function generation tasks than conventional gradient descent RNNs, including gradient-based LSTM.

Keywords

Recurrent neural networkGradient descentMargin (machine learning)Echo state networkComputer scienceMaxima and minimaArtificial intelligenceQuadratic programmingArtificial neural networkAlgorithmMachine learningMathematicsMathematical optimization

MeSH Terms

AnimalsArtificial IntelligenceHumansMemoryModelsNeurologicalNeural NetworksComputerNeuronsNonlinear DynamicsSerial LearningSignal ProcessingComputer-Assisted

Affiliated Institutions

Dalle Molle Institute for Artificial Intelligence Research CH

Related Publications

A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures

Yong Yu , Xiaosheng Si , Changhua Hu +1 more

Recurrent neural networks (RNNs) have been widely adopted in research areas concerned with sequential data, such as text, audio, and video. However, RNNs consisting of sigma cel...

2019 Neural Computation 4793 citations

Learning to forget: continual prediction with LSTM

Felix A. Gers , J. Schmidhuber , Fred Cummins

Long short-term memory (LSTM) can solve many tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks pro...

1999 9th International Conference on Artif... 2376 citations

Recurrent nets that time and count

Felix A. Gers , Jürgen Schmidhuber

The size of the time intervals between events conveys information essential for numerous sequential tasks such as motor control and rhythm detection. While hidden Markov models ...

2000 Proceedings of the IEEE-INNS-ENNS Int... 627 citations

Learning long-term dependencies with gradient descent is difficult

Yoshua Bengio , P. Simard , Paolo Frasconi

Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties hav...

1994 IEEE Transactions on Neural Networks 8111 citations

Recurrent Neural Network Regularization

Wojciech Zaremba , Ilya Sutskever , Oriol Vinyals

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizi...

2014 arXiv (Cornell University) 2273 citations

Publication Info

Year: 2007
Type: article
Volume: 19
Issue: 3
Pages: 757-779
Citations: 251
Access: Closed

External Links

Download PDF (Free) View on DOI.org PubMed Semantic Scholar

Social Impact

Altmetric

Training Recurrent Networks by Evolino

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

251

OpenAlex

Influential

162

CrossRef

Cite This

APA Style

                            
                                
                                    Jürgen Schmidhuber, 
                                
                                    Daan Wierstra, 
                                
                                    Matteo Gagliolo
                                
                                et al.
                            
                            (2007). 
                            Training Recurrent Networks by Evolino. 
                            Neural Computation
                            , 19
                            (3)
                            , 757-779.
                            https://doi.org/10.1162/neco.2007.19.3.757
                        

Identifiers

DOI: 10.1162/neco.2007.19.3.757
PMID: 17298232

Data Quality

Data completeness: 90%