Abstract
Recurrent neural networks (RNNs) are a powerful model for sequential data. End-to-end training methods such as Connectionist Temporal Classification make it possible to train RNNs for sequence labelling problems where the input-output alignment is unknown. The combination of these methods with the Long Short-term Memory RNN architecture has proved particularly fruitful, delivering state-of-the-art results in cursive handwriting recognition. However RNN performance in speech recognition has so far been disappointing, with better results returned by deep feedforward networks. This paper investigates deep recurrent neural networks, which combine the multiple levels of representation that have proved so effective in deep networks with the flexible use of long range context that empowers RNNs. When trained end-to-end with suitable regularisation, we find that deep Long Short-term Memory RNNs achieve a test set error of 17.7% on the TIMIT phoneme recognition benchmark, which to our knowledge is the best recorded score.
Keywords
Recurrent neural networkComputer scienceConnectionismTIMITSpeech recognitionArtificial intelligenceDeep learningContext (archaeology)Benchmark (surveying)Time delay neural networkArtificial neural networkHidden Markov modelPattern recognition (psychology)
Affiliated Institutions
Related Publications
A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Recurrent neural networks (RNNs) have been widely adopted in research areas concerned with sequential data, such as text, audio, and video. However, RNNs consisting of sigma cel...
Training Very Deep Networks
Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and tra...
Conditional Random Fields as Recurrent Neural Networks
Pixel-level labelling tasks, such as semantic segmentation, play a central role in image understanding. Recent approaches have attempted to harness the capabilities of deep lear...
An application of recurrent nets to phone probability estimation
This paper presents an application of recurrent networks for phone probability estimation in large vocabulary speech recognition. The need for efficient exploitation of context ...
Sparse Multilayer Perceptron for Phoneme Recognition
This paper introduces the sparse multilayer perceptron (SMLP) which jointly learns a sparse feature representation and nonlinear classifier boundaries to optimally discriminate ...
Publication Info
- Year
- 2013
- Type
- article
- Pages
- 6645-6649
- Citations
- 8613
- Access
- Closed
External Links
Social Impact
Altmetric
PlumX Metrics
Social media, news, blog, policy document mentions
Citation Metrics
8613
OpenAlex
Cite This
Alex Graves,
Abdelrahman Mohamed,
Geoffrey E. Hinton
(2013).
Speech recognition with deep recurrent neural networks.
, 6645-6649.
https://doi.org/10.1109/icassp.2013.6638947
Identifiers
- DOI
- 10.1109/icassp.2013.6638947