Massive Exploration of Neural Machine Translation Architectures

Abstract

Neural Machine Translation (NMT) has shown remarkable progress over the past few years, with production systems now being deployed to end-users. As the field is moving rapidly, it has become unclear which elements of NMT architectures have a significant impact on translation quality. In this work, we present a large-scale analysis of the sensitivity of NMT architectures to common hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on a WMT English to German translation task. Our experiments provide practical insights into the relative importance of factors such as embedding size, network depth, RNN cell type, residual connections, attention mechanism, and decoding heuristics. As part of this contribution, we also release an open-source NMT framework in TensorFlow to make it easy for others to reproduce our results and perform their own experiments.

Keywords

Machine translationComputer scienceHyperparameterArtificial intelligenceHeuristicsMachine learningTranslation (biology)Task (project management)Decoding methodsEmbeddingArtificial neural networkResidualVariance (accounting)USableField (mathematics)Algorithm

Affiliated Institutions

Google (United States) US

Related Publications

Effective Approaches to Attention-based Neural Machine Translation

Thang Luong , Hieu Pham , Christopher D. Manning

An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation.However, the...

2015 8413 citations

Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

Jie Zhou , Ying Cao , Xuguang Wang +2 more

Neural machine translation (NMT) aims at solving machine translation (MT) problems using neural networks and has exhibited promising results in recent years. However, most of th...

2016 Transactions of the Association for C... 223 citations

Attention Is All You Need

Ashish Vaswani , Noam Shazeer , Niki Parmar +5 more

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also co...

2025 6466 citations

Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization

Jia Wu , Xiu Yun Chen , Hao Zhang +3 more

Hyperparameters are important for machine learning algorithms since they directly control the behaviors of training algorithms and have a significant effect on the performance o...

2019 DOAJ (DOAJ: Directory of Open Access ... 1350 citations

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Yonghui Wu , Mike Schuster , Zhifeng Chen +28 more

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based...

2016 arXiv (Cornell University) 5624 citations

Publication Info

Year: 2017
Type: article
Citations: 458
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Massive Exploration of Neural Machine Translation Architectures

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

458

OpenAlex

Cite This

APA Style

                            
                                    Denny Britz, 
                                
                                    Anna Goldie, 
                                
                                    Minh-Thang Luong
                                
                                et al.
                            
                            (2017). 
                            Massive Exploration of Neural Machine Translation Architectures. 
                            
                            .
                            https://doi.org/10.18653/v1/d17-1151

Identifiers

DOI: 10.18653/v1/d17-1151