SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Alex Wang; Yada Pruksachatkun; Nikita Nangia; Amanpreet Singh; Julian Michael; Felix Hill; Omer Levy; Samuel R. Bowman

doi:10.48550/arxiv.1905.00537

Abstract

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

Keywords

Benchmark (surveying)Computer scienceSet (abstract data type)Metric (unit)SoftwareMachine learningLanguage modelArtificial intelligenceNatural language processingProgramming languageEngineeringOperations management

Affiliated Institutions

Related Publications

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

裕二池谷 , Robert Tinn , Hao Cheng +6 more

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on g...

2021 ACM Transactions on Computing for Hea... 1737 citations

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Alex Wang , Amanpreet Singh , Julian Michael +3 more

Human ability to understand language is general, flexible, and robust. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-o...

2018 3699 citations

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Alex Wang , Amanpreet Singh , Julian Michael +3 more

Human ability to understand language is general, flexible, and robust. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-o...

2018 International Conference on Learning ... 1865 citations

ALBERT: A Lite BERT for Self-supervised Learning of Language\n Representations

Zhenzhong Lan , Mingda Chen , Sebastian Goodman +3 more

Increasing model size when pretraining natural language representations often\nresults in improved performance on downstream tasks. However, at some point\nfurther model increas...

2019 arXiv (Cornell University) 4051 citations

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Mike Lewis , Yinhan Liu , Naman Goyal +5 more

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning...

2020 1222 citations

Publication Info

Year: 2019
Type: preprint
Citations: 985
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

985

OpenAlex

Cite This

APA Style

                            
                                    Alex Wang, 
                                
                                    Yada Pruksachatkun, 
                                
                                    Nikita Nangia
                                
                                et al.
                            
                            (2019). 
                            SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. 
                            arXiv (Cornell University)
                            
                            .
                            https://doi.org/10.48550/arxiv.1905.00537

Identifiers

DOI: 10.48550/arxiv.1905.00537