Relational Knowledge Distillation | RDL Research Database

Abstract

Knowledge distillation aims at transferring knowledge acquired in one model (a teacher) to another model (a student) that is typically smaller. Previous approaches can be expressed as a form of training the student to mimic output activations of individual data examples represented by the teacher. We introduce a novel approach, dubbed relational knowledge distillation (RKD), that transfers mutual relations of data examples instead. For concrete realizations of RKD, we propose distance-wise and angle-wise distillation losses that penalize structural differences in relations. Experiments conducted on different tasks show that the proposed method improves educated student models with a significant margin. In particular for metric learning, it allows students to outperform their teachers' performance, achieving the state of the arts on standard benchmark datasets.

Keywords

Margin (machine learning)DistillationBenchmark (surveying)Metric (unit)Computer scienceStatistical relational learningArtificial intelligenceMachine learningRelational databaseData miningEngineering

Affiliated Institutions

Related Publications

Knowledge Distillation: A Survey

Jianping Gou , Baosheng Yu , Stephen J. Maybank +1 more

In recent years, deep neural networks have been successful in both industry and academia, especially for computer vision tasks. The great success of deep learning is mainly due ...

2021 International Journal of Computer Vision 2894 citations

TinyBERT: Distilling BERT for Natural Language Understanding

Xiaoqi Jiao , Yichun Yin , Lifeng Shang +5 more

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usuall...

2020 Findings of the Association for Compu... 1491 citations

Deep Mutual Learning

Ying Zhang , Tao Xiang , Timothy M. Hospedales +1 more

Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful la...

2018 2018 IEEE/CVF Conference on Computer ... 1668 citations

Improving Transferability of Adversarial Examples With Input Diversity

Cihang Xie , Zhishuai Zhang , Yuyin Zhou +4 more

Though CNNs have achieved the state-of-the-art performance on various vision tasks, they are vulnerable to adversarial examples --- crafted by adding human-imperceptible perturb...

2019 1098 citations

Classification in Networked Data: A Toolkit and a Univariate Case Study

Sofus A. Macskassy , Foster Provost

This paper is about classifying entities that are interlinked with entities for which the class is known. After surveying prior work, we present NetKit, a modular toolkit for cl...

2007 Journal of Machine Learning Research 447 citations

Publication Info

Year: 2019
Type: article
Pages: 3962-3971
Citations: 1437
Access: Closed

External Links

Download PDF (Free) View on DOI.org arXiv Semantic Scholar

Social Impact

Altmetric

Relational Knowledge Distillation

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

1437

OpenAlex

194

Influential

1183

CrossRef

Cite This

APA Style

                            
                                    Wonpyo Park, 
                                
                                    Dongju Kim, 
                                
                                    Yan Lu
                                
                                et al.
                            
                            (2019). 
                            Relational Knowledge Distillation. 
                            2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
                            
                            , 3962-3971.
                            https://doi.org/10.1109/cvpr.2019.00409

Identifiers

DOI: 10.1109/cvpr.2019.00409
arXiv: 1904.05068

Data Quality

Data completeness: 84%