[Paper Review] Decoupled Knowledge Distillation

This is a review of

"Decoupled Knowledge Distillation"
presented at CVPR 2022.

Introduction

Types of Knowledge Distillation

Logits-based method

• $(+)$ Computational and storage cost ↓
• $(-)$ Unsatisfactory performance

Feature-based method

• $(+)$ Superior performance
• $(-)$ Extra computational cost and storage usage

∴ Potential of logit distillation is limited.

 

Decoupled Knowledge Distillation

Target classifiation knowledge distillation $($TCKD$)$

• Binary logit distillation

Non-target classification knowledge distillation $($NCKD$)$

• Knowledge among non-target logits

 

 

Method

Reformulation

$$p_i=\frac{\exp \left(z_i\right)}{\sum _{j=1}^C\exp \left(z_j\right)},\mathit{p}=[p_1,p_2,\dots ,p_t,\dots ,p_C]\in {\mathbb{R}}^{1\times C}$$

Binary probabilities

$$\begin{gathered} p_t=\frac{\exp \left(z_t\right)}{\sum _{j=1}^C\exp \left(z_j\right)},p_{\mathrm{\setminus }t}=\frac{\sum _{k=1,k\ne t}^C\exp \left(z_k\right)}{\sum _{j=1}^C\exp \left(z_j\right)}, \\ \mathit{b}=[p_t,p_{\mathrm{\setminus }t}]\in {\mathbb{R}}^{1\times 2} \end{gathered}$$

Probabilities among non-target classes

$${\hat{p}}_i=\frac{\exp \left(z_i\right)}{\sum _{j=1,j\ne t}^C\exp \left(z_j\right)},\hat{\mathit{p}}=[{\hat{p}}_1,\dots ,{\hat{p}}_{t-1},{\hat{p}}_{t+1},\dots ,{\hat{p}}_C]\in {\mathbb{R}}^{1\times (C-1)}$$

Vanilla KD

$$\begin{array}{rl}\mathrm{KD}& =\mathrm{KL}({\mathbf{p}}^{\mathcal{T}}\Vert {\mathbf{p}}^{\mathcal{S}})\\ & =p_t^{\mathcal{T}}\log \left(\frac{p_t^{\mathcal{T}}}{p_t^{\mathcal{S}}}\right)+\sum _{i=1,i\ne t}^C{p}_i^{\mathcal{T}}\log \left(\frac{p_i^{\mathcal{T}}}{p_i^{\mathcal{S}}}\right)\end{array}$$

$$\begin{array}{rl}\mathrm{KD}& =p_t^{\mathcal{T}}\log \left(\frac{p_t^{\mathcal{T}}}{p_t^{\mathcal{S}}}\right)+p_{\mathrm{\setminus }t}^{\mathcal{T}}\sum _{i=1,i\ne t}^C{\hat{p}}_i^{\mathcal{T}}(\log \left(\frac{{\hat{p}}_i^{\mathcal{T}}}{{\hat{p}}_i^{\mathcal{S}}}\right)+\log \left(\frac{p_{\mathrm{\setminus }t}^{\mathcal{T}}}{p_{\mathrm{\setminus }t}^{\mathcal{S}}}\right))\\ & =p_t^{\mathcal{T}}\log \left(\frac{p_t^{\mathcal{T}}}{p_t^{\mathcal{S}}}\right)+p_{\mathrm{\setminus }t}^{\mathcal{T}}\log \left(\frac{p_{\mathrm{\setminus }t}^{\mathcal{T}}}{p_{\mathrm{\setminus }t}^{\mathcal{S}}}\right)+p_{\mathrm{\setminus }t}^{\mathcal{T}}\sum _{i=1,i\ne t}^C{\hat{p}}_i^{\mathcal{T}}\log \left(\frac{{\hat{p}}_i^{\mathcal{T}}}{{\hat{p}}_i^{\mathcal{S}}}\right).\end{array}$$

$$\mathrm{KD}=\mathrm{KL}({\mathbf{b}}^{\mathcal{T}}\Vert {\mathbf{b}}^{\mathcal{S}})+(1-p_t^{\mathcal{T}})\mathrm{KL}({\hat{\mathbf{p}}}^{\mathcal{T}}\Vert {\hat{\mathbf{p}}}^{\mathcal{S}})$$

$$\therefore \mathrm{KD}=\mathrm{TCKD}+(1-p_t^{\mathcal{T}})\mathrm{NCKD}$$

• While NCKD focus on the knowledge among non-target classes, TCKD focus on the knowledge related to the target class.

 

Effects of TCKD and NCKD

• Solely applying TCKD is unhelpful or even harmful.
• Performane of NCKD are comparable and even better than vanilla KD

∴ Target-class related knwoedge could not be as important as knolwedge among non-target classes.

∴ The more difficult the training data is, the more benefits TCKD could provide.

 

Decoupled Knowledge Distillation

$$\mathrm{KD}=\mathrm{TCKD}+(1-p_t^{\mathcal{T}})\mathrm{NCKD}$$

• NCKD loss is coupled with $(1-p_t^{\mathcal{T}})$: More confident predictions results in smaller NCKD weights. $($highly suppressed weights$)$
• Weights of NCKD and TCKD are coupled.

$$\therefore \mathrm{DKD}=\alpha \mathrm{TCKD}+\beta \mathrm{NCKD}$$

 

 

Experiments

Ablation: $\alpha$ and $\beta$

 

CIFAR-100

 

ImageNet

 

COCO

 

 

Conclusions

• Reformulation of vanilla KD loss into two parts: TCKD and NCKD
• Decoupled Knowledge Distillation overcomes limitation of coupled formulation or effective transfer.
• Significant improvements on various datasets