知识蒸馏

我的经验是 碰到一个不熟悉的领域 一条线找找短文综述 就是每篇文章都简单概括一下 看了也不是很明白的那种目的就是找题目 知道哪些工作是知名的 另一条线去找这些知名工作以及这两年的sota 直接开读 读不懂就知乎 再读不懂就换一篇 总有一篇适合入门的领域代表作 入门之后就差不多有感觉 哪些可能有效果了

BERT-Related paper o MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices o XtremeDistil: Multi-stage Distillation for Massive Multilingual Models o BERT-of-Theseus: Compressing BERT by Progressive Module Replacing o Well-Read Students Learn Better: On the Importance of Pre-training Compact Models o MINILMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers ? Multi-Teacher Distillation o Confidence-Aware Multi-Teacher Knowledge Distillation o Reinforced Multi-Teacher Selection for Knowledge Distillation o Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification o Amalgamating Knowledge towards Comprehensive Classification o Born-Again Neural Networks

谢谢老哥，不过我是做cv的[em9] 【 在 xm53247141 的大作中提到: 】 : BERT-Related paper : o MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices : o XtremeDistil: Multi-stage Distillation for Massive Multilingual Models : ............

蒸馏提升效果有限，而且还得看你模型压缩比例还有模型结构设计，前期能够给一个比较好的weight分布，后面还不如针对实际case采集数据，省时省力

rkd