Knowledge Distillation

Copy the Same, Distill the Difference: A Smart Initialization Paradigm for Linear Vision Transformers

複製相同、蒸餾相異：為線性視覺 Transformer 注入預訓練威力的全新初始化策略

This study proposes CSDD, a novel initialization strategy that copies operator-agnostic MLP weights and distills operator-specific attention routing to transfer pre-trained Softmax ViT capabilities into Linear ViTs.