FFSD

FFSD performs online knowledge distillation by fusing feature maps from multiple student models and applying self-distillation to improve the leader student's generalization while maintaining deployment efficiency.

Key Features:

• Student grouping: Divides student models into a leader student set and a common student set to structure knowledge transfer.
• Feature fusion module: Amalgamates feature maps from all common students into a single fused feature map used to enhance the leader student.
• Enhancement strategy: Introduces variability among student models to augment diversity and improve generalization.
• Self-distillation module: Transforms deeper-layer feature maps into shallower ones so shallower layers mimic modified deeper representations.
• Deployment efficiency: Uses only the trained leader student for deployment to avoid additional storage and inference costs from multiple models.
• Experimental validation: Demonstrated superior performance over existing knowledge distillation methods on CIFAR-100 and ImageNet.

Scientific Applications:

• Online knowledge distillation research: Provides a framework for studying knowledge transfer among multiple student networks in an online setting.
• Image classification benchmarking: Applied to evaluate and improve student model performance on datasets such as CIFAR-100 and ImageNet.
• Efficient deployment studies: Enables evaluation of strategies that retain model performance while minimizing storage and inference overhead compared to ensembles.

Methodology:

Divide students into leader and common sets; fuse feature maps from common students into a single fused feature map via a feature fusion module; apply an enhancement strategy to increase student variability; use a self-distillation module to transform deeper-layer feature maps into shallower ones; after training, deploy only the leader student.