HRel

HRel computes Mutual Information between convolutional filter activation maps and class labels under the Information Bottleneck framework and prunes low-relevance filters to reduce FLOPs and parameters while preserving classification performance.

Key Features:

• Information Bottleneck and Mutual Information (MI): Employs the Information Bottleneck principle to compute MI between filter activation maps and class labels, defining this quantity as Relevance.
• Relevance-based filter identification: Quantifies filter importance via MI to mark high-Relevance filters as essential and lower-MI filters as candidates for pruning.
• Activation map–label focus: Uniquely evaluates the relationship between activation maps and class labels rather than other MI proxies to determine filter significance.
• Performance metrics: Demonstrated reductions include LeNet-5: 97.98% FLOP reduction with a 0.52% accuracy drop; VGG-16: 94.98% parameter reduction with a 0.36% top-1 accuracy decrease; ResNet-50: 66.42% FLOP pruning with a 1.17% top-5 accuracy decline.
• Information Plane analysis: Analyzes Information Plane dynamics to provide insights into the effects of pruning on network performance.

Scientific Applications:

• CNN architectures: Applied to LeNet-5, VGG-16, ResNet-56, ResNet-110, and ResNet-50 for pruning and efficiency evaluation.
• Datasets: Evaluated on MNIST, CIFAR-10, and ImageNet for performance and generalization assessment.
• Domains: Suitable for image recognition, bioinformatics, and scenarios requiring deployment of CNNs on resource-constrained devices or large-scale data processing.

Methodology:

Compute the mutual information between each filter's activation map and class labels under the Information Bottleneck framework to determine Relevance, then prune filters with lower relevance.