FusionM4Net

FusionM4Net performs multi-label skin lesion classification using a two-stage multi-modal learning algorithm that integrates clinical images, dermoscopy images, and patient meta-data to improve diagnostic accuracy.

Key Features:

• Multi-Stage Multi-Modal Learning: FusionM4Net employs a two-stage approach combining feature-level and decision-level fusion for multi-modal data integration.
• FusionNet (First Stage): FusionNet performs feature-level integration by extracting and merging features from clinical images and dermoscopy images.
• Fusion Scheme 1: Fusion Scheme 1 applies decision-level information fusion to the outputs of FusionNet to produce initial multi-label predictions.
• Second Stage with Fusion Scheme 2 and SVM: In the second stage, Fusion Scheme 2 integrates patient meta-data with first-stage multi-label predictions and trains a Support Vector Machine (SVM) cluster to refine final diagnostic outputs.
• Prediction Combination: Final diagnostic outputs are obtained by combining predictions from both stages.
• Comprehensive Data Utilization: The method integrates clinical images, dermoscopy images, decision-level fusion, and patient meta-data across stages to utilize all available modalities.
• Performance Evaluation: Evaluated on the seven-point checklist dataset, FusionM4Net-FS achieved 75.7% average accuracy for multi-classification and 74.9% diagnostic accuracy without meta-data, while FusionM4Net-SS with meta-data achieved 77.0% average accuracy and 78.5% diagnostic accuracy.
• Robustness to Label Imbalance: Incorporation of patient meta-data in the second stage improves performance on label-imbalanced datasets.

Scientific Applications:

• Dermatological Diagnostic Support: Provides multi-label skin disease classification for dermatology by integrating clinical images, dermoscopy images, and patient meta-data.
• Multi-Modal Fusion Research: Serves as a framework for research into feature-level and decision-level fusion strategies in deep learning for medical imaging.
• Imbalanced Dataset Evaluation: Enables evaluation and development of methods robust to label-imbalanced skin disease classification datasets.

Methodology:

FusionM4Net uses a two-stage computational pipeline: a first stage (FusionNet) that extracts and merges features from clinical and dermoscopy images followed by Fusion Scheme 1 decision-level fusion, and a second stage that integrates patient meta-data via Fusion Scheme 2 to train an SVM cluster and combine stage predictions.