SCNET

SCNET performs semantic-level fusion of gastrointestinal imaging and textual medical records to improve screening for upper gastrointestinal (UGI) cancer.

Key Features:

• Semantic-level multimodal fusion: Fuses gastrointestinal image recognition outputs with textual medical record features at the semantic level to combine complementary information across modalities.
• Gastrointestinal image recognition: Processes gastroscope and other gastrointestinal images to extract spatial image features for downstream fusion.
• Textual medical record processing: Extracts semantic features from clinical text to provide complementary contextual information to imaging data.
• Cross-modal feature correlation: Correlates textual information with the spatial structure of image features to derive high-level multimodal representations.
• Effective feature channel identification: Identifies effective feature channels that enhance the correlation and information transfer between modalities.
• Empirical performance improvement: Demonstrated an average screening performance improvement of 4.01% compared with existing single-modality or state-of-the-art methods.

Scientific Applications:

• UGI cancer screening: Enhances screening workflows for upper gastrointestinal cancer by integrating imaging and clinical text data.
• Improved lesion detection: Increases detection accuracy of UGI lesions relative to single-modality gastroscope imaging.
• Multimodal diagnostic research: Supports research into semantic-level multimodal data fusion methods for medical diagnostics.

Methodology:

Integrates gastrointestinal image recognition and textual medical record processing flows, identifies effective feature channels to enhance cross-modal correlation, and fuses multimodal inputs at the semantic level by correlating textual information with the spatial structure of image features.