VQAMix

VQAMix enhances medical visual question answering by generating augmented labeled training samples through linear mixing of existing VQA image-question pairs to mitigate limited annotated data.

Key Features:

• Data Augmentation Methodology: Generates additional labeled training samples by linearly combining pairs of existing VQA samples and integrating the mixed samples into visual-language models.
• Learning with Missing Labels (LML): Excludes or mitigates the impact of fabricated mixed pairs that lack meaningful answers by filtering or down-weighting missing labels.
• Learning with Conditional-mixed Labels (LCL): Leverages language-type priors to assign reasonable and category-consistent answers to mixed image-question pairs.
• Performance Improvement: Experimental evaluations report approximately 7% and 5% improvements on VQA-RAD and PathVQA respectively across two different backbones.
• Confidence Calibration and Interpretability: Improves confidence calibration and model interpretability for medical VQA models.

Scientific Applications:

• Medical imaging VQA: Answering clinical questions based on medical images using augmented training data.
• Annotated data augmentation: Generating additional labeled samples from existing VQA datasets to mitigate scarcity of manual annotations.
• Clinical model reliability: Enhancing robustness, confidence calibration, and interpretability of models used in clinical decision contexts.

Methodology:

Linearly combine pairs of existing VQA image-question samples to create mixed samples integrated into visual-language models, apply Learning with Missing Labels (LML) to filter or mitigate missing answers, and apply Learning with Conditional-mixed Labels (LCL) using language-type priors to assign category-consistent labels.