FastSAM3D

FastSAM3D accelerates Segment Anything Model (SAM) inference for 3D volumetric medical images through a novel layer-wise progressive distillation scheme and 3D sparse flash attention, enabling low-cost interactive segmentation on commonly used GPU hardware.

Key Features:

• Layer-wise Progressive Distillation: Enables knowledge transfer from a complex 12-layer ViT-B to a lightweight 6-layer ViT-Tiny variant encoder without training from scratch.
• 3D Sparse Flash Attention: Replaces vanilla attention operators to substantially reduce memory requirements and improve parallelization.
• High Inference Speed: Achieves 8 milliseconds per 128 x 128 x 128 3D volumetric image on an NVIDIA A100 GPU.
• Performance Preservation: Maintains segmentation performance without significant decline while achieving substantial speedups over both 2D and 3D SAM approaches.

Scientific Applications:

• Interactive 3D Medical Image Segmentation: Enables truly interactive SAM-based segmentation of 3D volumetric medical images with iterative user prompting and real-time refinement.
• Zero-shot Medical Image Segmentation: Leverages SAM zero-shot generalization capability for segmenting objects of unseen classes and in unseen domains in medical imaging.

Methodology:

FastSAM3D employs a layer-wise progressive distillation scheme for knowledge transfer from a 12-layer ViT-B encoder to a 6-layer ViT-Tiny encoder, combined with 3D sparse flash attention to replace standard attention mechanisms.