ART-Net

ART-Net performs detection, segmentation, geometric primitive extraction, and 3D pose estimation of surgical tools from laparoscopic images to support computer-assisted laparoscopy.

Key Features:

• Integrated Framework: Unifies tool detection, segmentation, and 3D pose estimation into a single system that conditions 3D pose computation on confirmed detection.
• Generic Requirements: Requires only the assumption of a cylindrical shaft for 3D pose estimation, broadening applicability across surgical instruments without specific CAD models.
• Deep Learning and Algebraic Geometry: Leverages a Convolutional Neural Network (ART-Net) combined with algebraic geometry for accurate tool analysis.
• SIMO Architecture: Employs a Single Input Multiple Output (SIMO) design with one encoder and multiple decoders that concurrently perform detection, segmentation, and extraction of geometric primitives (tool edge-lines, mid-line, tip) to enable fast algebraic 3D pose estimation.
• Full Resolution Feature Map Generator (FrG): Incorporates an FrG module to improve segmentation and geometric primitive extraction accuracy under laparoscopic imaging conditions.

Scientific Applications:

• Tool-Aware Rendering in Augmented Reality (AR): Provides accurate tool detection and segmentation to enable AR overlays that reflect real tool geometry and location.
• Tool-Based 3D Measurement: Enables precise 3D measurements of surgical instruments via extracted geometric primitives for planning and intraoperative assessment.

Methodology:

Implements a Convolutional Neural Network (ART-Net) with a SIMO architecture and a Full Resolution Feature Map Generator (FrG), combining deep learning and algebraic geometry to extract geometric primitives (edge-lines, mid-line, tip) and compute 3D pose via algebraic procedures; evaluated on the EndoVis dataset and newly proposed surgery-video datasets with reported metrics including detection average precision and accuracy, segmentation mean IoU (mIoU) versus FCN and U-Net, geometric primitive extraction errors, and mean absolute errors for 3D pose on animal and patient data.