HMD-EgoPose

HMD-EgoPose estimates six-degree-of-freedom (6DoF) position and orientation of surgical instruments, tissue, and hand grasping from monocular RGB egocentric (head-mounted) video to enable marker-less pose estimation for computer-assisted surgery.

Key Features:

• Single-Shot Learning-Based Framework: Implements a single-shot learning paradigm for end-to-end pose estimation from single RGB frames.
• CNN Backbone and Multi-Scale Feature Extraction: Uses a convolutional neural network (CNN) backbone to extract multi-scale features for downstream pose prediction.
• Joint Pose and Grasp Subnetworks: Employs subnetworks that jointly learn 6DoF pose representations of rigid surgical instruments and the grasping orientation of the user's hand.
• Marker-less Monocular RGB Tracking: Performs marker-less tracking using monocular RGB input and reports state-of-the-art performance on benchmark datasets for hand and instrument pose estimation.
• Low-Latency Streaming and OST-HMD Integration: Provides a low-latency video and data communication pipeline for integration with optical see-through head-mounted displays (OST-HMDs) such as Microsoft HoloLens 2, with a reported round-trip latency of 199.1 ms.
• Robustness to Occlusion and Complex Surfaces: Designed to maintain pose estimation accuracy in the presence of occlusions and complex surface geometry typical of surgical environments.

Scientific Applications:

• Augmented Surgical Guidance: Supplies real-time 6DoF poses of hands and instruments to support augmented-reality guidance in computer-assisted surgery.
• Marker-Free Tracking for Minimally Invasive Procedures: Enables marker-less tracking strategies applicable to minimally invasive and image-guided interventions.
• Egocentric Pose Estimation Research: Serves as a method for benchmarking and advancing HMD-based egocentric pose estimation on public and proprietary datasets.

Methodology:

Single-shot learning-based framework using a CNN backbone for multi-scale feature extraction; subnetworks jointly predict 6DoF poses of rigid instruments and hand grasping orientation from monocular RGB head-mounted video; low-latency video/data pipeline to OST-HMDs with a reported 199.1 ms round-trip latency.