TransVOD

TransVOD implements end-to-end video object detection using spatial-temporal Transformer architectures to fuse object queries across frames and capture temporal dynamics for improved detection accuracy.

Key Features:

• Streamlined Pipeline: Implements an end-to-end VOD pipeline using spatial-temporal Transformers that removes reliance on hand-crafted components such as optical flow models and relation networks and integrates with DETR frameworks.
• Temporal Query Encoder (TQE): Fuses object queries across frames to enhance temporal tracking and detection.
• Temporal Deformable Transformer Decoder (TDTD): Generates per-frame detection results by processing spatial object queries and feature memories.
• Elimination of Post-Processing: Adopts the DETR object query design to negate the need for post-processing methods such as Seq-NMS.
• Enhanced Performance: Improves mean Average Precision (mAP) by 3%–4% over baseline deformable DETR on the ImageNet VID dataset.
• TransVOD++: Incorporates object-level information into object queries via dynamic convolution, achieving 90.0% mAP on ImageNet VID.
• TransVOD Lite: Models entire video clips as outputs to prioritize speed, achieving 83.7% mAP at approximately 30 FPS on a single V100 GPU.

Scientific Applications:

• Surveillance systems: Continuous object detection and temporal tracking in surveillance video streams.
• Autonomous vehicle navigation: Dynamic object detection for vehicle perception and scene understanding.
• Advanced computer vision in bioinformatics: Analysis of dynamic visual data for bioinformatics-related computer vision tasks.

Methodology:

Leverages spatial-temporal Transformer architectures comprising a Temporal Query Encoder (TQE) that fuses object queries across frames and a Temporal Deformable Transformer Decoder (TDTD) that processes spatial object queries and feature memories, integrates DETR object query design, and uses dynamic convolution in TransVOD++ or models entire video clips as outputs in TransVOD Lite.