DeepEthogram

DeepEthogram automates frame-by-frame classification of animal behaviors from video to quantify behaviors for neuroscience studies of nervous system function, genetic perturbations, and pharmacological interventions.

Key Features:

• Machine Learning Integration: Uses convolutional neural networks (CNNs) operating on raw pixel values to compute motion, extract features from motion and individual frames, and classify behaviors.
• High Accuracy: Demonstrates greater than 90% single-frame accuracy on videos of flies and mice, comparable to expert human performance.
• Rare Behavior Detection: Effective at identifying extremely rare behaviors within video datasets.
• Minimal Training Data Requirement: Models require minimal training data yet generalize effectively to new videos and subjects.
• Rapid Processing on Standard Hardware: Optimized for efficient execution on common scientific computer hardware to enable rapid analysis.

Scientific Applications:

• Neuroscience research: Quantifies behavior to study nervous system function.
• Behavioral phenotyping of genetic perturbations: Assesses impacts of gene mutations on animal behavior.
• Pharmacological evaluation: Evaluates behavioral effects of pharmacological interventions.
• General automated action detection: Applies to any application requiring automated detection and labeling of actions from video frames.

Methodology:

Implemented in Python and based on convolutional neural networks applied to raw pixel values to compute motion, extract features from motion and individual frames, and classify each frame into predefined behaviors; models require minimal training data and were evaluated for single-frame accuracy (>90%) on flies and mice.