A Natural Language-Driven Animal Pose Estimation Module Based on Markerless, Zero-Shot Methods
1. Introduction
In recent years, advances in computer vision and deep learning have greatly propelled the analysis of animal behavior [1]. Extracting high-fidelity kinematic data of animals during experiments is foundational for identifying complex behavioral patterns and exploring the neural correlates of behavior. Although advanced frameworks like DeepLabCut [2] and SLEAP [3], along with recently emerged methods for 3D pose estimation and behavioral mapping in single-animal and social contexts [4, 5], provide accurate pose estimation capabilities, researchers still face two major pain points in practical deployment and application. First, the configuration and technical documentation of these computational tools are often too complex, making them highly unfriendly to life science researchers without a computer science background. Second, cloud-based AI agent services attempting to lower the operational threshold often require users to upload gigabytes of raw experimental videos. This is not only constrained by high bandwidth costs and transmission latency but also poses data privacy risks.
To address these application bottlenecks, we developed a dedicated, fully automated animal pose estimation workflow (ethoclaw-animal-pose-estimation). The primary innovation of this module lies in the introduction of a conversational analysis paradigm. Users can drive the system using natural language without writing code or consulting obscure computer documentation; meanwhile, it retains heavy-load video analysis tasks entirely on local workstations, fully unleashing local computing potential. In addition, by seamlessly integrating the SuperAnimal method [6] under the DeepLabCut framework, this module cleverly leverages the existing open-source ecosystem, eliminating tedious model training steps and flexibly enabling out-of-the-box, zero-shot, high-dimensional skeleton extraction.
2. Methods
This pipeline is designed for markerless, multi-node, high-dimensional pose analysis. Its core workflow encompasses underlying deep learning inference calls, localized asynchronous computational acceleration, and strict data quality control.
2.1 Zero-Shot Inference Integration with DeepLabCut-SuperAnimal
Rather than developing the underlying pose estimation algorithm from scratch, this module deeply integrates and schedules the outstanding SuperAnimal method [6] from the DeepLabCut ecosystem as its core inference engine. Targeting the classic top-down mouse experimental scenario, the system invokes this method to achieve a two-stage inference process without manual annotation:
- Object Detection: A Faster R-CNN detector with a ResNet-50 FPN v2 backbone is used to localize the animal subject. The system sets a strict bounding box detection threshold (>0.9) to automatically filter out background interference.
- High-Resolution Pose Estimation: The localized tensors are fed into a High-Resolution Network (HRNet_w32). Unlike traditional heavily downsampled convolutional networks, HRNet maintains high-resolution representations throughout the forward pass, enabling the precise extraction of 27 fine-grained anatomical keypoints (covering the head, torso, limbs, and tail nodes).
2.2 Multi-Process Architecture Maximizing Local Computing Power
To ensure a smooth analysis experience on local hardware and maximize the efficacy of external algorithms, we implemented a multi-process parallelization strategy: the system asynchronously distributes video frame preprocessing tasks (decoding, resizing/padding, normalization) across multiple local CPU cores, then packages the processed data into tensors and loads them onto the local GPU for centralized inference. It should be noted that, according to official recommendations, computers deploying this workflow are best equipped with an Nvidia GPU to fully utilize the deep learning inference performance of the SuperAnimal method. This overlapping design of CPU preprocessing and GPU inference maximizes the utilization of local computing hardware.
2.3 Confidence Gating Mechanism
The system extracts spatial coordinates from 2D probability heatmaps and enforces strict confidence gating to ensure data purity. The network assigns a prediction confidence score p to each identified keypoint. Any spatial coordinate with p < 0.8 is automatically deemed invalid and encoded as NaN.
3. Results
Through practical deployment in standardized experimental scenarios, this automated pipeline demonstrates the following outstanding performance:
- Conversational Drive and Full Localization: Experimenters can issue commands via natural language, and the system automatically schedules local computing power (e.g., a local workstation equipped with conventional acceleration hardware) to complete the entire process from video reading to skeleton extraction. This entirely avoids the latency risks of cloud uploads and greatly enhances the user experience.
- Zero-Shot, Annotation-Free Extraction: Relying on the powerful generalization capabilities of the integrated DeepLabCut-SuperAnimal method, we successfully eliminated the time-consuming process of manual video frame annotation and iterative model training. It stably outputs accurate 27-point skeleton coordinates directly from raw videos (specifically including: nose, left_ear, right_ear, left_ear_tip, right_ear_tip, left_eye, right_eye, head_midpoint, neck, mid_back, mouse_center, mid_backend, mid_backend2, mid_backend3, left_midside, right_midside, left_shoulder, right_shoulder, left_hip, right_hip, tail_base, tail_end, tail1, tail2, tail3, tail4, tail5).
- Highly Compatible Standardized Output: The module automatically converts the high-fidelity coordinate sets into HDF5/CSV formats compatible with industry standards (the DeepLabCut ecosystem), achieving seamless integration with downstream trajectory heatmap generation, kinematic parameter calculation, and clustering analysis.
4. Implementation
The integrated pose estimation module proposed in this study has been deployed as a core skill (ethoclaw-animal-pose-estimation) within the open-source, AI-driven neuroethology workflow platform EthoClaw.
- Project Repository: https://github.com/penciler-star/EthoClaw (under the MIT License)
- Skill Code Path:
skills/ethoclaw-animal-pose-estimation/
5. Conclusion
The automated pose estimation pipeline presented in this paper successfully addresses the "steep learning curve" of complex computational tools and the "cloud transmission bottlenecks" of large-volume videos faced by neuroscientists, by combining conversational AI scheduling with localized, efficient processing strategies. This module flexibly integrates the SuperAnimal method [6] under the DeepLabCut framework, achieving truly annotation-free, zero-shot inference in a local environment. This not only eliminates technical barriers for researchers regarding code writing and data cleaning, but also significantly enhances the reproducibility [7] of behavioral experimental results through strict parameter fixation and intervention-free processes, making it an ideal infrastructure for modern high-throughput neuroethological research.
6. References
[1] Pereira, T. D., Shaevitz, J. W., & Murthy, M. (2020). Quantifying behavior to understand the brain. Nature Neuroscience, 23(12), 1537-1549.
[2] Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., & Bethge, M. (2018). DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21(9), 1281-1289.
[3] Pereira, T. D., Aldarondo, D. E., Willmore, L., Kislin, M., Wang, S. S., Murthy, M., & Shaevitz, J. W. (2022). Fast animal pose estimation using deep neural networks. Nature Methods, 16(1), 117-125.
[4] Huang, K., Han, Y., Chen, K., Pan, H., Zhao, G., Yi, W., Li, X., Liu, S., Wei, P., & Wang, L. (2021). A hierarchical 3D-motion learning framework for animal spontaneous behavior mapping. Nature Communications, 12(1), 2784.
[5] Han, Y., Huang, K., Chen, K., Pan, H., Ju, F., Long, Y., Gao, G., Wu, R., Wang, A., Wang, A., Wang, L., & Wei, P. (2022). MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice. Neuroscience Bulletin, 38(3), 303-317.
[6] Ye, S., Filippova, A., Lauer, J., Schneider, S., Vidal, M., Qiu, T., Mathis, A., & Mathis, M. W. (2024). SuperAnimal pretrained pose estimation models for behavioral analysis. Nature Communications, 15(1), 5165.
[7] Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452-454.
Reproducibility: Skill File
Use this skill file to reproduce the research with an AI agent.
---
name: ethoclaw-animal-pose-estimation
description: Animal pose estimation using DeepLabCut SuperAnimal pre-trained models, supporting analysis of local videos and images.
homepage: https://github.com/DeepLabCut/DeepLabCut
metadata:
{
"openclaw":
{
"emoji": "🐭",
"requires": { "python": ["deeplabcut"] },
"install":
[
{
"id": "pip",
"kind": "pip",
"package": "deeplabcut",
"version": "--pre",
"label": "Install DeepLabCut with model zoo support",
},
],
},
}
---
# Animal Pose Estimation (DeepLabCut SuperAnimal)
Use DeepLabCut's SuperAnimal pre-trained models to perform animal pose estimation (keypoint detection) on local videos or images.
## Supported Models
- **superanimal_topviewmouse**: Top-view mouse model
- **superanimal_quadruped**: Quadruped animal model
## Supported Model Architectures
- **hrnet_w32**: HRNet w32 (recommended, higher accuracy)
- **resnet_50**: ResNet-50 (faster speed)
## Supported Detectors
- **fasterrcnn_resnet50_fpn_v2**: Faster R-CNN (recommended, higher accuracy)
- **fasterrcnn_mobilenet_v3_large_fpn**: MobileNet (faster speed)
## Quick Start
### Analyze a Single Video
```python
import deeplabcut
from pathlib import Path
# Video path
video_path = "/path/to/your/video.mp4"
# Optional: specify output directory, if not specified uses the same directory as the video
output_folder = "/path/to/output" # Optional
# Run SuperAnimal analysis
deeplabcut.video_inference_superanimal(
videos=[video_path],
superanimal_name="superanimal_topviewmouse",
model_name="hrnet_w32",
detector_name="fasterrcnn_resnet50_fpn_v2",
video_adapt=False,
max_individuals=1,
pseudo_threshold=0.1,
bbox_threshold=0.9,
dest_folder=output_folder # If None, results are saved to the same directory as the video
)
```
### Analyze Multiple Images
```python
from deeplabcut.pose_estimation_pytorch.apis import superanimal_analyze_images
from pathlib import Path
# List of image paths
image_paths = [
"/path/to/image1.jpg",
"/path/to/image2.jpg",
]
# Optional: specify output directory
output_folder = "/path/to/output" # Optional
# Run SuperAnimal image analysis
superanimal_analyze_images(
images=image_paths,
superanimal_name="superanimal_topviewmouse",
model_name="hrnet_w32",
detector_name="fasterrcnn_resnet50_fpn_v2",
max_individuals=1,
dest_folder=output_folder # If None, results are saved to the same directory as the images
)
```
### Batch Analysis of Multiple Videos
```python
import deeplabcut
from pathlib import Path
# Multiple video paths
video_paths = [
"/path/to/video1.mp4",
"/path/to/video2.mp4",
"/path/to/video3.avi",
]
# Specify output directory (all video results will be saved here)
output_folder = "/path/to/output"
# Batch analysis
deeplabcut.video_inference_superanimal(
videos=video_paths,
superanimal_name="superanimal_topviewmouse",
model_name="hrnet_w32",
detector_name="fasterrcnn_resnet50_fpn_v2",
video_adapt=False,
max_individuals=1,
dest_folder=output_folder
)
```
## Parameter Description
### Video Analysis Parameters
| Parameter | Type | Default | Description |
| ------------------ | ----- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `videos` | list | Required | List of video file paths |
| `superanimal_name` | str | Required | SuperAnimal model name |
| `model_name` | str | "hrnet_w32" | Pose estimation model name |
| `detector_name` | str | "fasterrcnn_resnet50_fpn_v2" | Object detector name |
| `video_adapt` | bool | False | Whether to enable video adaptation. **Disabled by default, only enable if specifically requested by the user** |
| `max_individuals` | int | 1 | Maximum number of animals to detect. **Default is 1, only increase if specifically requested by the user** |
| `pseudo_threshold` | float | 0.1 | Pseudo-label threshold |
| `bbox_threshold` | float | 0.9 | Bounding box detection threshold |
| `detector_epochs` | int | 1 | Number of detector training epochs |
| `pose_epochs` | int | 1 | Number of pose estimation training epochs |
| `dest_folder` | str | None | Result output directory, if None saves to the same directory as the video |
| `scale_list` | range | None | Multi-scale test list, e.g., `range(200, 600, 50)`. **Disabled by default, only enable if specifically requested by the user** |
### Image Analysis Parameters
| Parameter | Type | Default | Description |
| ------------------ | ---- | ---------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `images` | list | Required | List of image file paths |
| `superanimal_name` | str | Required | SuperAnimal model name |
| `model_name` | str | "hrnet_w32" | Pose estimation model name |
| `detector_name` | str | "fasterrcnn_resnet50_fpn_v2" | Object detector name |
| `max_individuals` | int | 1 | Maximum number of animals to detect. **Default is 1, only increase if specifically requested by the user** |
| `dest_folder` | str | None | Result output directory, if None saves to the same directory as the images |
## Output Results
### Video Analysis Output
After analysis is complete, the following files will be generated in the specified directory (or the same directory as the video):
- **`video_nameDLC_snapshot-....h5`**: HDF5 file containing keypoint coordinate data
- **`video_nameDLC_snapshot-....csv`**: CSV file containing keypoint coordinate data (easy to view)
- **`video_nameDLC_snapshot-....pickle`**: Pickle file containing complete analysis results
- **`video_name_labeled.mp4`** (optional): Visualization video with keypoint annotations
### Image Analysis Output
- **`image_nameDLC_snapshot-....h5`**: Keypoint coordinate data
- **`image_nameDLC_snapshot-....csv`**: Keypoint coordinate CSV
- **`image_name_labeled.png`** (optional): Visualization image with keypoint annotations
### Result Data Structure
CSV/H5 files contain the following columns:
- `scorer`: Model name
- `individuals`: Animal individual ID (e.g., individual1, individual2...)
- `bodyparts`: Body part names (e.g., nose, tailbase, leftear, etc.)
- `coords`: Coordinate type (x, y, likelihood)
## Complete Example Scripts
### Complete Video Analysis Example
```python
import os
from pathlib import Path
import deeplabcut
# ==================== User Configuration Area ====================
# Video path (Required)
video_path = "/path/to/your/video.mp4"
# Output directory (Optional, set to None to use the same directory as the video)
output_folder = None # Example: "/path/to/output"
# SuperAnimal model selection
superanimal_name = "superanimal_topviewmouse" # or "superanimal_quadruped"
# Model architecture selection
model_name = "hrnet_w32" # or "resnet_50"
# Detector selection
detector_name = "fasterrcnn_resnet50_fpn_v2" # or "fasterrcnn_mobilenet_v3_large_fpn"
# Number of animals (default is 1, only increase if specifically requested by the user)
max_individuals = 1
# Whether to use multi-scale testing (disabled by default, only enable if specifically requested by the user)
use_multiscale = False
scale_list = range(200, 600, 50) if use_multiscale else None
# ==================== Run Analysis ====================
# Verify video exists
if not os.path.exists(video_path):
raise FileNotFoundError(f"Video file does not exist: {video_path}")
# Determine output directory
if output_folder is None:
output_folder = str(Path(video_path).parent)
# Ensure output directory exists
os.makedirs(output_folder, exist_ok=True)
print(f"Starting video analysis: {video_path}")
print(f"Output directory: {output_folder}")
print(f"Using model: {superanimal_name}")
# Run analysis
kwargs = {
"videos": [video_path],
"superanimal_name": superanimal_name,
"model_name": model_name,
"detector_name": detector_name,
"video_adapt": False,
"max_individuals": max_individuals,
"pseudo_threshold": 0.1,
"bbox_threshold": 0.9,
"detector_epochs": 1,
"pose_epochs": 1,
"dest_folder": output_folder,
}
if scale_list is not None:
kwargs["scale_list"] = scale_list
deeplabcut.video_inference_superanimal(**kwargs)
print("Analysis complete!")
print(f"Results saved to: {output_folder}")
```
### Complete Image Analysis Example
```python
import os
from pathlib import Path
from deeplabcut.pose_estimation_pytorch.apis import superanimal_analyze_images
# ==================== User Configuration Area ====================
# Image paths (Required) - can be single or multiple
image_paths = [
"/path/to/image1.jpg",
"/path/to/image2.png",
]
# Output directory (Optional, set to None to use the directory of the images)
output_folder = None # Example: "/path/to/output"
# SuperAnimal model selection
superanimal_name = "superanimal_topviewmouse"
# Model architecture selection
model_name = "hrnet_w32"
# Detector selection
detector_name = "fasterrcnn_resnet50_fpn_v2"
# Number of animals (default is 1, only increase if specifically requested by the user)
max_individuals = 1
# ==================== Run Analysis ====================
# Verify all images exist
for img_path in image_paths:
if not os.path.exists(img_path):
raise FileNotFoundError(f"Image file does not exist: {img_path}")
# Determine output directory
if output_folder is None:
# Use the directory of the first image
output_folder = str(Path(image_paths[0]).parent)
# Ensure output directory exists
os.makedirs(output_folder, exist_ok=True)
print(f"Starting analysis of {len(image_paths)} images")
print(f"Output directory: {output_folder}")
# Run analysis
superanimal_analyze_images(
images=image_paths,
superanimal_name=superanimal_name,
model_name=model_name,
detector_name=detector_name,
max_individuals=max_individuals,
dest_folder=output_folder,
)
print("Analysis complete!")
print(f"Results saved to: {output_folder}")
```
## Notes
1. **First Run**: When using a SuperAnimal model for the first time, pre-trained weights will be automatically downloaded, requiring an internet connection.
2. **GPU Acceleration**: If you have a CUDA-enabled GPU, DeepLabCut will automatically use GPU acceleration for analysis.
3. **Memory Usage**: Analyzing long videos or high-resolution images may require significant memory; it is recommended to process in batches.
4. **Result Interpretation**:
- The `likelihood` value indicates detection confidence (0-1), closer to 1 is more reliable
- It is recommended to filter out keypoints with likelihood < 0.5
5. **Multi-Animal Scenarios**: If there are multiple animals in the video, please adjust the `max_individuals` parameter.
6. **Model Selection Recommendations**:
- Top-view mouse experiments → `superanimal_topviewmouse`
- Other quadruped animals → `superanimal_quadruped`
- Prioritize accuracy → `hrnet_w32`
- Prioritize speed → `resnet_50`Discussion (0)
to join the discussion.
No comments yet. Be the first to discuss this paper.