Introduction
Monitoring fish activity in aquaculture and marine conservation is essential for evaluating animal welfare and optimizing production. Accurate tracking of fish posture provides quantitative indicators of stress and environmental adaptation. However, deploying reliable computer vision systems in underwater environments remains challenging due to the scarcity of annotated data. Manual annotation of underwater footage is labor-intensive and prone to human bias, making large-scale dataset construction impractical.
While deep learning has advanced in terrestrial pose estimation, underwater settings introduce domain-specific obstacles, such as optical distortion, backscatter, and complex 3D movement. A critical technical challenge is the reliable determination of body orientation; visual similarities between head and tail regions, compounded by poor visibility and partial occlusions, often lead to incorrect anterior-posterior assignment.
This work proposes a multi-stage, annotation-efficient pipeline that bridges the gap between unsupervised geometric approximations and supervised deep learning. Building upon the annotation-free framework by Cebri��n et al. (2025), we introduce a transfer learning strategy using curated pseudo-labels to fine-tune a ConvNeXt-based (Liu et al., 2022) pose estimator. Our primary contribution is the systematic resolution of orientation ambiguity through a "human-in-the-loop" refinement process, enabling robust behavioral monitoring with minimal manual labeling effort.
Methodology
The methodology follows a multi-stage pipeline, transitioning from an annotation-free approach to a minimally supervised transfer learning strategy:
Stage 1: Pseudo-label Generation. An initial annotation-free pipeline (Cebri��n et al., 2025) employs GroundingDINO and the Segment Anything Model (SAM) for zero-shot detection and segmentation. Keypoints are localized using classical image processing (contour analysis and morphological operations). While efficient, this stage is prone to systematic head/tail flips because classical heuristics struggle to distinguish the snout from the caudal region under variable underwater lighting.
Stage 2: Curation and Transfer Learning. To rectify these limitations, a refined subset of 100 images from a dataset of 4280 pseudo-labels is selected through a manual quality filtering. During curation, frames with orientation inversions (identifiable by anatomical inconsistency) are either corrected or removed depending on the reliability of their pseudo-labels: frames with high-quality pseudo-labels are manually adjusted to fix errors, whereas those with poor-quality are discarded to reduce manual annotation effort. This refined subset is used to train a deep learning model within the SLEAP framework (Pereira et al., 2022). We adopt a top-down architecture for pose estimation composed of two ConvNeXt-based models: the first stage (top) localizes instance centroids, and the second stage (down) predicts keypoints. Both models are initialized with weights pre-trained on the ImageNet dataset (Deng et al., 2009) and fine-tuned on the curated aquatic pseudo-labels.
To evaluate this pipeline, a progressive training strategy was designed to analyze performance as a function of the amount of curated data. From the refined subset of 100 images, 10 were held out as an independent test set, while the remaining 90 images were used exclusively for training. These 90 images were incrementally partitioned into subsets of increasing size (10, 20, 30, ���, up to 90 images), and a separate model was trained for each subset under identical conditions. This iterative scheme enables a systematic assessment of how model performance scales with the availability of annotated data, providing insights into data efficiency and the marginal benefit of additional training samples within the proposed minimally supervised pipeline.
Results and discussion
The pipeline was evaluated on underwater sequences of luderick (Girella tricuspidata) and Australian bream (Acanthopagrus australis) (Ditria et al., 2021), using Mean Average Precision (mAP) at an Object Keypoint Similarity (OKS) threshold of 0.5 (mOKS), F1-score for visibility (annotation-free approach is excluded from visibility metrics since it forces all keypoint detections regardless of occlusion), and the average of Percentage of Correct Keypoints (PCK) only focusing on head and tail targets (mPCK_TH) to assess pose and orientation performance. Table 1 reports strong performance with small training sets and clear improvements over the annotation-free baseline. The smallest training set (10 images) does not exceed the annotation-free baseline, whereas performance improves and stabilizes from ~30 images onward, indicating an efficient trade-off between annotation effort and accuracy. The proposed method corrects systematic orientation errors in annotation-free approaches by learning discriminative visual features that capture head–tail asymmetries, improving the reliability of behavioral metrics.
Table 1. Quantitative analysis of F1-Score, mPCK_TH, and mOKS relative to image count.
N�� images
Annotation-free
10
20
30
40
50
60
70
80
90
F1-Score
N/A
0.54
0.94
0.96
0.96
0.96
0.97
0.96
0.96
0.97 mPCK_TH
0.20
0.00
0.37
0.47
0.43
0.47
0.48
0.50
0.49
0.49 mOKS
0.21
0.09
0.33
0.41
0.42
0.41
0.44
0.47
0.44
0.45
Conclusions
This work presents a multi-stage pipeline for fish pose estimation bridging annotation-free and supervised learning, enabling orientation-consistent keypoint detection with minimal annotation effort. The method improves accuracy over annotation-free heuristics by correcting systematic orientation errors through discriminative visual features capturing head–tail asymmetries. Future work will delve to iteratively expand training dataset and self-supervised learning to improve generalization and leverage unannotated data.
Acknowledgements
This research was funded by the Spanish Ministry of Science and Innovation through the OASIS-RETIRO project (PID2023-148285OB-C44).
References
Cebri��n, P. L., Rosa, G, Obelleiro, G., Vi��uelas, Juan A., Chavarr��as, M. & Sanz, C. (2025). Towards annotation-free fish monitoring in aquatic environment based on vision transformers. Aquaculture Europe 2025, Valencia, Spain.
Liu, Z., Mao, H., Wu, C. Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022). A ConvNet for the 2020s. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Pereira, T. D., Tabris, N., Matsliah, A., Turner, D. M., Li, J., Ravindranath, S., & Murthy, M. (2022). SLEAP: A deep learning system for multi-animal pose tracking. Nature Methods, 19, 486–495.
Deng, J., Dong, W., Socher, R., Li, Li-Jia., Li, K., Fei-Fei, L. (2009). ImageNet: A Large-Scale Hierarchical Image Database [dataset]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2009.5206848
Ditria, E.M., et. al. (2021). Annotated video footage for automated identification and counting of fish in unconstrained marine environments [dataset]. PANGAEA, https://doi.org/10.1594/PANGAEA.926930