Introduction
The problem of segmentation and pose prediction of animals in images is still a significant challenge. Part of the difficulty lies in the complexity, accuracy and volume of training data required. This work presents a novel method for creating synthetic training data. This data is used to train a convolutional neural network that can analyze salmon behavior in consecutive frames. The use of synthetic data solves the problem of data amount and accuracy and allows for model evaluation with a known ground truth.
Synthetic training data
The synthetic training data is primarily created by manipulating a single 3D model of a salmon. This 3D model is copied and given random poses within the view cone of a virtual camera along with some light sources. The image of the salmon instances is rendered and combined with a randomized gradient background. The image is completed by adding some random noise and a soft Gaussian blur. This process is illustrated in Figure 1. An arbitrary number of randomized images can then be created using this process.
The use of a known 3D object also allows for attaching key points to the model mesh. These key points can then be used to train a pose prediction model like YOLO11 pose. The model along with the key points representing a simple fish spine is shown in Figure 2.
Initial AI model results
The synthetic training data has been used to train a YOLO11 instance segmentation model that has never seen a real salmon. The initial results from this training are promising, as can be seen in Figure 3, where the trained model is evaluated on a real video capture of salmon. This initial model can produce serviceable masks for the same individual over several frames. The positive results have prompted further research into pose estimation of salmon using key points. Research on pose estimation is currently in the works and will be further elaborated on.