FROM BEHAVIOURAL INDICATORS TO TRANSFORMER-BASED MODELLING: LEARNING FISH DYNAMICS FROM 3D SONAR DATA FOR INTELLIGENT FEEDING

Previous work [3] showed that combining 3D sonar with environmental sensing can extract useful behavioural indicators, such as vertical fish distribution and activity patterns linked to feeding. However, these approaches mainly rely on static or aggregated features and do not fully capture how behaviour evolves over time. Feeding behaviour is inherently dynamic, with gradual changes in fish distribution and activity.

In this study, we introduce FEAT (Fish Echo Activity Transformer), a transformer-based framework for modelling fish feeding behaviour directly from sonar data. Each sonar recording is represented as a sequence of depth-resolved acoustic frames, preserving how fish distribution changes over time (Fig. 2). This enables the use of sequence modelling to capture temporal behaviour patterns.

Fig. 1: Overview of the FEAT (Fish Echo Activity Transformer) pipeline, including data preprocessing, self-supervised pre-training via masked frame reconstruction, supervised fine-tuning for feeding classification, and evaluation.

A complete processing pipeline is developed (Fig. 1), including cage masking, transformation of sonar data into compact depth–intensity profiles, and sequence modelling using a transformer encoder. The model is inspired by recent advances in sequence learning and adapted to continuous acoustic data.

The method follows a two-stage training strategy. First, a self-supervised pre-training stage uses masked frame reconstruction, where part of the input is removed and the model learns to reconstruct it from context. This allows the model to learn general patterns of fish activity from large amounts of unlabeled data. Second, the model is fine-tuned using labelled data to classify feeding and non-feeding sessions.

Fig. 2: Temporal structure of the sonar dataset, showing session-based acquisition with variable lengths and sampling frequencies during feeding and non-feeding periods, along with depth-resolved acoustic intensity.

To ensure a fair evaluation, the dataset is split by calendar date, so the model is tested on unseen recording periods. The data consist of sessions with different lengths and sampling frequencies, reflecting different acquisition modes during feeding and non-feeding periods (Fig. 2). This introduces additional complexity, as recording protocol and feeding activity are partly correlated.

Fig. 3: Confusion matrix of FEAT predictions on the held-out test set (n = 261 sessions), showing high recall for feeding detection and moderate precision due to false positives.

Evaluation on a held-out test set (n = 261 sessions) shows that the model captures meaningful behavioural signals. It achieves an AUC-ROC of 0.865 and a recall of 0.872, indicating strong ability to detect feeding events. Precision is lower (0.506), due to false positives (Fig. 3). This suggests that the model prioritizes detecting feeding events, which may be desirable in practice where missed feeding is more critical than false alarms.

Further analysis shows an important limitation: session length and sampling frequency are not independent of feeding activity. This means that part of the model's performance may be influenced by differences in data acquisition rather than behaviour alone. This highlights the need for improved data collection, ideally continuous acoustic recordings, to better isolate behavioural signals.

Despite this limitation, the proposed transformer-based approach is a clear step forward in modelling fish behaviour from acoustic data. By learning directly from temporal sequences, the model captures behavioural dynamics that are not represented in traditional feature-based methods. This opens new possibilities for understanding fish behaviour and detecting subtle changes related to feeding.

The results demonstrate the potential of transformer-based models for aquaculture monitoring and their role in future intelligent feeding systems. By enabling data-driven interpretation of fish behaviour at scale, such methods can support more efficient and sustainable feeding strategies.

Acknowledgements

The work described has received funding from the Research Council of Norway under grant 341001. We are also grateful for the contribution of the industry partners, WAIVE, Aanderaa, and Lingalaks in making this project possible.

References

[1] Håkon Måløy. "EchoBERT: A transformer-based approach for behavior detection in echograms". In: IEEE

Access 8 (2020), pp. 218372–218385.

[2] Johansson, D., Laursen, F., Fernö, A., Fosseidengen, J. E., Klebert, P., Stien, L. H., ... & Oppedal, F. (2014). The interaction between water currents and salmon swimming behaviour in sea cages. PloS one, 9(5), e97635.

[3] King, D. M., Dunn, M., Kerhervé, S., Nissen, O., & Saad, A. (2025). Robust Inference of Fish Behaviour from Underwater Acoustic and Environmental Data. Procedia Computer Science, 270, 5490-5499.

[4] Nissen, O. (2026).FEAT: Fish Echo Activity Transformer for Feeding Detection from Sonar Data.OptiFe3d WP3 Report, SINTEF Ocean.s