In Tīkapa Moana - Te Moananui-ā-Toi – the Hauraki Gulf, the Marine Mammal Ecology Group (MMEG) at the University of Auckland have been researching multi-species foraging associations, colloquially known as workups or boil-ups. These workups are a familiar sight for those who go out on a boat or live near the coast, with animals from different species, including seabirds and cetaceans, coming together to feed on ephemeral and patchily distributed prey. As highlighted in MMEG's recent NZ Geographic feature, the workups are short-lived, dynamic and temporary communities that are changing with the health of the Gulf. Changing prey abundance and distribution is affecting the health of the Gulf as a whole, with anecdotal evidence suggesting these workups are decreasing in frequency and size.
Our AI detects marine workups.
To better understand these workups, the MMEG team used boat-based observations to determine who was present and what they were doing. However, this is not an easy task! With thousands of individuals coming together, they needed to determine who was there, how many animals there were, and what they were doing. While it's possible to observe the animals from a boat, drones and artificial intelligence have completely changed the game.
By collecting drone footage (under DOC permits), they have gained a new perspective on these dynamic events. These drone videos provide a high-definition bird's eye view of the workups, capturing thousands of images, often containing over a hundred animals in one frame. To analyse the drone footage collected, MMEG's Wednesday Davis and MAUI63's Tane van der Boon trained an artificial intelligence tool to recognise five seabird species, common dolphins, and Bryde's whales. Now, rather than someone having to sit through hours of footage, the AI can do it for them, automatically classifying the animals within images to a species level and, with a less certain accuracy, their behavioural states.
MSFA event captured by drone, with common dolphins, gannets and Bryde's whales foraging together.
Using the same approach to identify boats, and Māui dolphins (See our computer vision post), the species classifying and counting model has high accuracy, meaning that these high-definition videos can be analysed quickly and precisely. The ResNet-50 model created has 93-99% accuracy (mAP), tagging images 93-99% as accurately as a human would. While sea-surface conditions and animal size affect the accuracy of identification, it will be a valuable tool to complement the boat-based observations of workups in the Hauraki Gulf and other marine environments. The partnering of drones and AI will allow researchers, such as team MMEG, to understand dynamic ocean communities better, increasing the efficiency and scale of data collection and providing novel insights into environmental challenges.
The next steps for this project will be to share the model's output, further enhancing the usability of AI, and remove the need to start a model from scratch for projects on trained species. If you would like to read more about Wednesday's research on workups in the Gulf, you can read her thesis here.
MSFA event in the inner Hauraki Gulf, with common dolphins, gannets and fluttering shearwaters foraging together.