One of the philanthropic focus areas of the Product Engineering team at Vulcan is ocean health. On a grand scale, this is best captured by projects like Skylight and the Allen Coral Atlas. Closer to home here in Seattle, killer whales (commonly known as Orcas) are deeply symbolic of wider ocean health problems. The Endangered population of southern resident killer whales is dependent on its primary prey, Chinook salmon, and – as salmon availability varies – so too does killer whale health. Current population estimates put the number of killer whale individuals at only 73 as of August 2019, a 30 year low.
Towards recovery of this Endangered population, researchers at SeaLife Response, Rehabilitation, and Research (SR3) have been collecting imagery of killer whales over the past eight years using unmanned aerial vehicles (UAVs). The imagery is analyzed by scientists to identify individual killer whales and assess their health over time, which can trigger policy changes (like fishery management, for example). Performing this analysis manually, while scientifically valuable and accurate, is also time intensive. It can take up to several months for annual updates, which unfortunately makes it difficult to expeditiously take action for killer whales of declining health.
In order to help close this gap, Vulcan is kicking off a new project over the coming year that will seek to use machine learning (ML) to automate some of the aforementioned labor-intensive work by scientists and hopefully lead to more timely action toward improving killer whale health.
Aerial images of adult female Southern Resident killer whale “J17," displaying very poor body condition on May 6th 2019. Note the white eye patches that trace the outline of her skull due to a reduction in fat around the head. Her condition is contrasted to September 2018, when she was also very lean but had not yet developed such an obvious "peanut head", and to September 2015 when she was clearly pregnant (note width at mid body) and in peak recent condition. Images obtained by Holly Fearnbach (SR3) and John Durban (NOAA Fisheries’ Southwest Fisheries Science Center) using a remotely-piloted drone under NMFS Research Permit #19091.
Machine Learning to the Rescue
The Vulcan Machine Learning Center for Impact (VMLCI) is a team within Product Engineering that seeks to apply the tools of machine learning toward making and leaving the world a better place. At a high level, ML is the study of finding patterns in data. The most common type, supervised learning, uses a large quantity of training examples to train a model. After training, the model can be run to automatically discover patterns similar to those that were provided in the training examples. For example, in the past we’ve used ML to automatically detect and draw bounding boxes around elephants in aerial imagery from Africa. We’ve also used ML to analyze vessel (ships, boats and other watercraft) trajectories to detect illegal shipping activity.
In this project, Vulcan – through automation – seeks to replicate the manual process performed by scientists at SR3. We’ve proposed a pipeline of ML tasks that move us toward that goal.
Killer Whale Instance and Primitive Segmentation
We first produce killer whale body profiles using instance segmentation, a type of machine learning that assigns each image pixel to a class instance (e.g. killer whale 1, killer whale 2) with confidence scores. Mask R-CNN, one of today's leading methods in instance segmentation is currently used, though we have a number of improvements in mind corresponding to the needs of our specific domain. For example, we will use post-processing to refine the masks and remove wake artifacts.
A similar method will be used on segmented killer whales to discover primitives needed for assessing health, such as blowholes, rostrum (snout) and eye patches.
Killer Whale Measurability
To measure killer whale condition in an image, the individual must be visible at or near the water surface, and in a flat, straight orientation. The images used in this study have been annotated by SR3 to identify whether individuals are in a measurable position, and this information will act as a ground truth around which to build computer-vision heuristics. Also, we plan to use histogram information to cluster and segment different pixel regions of the killer whale to determine its position. This step will rule out certain erroneous killer whale detections and provide a confidence level for measurability on those that remain.
Killer Whale Identification and Tracking
Expert annotations have also provided killer whale identifications in multiple images, allowing us to develop an ID system akin to face recognition, where killer whale images are compared to existing individuals via a Siamese network. Additionally, killer whales live in a matrilineal social system, which allows for well-known associations between individuals. This social information will help us determine which individuals are likely to be near one another, aiding in killer whale identification.
Tracking killer whales throughout a UAV flight also provides additional scientific insights, even if measurability is low. In most frames, killer whales are highly visible and identifiable. Frame locations and camera altitude are all known, allowing killer whale tracks to be generated along with speed and territory data. Kalman filtering techniques will be used to fill in the blanks when killer whales are not visible in certain frames due to wake, water splashes or various other occlusions. Durban et al. are working on movement models for inferring behavioral metrics, separate from this collaboration.
Killer Whale Automated Photogrammetric Analysis
Probably the most important health metric that we’ll seek to replicate is the eye patch ratio (EPR) measurement. killer whales in poor nutritional health exhibit a peanut-shaped head, which is directly reflected in the EPR measurement that quantitatively describes fatness behind the head. To produce automated photogrammetric metrics like EPR, we will use classic computer vision to refine and validate the killer whale primitives previously detected, and then use them to calculate the associated health metrics. Known altitudes and camera parameters will allow for photogrammetric metrics to produce precise measurements.
Finally, to make automated marine mammal health metrics available to scientists, we will build a configuration-based open-source end-user tool that produces automatic health-metric calculations on incoming marine mammal data. To produce a configuration specific for a particular marine mammal, requirements will be documented in an effort to make this tool as widely useful as possible. The tool will allow scientists to interact with the data and make adjustments to the automatically produced metrics. Ideally, scientists will be able to import a new set of images into the tool, run the automation, make minor adjustments, and export the results to help support conservation policy actions.
Vulcan and the VMLCI are excited to embark on this new project and contribute toward the conservation of one of the Pacific Northwest’s most iconic species.