AI tells hundreds of fish apart

Where’s Wally? was a childhood favourite for many, but could you have found him if the image was moving and if Wally, and the rest of the crowd, were all fish?

Probably not, and that’s because humans are not able to process that amount of information in live time. But now a new artificial intelligence (AI) system, built by researchers at the Collective Behaviour Lab at the Champalimaud Center for the Unknown in Lisbon, can keep track of a hundred different fish simultaneously.

In a tank of 100 zebrafish, this "sofishticated" software knows where any individual fish is at any given time. It is also capable of processing videos, breaking them down into individual frames and cataloguing defining features of each individual animal. In tests, when the software was presented with a new image it had never seen before of any one of the 100 fish in the tank, it was able to identify it with 99.99% accuracy.

This new technology will allow researchers to investigate complex group behaviours of animals more accurately. “We found that there were interesting behaviours we wanted to study,” says first author of the Nature Methods paper Francisco Romero-Ferrero. But the team was limited by their ability to track individuals in a group. Previous software couldn’t handle large numbers of animals and was too slow to process the data. 

To overcome this, the team used a technique in artificial intelligence called "deep learning" and, in particular, a process called a convolutional neural network. The first process detects when animals touch or cross paths. Once isolated images of each fish are secured, the second neural network assigns a number and therefore identity to each fish. The software learns to distinguish the zebrafish by unique features on their body, and these are detected through the camera lens and translated into pixels.

“Artificial intelligence trains machines to behave like humans... and sometimes the machines can even perform tasks that humans cannot do, like in our case,” Romero-Ferrero explains.

By extracting so much accurate data from the videos of swimming fish, the researchers were able to observe how an animal group decides and learns together. Initial findings show that 100 juvenile zebrafish will form mills (a whirlpool shape), where those individuals who preferred the outermost edges of the group, swim much faster. There are many more questions to be answered through these type of observations, for example, do more dominant ‘leaders’ in a group exert more influence in the decision making, or does the group reach a decision by consensus?

The team also used the tracking software to study fruit flies, ants and mice. According to Romero-Ferrero, “the system is very general, the reason is the pre-processing algorithms are species agnostic, meaning that it is irrelevant to the animal your trying to track, and that’s what makes it powerful.”

So could the software also keep tabs on people? It's probably a case of overkill for that because, “humans are easier to identify, because we have clothes that can be different colours or have different patterns,” says Romero-Ferrero. That said, as he points out, “the software is open source and free... so it could be possible that the algorithm that we developed is modified to other applications.”