Efficient computer vision and machine learning methods for automating large-scale analysis of collective animal behavior

Research into biological systems has, as many other fields of research have, greatly profited from an ever-increasing range of computational tools. This is important, not just because automation can make life easier, but because it enables researchers to increasingly address questions that were previously impossible to answer. Around 50 years ago, in 1972, D. Radakov published his book on “Schooling in the Ecology of Fish”, detailing a vast number of interesting hypotheses and experiments to investigate the underlying factors of schooling — none of which he was able to evaluate quantitatively. Not for the lack of trying, but simply because the required technology did not yet exist. Back then, while already enjoying the benefits of the “motion picture film”, the paths and shapes of individuals had to be traced manually. This limited researchers to work with mere seconds of trajectory data, which is hardly enough to form any scientific conclusions about the rules governing each individual’s movements, or the sub-second structure of alarm responses. High-speed cameras, automated tracking and analysis, as well as a much more interconnected scientific community (especially because Radakov lived in the Soviet Union), are developments of the past few decades that significantly improved the prospects in, but not limited to, animal behavior research. In this thesis, I present multiple methods which are specifically designed to assist research in collective behavior, ecology, biomechanics, and hopefully others, that are powerful, yet pragmatic.

In the first chapter of my thesis, I present a tool, called TRex, that combines implementations of various methods, each addressing different aspects of typical video-based analyses of behavior, and highlight the importance of their synthesis into a single piece of software — as opposed to the prevalent fragmentation of tools in our field of research. The main focus of this chapter is a machine learning based method for the unsupervised visual identification of individuals, which I demonstrate to be superior to previously available implementations. Importantly, this chapter also builds the foundation for the later chapters, where I take these approaches further, addressing increasingly more difficult scenarios. In Chapter 2, where I focus on the exploration and handling of very large datasets using, as an example, field-data of the termite species Constrictotermes cyphergaster (Silvestri 1901). These eusocial insects are known to live in large colonies, making them an ideal benchmark for the application and, consequantially, improvement of my methods. Furthermore, on this basis, using a simplified representation of trajectory data, I present two methods for behavioral anomaly detection that are able to identify both the temporal and spatial coordinates of unusual events. Finally, in Chapter 3, I address the importance of inter-individual morphological differences, from both a methodological and a biological perspective. There, I present improvements for the matching algorithm I developed in Chapter 1, as well as a machine learning algorithm specifically designed to automatically identify different morphological “types” of individuals under minimal supervision. I end this chapter by using many of the previously described tools to further investigate the dataset I used in Chapter 2, primarily focusing on the differences between two different types of termites (castes), “workers” and “soldiers”, in terms of their movement and an approximation of tactile interactions. This is, to our knowledge, the largest quantitative study of our study species (C. cyphergaster).

Together, the methods presented in this thesis form a versatile chain of efficient computer vision and machine learning tools to automate tracking, posture estimation, recording, dataset exploration and to classify morphological types of individuals. Despite the focus on termites in the last two chapters, all of these approaches are designed to work for a wide range of species and in different situations, with many possible applications in a wide range of research fields, including computational biology, neurobiology and ecology.