Beyond Keypoints: Foundation Models and AI Agents for Measuring Behaviour Across Species

Organisers

Jack Bray and Gernot Riedel (Aberdeen University, UK)

Summary

Behavioural quantification has been transformed by markerless pose estimation, but the field is now moving beyond it. A new generation of methods treats behaviour as a problem of representation rather than one of keypoints: foundation models pretrained on large video corpora that transfer to new species with few or no labels; vision-language models that allow an ethogram to be specified in natural language; and LLM-driven agents that select an analysis strategy, write the code and return a scored result. Together these shift the bottleneck away from labelling and bespoke classifiers, and towards questions of validation, generalisation and trust.

This symposium brings together developers and early adopters of these methods across rodents, primates, insects, livestock and wildlife. Speakers will address what transfers across species and what does not, how zero-shot and few-shot performance compares with purpose-built supervised pipelines, and what new validation standards are needed when a model has not been trained on your animals, your arena or your behaviours. The aim is a practical, cross-disciplinary assessment of where these tools are genuinely usable today, framed for an audience spanning neuroscience, ethology, welfare science and computer vision.

Speakers will be listed here once their papers have been acceped.

Beyond keypoints
Image generated by Claude and ChatGPT