Behavior forecasting for social interactions: a multimodal skeleton-based approach

Author: German Barquero García

Virtual Room:

Date & time: 22/09/2021 – 11:00 h

Session name: Human modelling

Supervisors: Xavier Baró, Sergio Escalera i Cristina Palmero


Many works focus on predicting the motion or trajectory of individuals engaged in a particular action, which intends to reduce the inherent stochasticity of the future. We open a new horizon by aiming at forecasting human behavior in dyadic interactions. In such scenarios, the ability to anticipate human behavior implies an implicit knowledge of the underlying mechanisms of communication involving cognitive, affective, and behavioral perspectives. This knowledge is key for many applications in robotics, medicine and psychology. In this work, we introduce an extended version of the UDIVA dataset which contains automatically extracted face, body and hands landmark annotations for 145 dyadic sessions among 134 participants. We use it to deeply analyze the current limitations of interaction forecasting, most of them derived from the multimodal nature of the future and the huge dimensionality attached to human behavior. In parallel, we propose a multimodal recurrent model based on the popular seq2seq model, which serves as a baseline for future research on this topic. Finally, we present an ablation study to discuss the effects of leveraging multimodal data such as audio and participants metadata.


– President: David Masip(UOC)
– Secretary: Javier Ruiz Hidalgo(UPC)
– Vocal: Juan Felipe Montesinos Garcia(UPF)