Project Preparation

During my third year of university, we explored and researched an area of our choice to produce the first half of an academic paper and poster. I chose "Bias in Multimodal Emotion Recognition", which would seek to explore different methods for debiasing multimodal data within the emotion recognition context. Please see the abstract below:
Context/Background - Multimodal Emotion Recognition is the process of extracting emotion and opinion by leveraging more than one modality such as facial imaging, textual and vocal information. Bias is inherent in this system as it will perform better for some protected group of individuals such as race or gender, which raises ethical concerns where sub-groups of people are disadvantaged. This could also amplify cultural and social bias in society.
Aims - This paper seeks to evaluate the bias in a multimodal emotion recognition system, while experimenting with potential mitigation strategies to alleviate the overall bias.
Method - We utilise an early fusion method for merging textual data (from bi-LSTM latent space), visual data (from 3D-convolutional projected latent space), and finally vocal data (from a projected latent space), to build our multimodal emotion recognition system. From here, we experiment with different combinations of debiasing techniques for each modality and compare our metrics with recent literature.
Theories & Technologies used: LaTeX, Multimodal Emotion Recognition, Debias, Explainable AI, Computer Vision, NLP