Doctoral Thesis Defense Session No. 51 of PPGEE - MULTI-MODAL AND MULTI-CLASS EMOTION RECOGNITION SYSTEM FOR HUMAN-ROBOT INTERACTION

STUDENT: LARA TOLEDO CORDEIRO OTTONI

 

DATE: 10/04/2024

 

TIME: 09:00

 

LOCATION: https://us02web.zoom.us/j/89785133180?pwd=pGJBLQh9csArDCXNA1v5NLF3DV6ZBh.1

 

TITLE: MULTI-MODAL AND MULTI-CLASS EMOTION RECOGNITION SYSTEM FOR HUMAN-ROBOT INTERACTION

 

KEYWORDS: Emotion Recognition; Multimodal System; Human-Robot Interaction; Machine Learning; Fuzzy System;

 

ABSTRACT:

The challenge of Human-Robot Interaction (HRI) is to build intelligent systems that can adapt to changes in users and the environment in order to improve real-time interaction. Thus, a growing approach is the use of emotions in HRI. In this sense, there are multimodal emotion recognition systems, which classify emotions in various modalities (facial expression, gestures, speech, and others). However, although there are studies that deal with multimodal emotion recognition, they still have limitations in the methodology of emotion classification, in addition to considering emotions as binary and ignoring the various emotions that may be present in the user. Thus, the objective of this work is to propose a multimodal and multiclass emotion recognition system for human-robot interaction. The use of facial expression and speech modalities is proposed, as well as the fusion of emotions. The Speech Emotion Recognition Module (SEM) is responsible for inferring the emotion in the user's speech, in which a deep learning model is used to classify the emotion. The Facial Expression Emotion Recognition Module (MREEF) is also proposed, which classifies the emotion by the user's face using a convolutional neural network (CNN). Finally, the fusion of recognized emotions using a fuzzy system is proposed. When the proposed system uses the MELD database, only the use of MREF achieved an accuracy of 73%, MREEF 78.06% and the fusion of the modules achieved an accuracy of 78.94%.

 

MEMBERS OF THE BOARD:

JES DE JESUS ​​FIAIS CERQUEIRA (UFBA) - Advisor

ANTONIO CARLOS LOPES FERNANDES JUNIOR (UFBA)

MARIANA SCHIAVO NETTO (Université Gustave Eiffe - France)

ADRIÃO DUARTE DÓRIA NETO (UFRN)

MARCOS YUZURU DE OLIVEIRA CAMADA - (IFBaiano)

 

On 10/01/2024

Em 01/10/2024