Grading of Facial Emotions Using Multimodal Approach

Emotion analysis is an area which is been widely used in forensic crime detection domain,mentoring device for depressed students, psychologically affected patient treatment. Although significant work has been done in this area, the current system helps only in identifying the emotions but not in identifying the level of emotions like whether an individual is truly happy /sad or pretending to be happy/sad. Instead of classifying the image of the face into one of the categories of facial expression, we need to assess the depth of facial expression. In this proposed work using pre-processed video images, algorithms such as LBP and Euclidean distance are fused to extract emotional features and obtain up to 19 features of the human face. We reconstructed the traditional local binary model (LBP) function operator by merging the algorithm and combining the abstract facial expression characteristics learned from deep convolution neural network. These 19 features are used in the Confusion Matrix for grading analysis of emotions such as Grades 1, 2, and 3, with features generated automatically by CNN. The combination of emotion analysis and data analysis provides a suitable emotional identification and verification system for grading the emotion in a given image