Characterization and Detection of Behavioral Patterns in Videos

We applied Deep learning Convolutional Neural Network for detecting visual featureslike happy, sad, neutral and expressive by training 1,00,000 video frames of TED talk and various other talks. For audio feature extraction we have extracted the main audio features like STE (Short Term Energy, ZCR (Zero Crossing Rate), Magnitude and Standard Deviation of the audio file. Analyzing textual content in videos to characterize the behavior of speaker is very crucial point in our thesis as only audio and visual features are not sufficient to robustly detect and characterize the behavior of speaker. The IBM Watson Tone Analyzer service uses language analysis to identify emotional and language tones in written documents. This service detects tone at both document level and text (sentence) level. This service helps us to understand perceiving of our written communications and also to improve tone of the communications. We are also using Long Short Term Memory (LSTM) systems to categorize a text blob's sentiment into positive, neutral and negative. Finally we combine audio, visual and text features to develop a recommendation system of 27 categories, to analyze the behavior and ability of the speaker and recommend the future speakers to know the main factors which affect their public speaking skills and how to work for betterment of their speech