Emotion Recognition and Detection Methods: A Comprehensive Survey

Deepak Gupta

1. Computer Science and Engineering Department, Guru Gobind Singh Indraprastha University, New Delhi, India
Email: [email protected]; [email protected]; [email protected]
*Corresponding Author: Deepak Gupta, Email: [email protected]

Human emotion recognition through artificial intelligence is one of the most popular research fields among researchers nowadays. The fields of Human Computer Interaction (HCI) and Affective Computing are being extensively used to sense human emotions. Humans generally use a lot of indirect and non-verbal means to convey their emotions. The presented exposition aims to provide an overall overview with the analysis of all the noteworthy emotion detection methods at a single location. To the best of our knowledge, this is the first attempt to outline all the emotion recognition models developed in the last decade. The paper is comprehended by expending more than hundred papers; a detailed analysis of the methodologies along with the datasets is carried out in the paper. The study revealed that emotion detection is predominantly carried out through four major methods, namely, facial expression recognition, physiological signals recognition, speech signals variation and text semantics on standard databases such as JAFFE, CK+, Berlin Emotional Database, SAVEE, etc. as well as self-generated databases. Generally seven basic emotions are recognized through these methods. Further, we have compared different methods employed for emotion detection in humans. The best results were obtained by using Stationary Wavelet Transform for Facial Emotion Recognition , Particle Swarm Optimization assisted Biogeography based optimization algorithms for emotion recognition through speech, Statistical features coupled with different methods for physiological signals, Rough set theory coupled with SVM for text semantics with respective accuracies of 98.83%,99.47%, 87.15%,87.02% . Overall, the method of Particle Swarm Optimization assisted Biogeography based optimization algorithms with an accuracy of 99.47% on BES dataset gave the best results.

Emotion Recognition, Emotion Detection, Facial expressions, Speech Signals, Physiological signals (Electroencephalogram signals (EEG), Electrocardiogram signals (ECG)), Text semantics.

Anvita Saxena, Ashish Khanna, Deepak Gupta (2020). Emotion Recognition and Detection Methods: A Comprehensive Survey. Journal of Artificial Intelligence and Systems, 2, 53–79. https://doi.org/10.33969/AIS.2020.21005.

[1] McCarthy, John. What is artificial intelligence? URL: http://wwwformal.  stanford.edu!jmciwhatisai.html, 1998.
[2]   L. Steels,The artificial life roots of artificial intelligence , Artificial Life. 1(1994). 75-110.
[3] Michael Brady, Artificial intelligence and robotics , Artificial Intelligence. 26(1985). 79-121.https://.org/10.1016/0004-3702(85)90013-X.
[4] E. Cambria, Affective Computing and Sentiment Analysis, IEEE Intelligent Systems.31(2016)102-107.
[5] Tao J., Tan T., Affective Computing: A Review. In: Tao J., Tan T., Picard R.W. (eds) Affective Computing and Intelligent Interaction. ACII 2005, Lecture Notes in Computer Science,3784.Springer, Berlin, Heidelberg.
[6] Yashaswi Alva M, Nachamai M and J. Paulose, A comprehensive survey on features and methods for speech emotion detection, 2015 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT), Coimbatore.(2015)1-6. 10.1109/ICECCT.2015.722604.
[7] Vandervoort, D. J.,The importance of emotional intelligence in higher education. Current Psychology: Development, Learning, Personality, Social, 25(1) (2006)4–7.
[8] Bagozzi, R. P., Gopinath, M., Nyer, P.  U. The Role of Emotions in Marketing. Journal of the Academy of Marketing Science. 27(2) (1999)184–206. 10.1177/0092070399272005
[9] Scotty Craig, Arthur Graesser, Jeremiah Sullins Barry Gholson , Affect and learning: An exploratory look into the role of affect in learning with AutoTutor , Journal of Educational Media.29:3(2004) . 241-250. 10.1080/1358165042000283101
[10] Nussbaum M., Emotions as Judgments of Value and Importance. In R. C. Solomon (Ed.), Series in affective science. Thinking about feeling: Contemporary philosophers on emotions. (2004). 183-199. New York, NY, US: Oxford University Press.
[11] M. S. Bartlett, G. Littlewort, I. Fasel and J. R. Movellan, Real Time Face Detection and Facial Expres- sion Recognition: Development and A lications to Human Computer Interaction., 2003 Conference on Computer Vision and Pattern Recognition Workshop, Madison, Wisconsin, USA( 2003). 53-53. 10.1007/BF02884429.
[12] Moataz El Ayadi, Mohamed S. Kamel, Fakhri Karray, Survey On Speech Emotion Recognition: Features, Classification Schemes, And Databases , Pattern Recognition, 44(2011). 572-587. 10.1016/J.Patcog.2010.09.020.
[13] Nicu Sebe, Ira Cohen, Theo Gevers, and Thomas S. Huang Multimodal approaches for emotion recog- nition: a survey , Internet Imaging VI.( 2005) .https:// .org/10.1117/12.600746.
[14] Anagnostopoulos, CN., Iliou, T. Giannoukos, I., Artifical Intelligence Review . ACM. 2()155-177. 10.1007/s10462-012-9368-5.
[15] Z. Zeng, M. Pantic, G. I. Roisman and T. S. Huang, A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions, in IEEE Transactions on Pattern Analysis and Machine Intelligence. 31(2009). 39-58 . 10.1109/TPAMI.2008.52
[16] H. Gunes, B.  Schuller,  M.  Pantic  and  R.  Cowie,  Emotion  representation,  analysis  and  synthe-  sis in continuous space: A survey, Face and Gesture 2011, Santa Barbara, CA(2011). 827-834. 10.1109/FG.2011.5771357
[17] B. Fasel, Juergen Luettin, Automatic facial expression analysis: a survey , Pattern Recognition,36(2003). 259-275.10.1016/S003.
[18] S. Mitra and T. Acharya, Gesture Recognition: A Survey, in IEEE Transactions on Systems, Man, and Cybernetics, Part C (A lications and Reviews). 37(2007). 311-324. 10.1109/TSMCC.2007.893280.
[19] Chan HL, Kuo PC, Cheng CY, Chen YS. Challenges and Future Perspectives on Electroencephalogram- Based Biometrics in Person Recognition Front Neuroinform .(2018).12:66. 10.3389/fninf.2018.00066.
[20] M. Gargesha, P. Kuchi, Facial expression recognition using a neural network, Artificial Neural Compu- tation Systems, 31 (2002). 709-724.
[21] S. C. Tai and K. C. Chung, Automatic facial expression recognition system using Neural Networks, TENCON 2007 - 2007 IEEE Region 10 Conference, Taipei.( 2007). 1-4. 10.1109/TENCON.2007.4429124.
[22] F. Chen, Z. Wang, Z. Xu, J. Xiao and G. Wang, Facial Expression Recognition Using Wavelet Trans- form and Neural Network Ensemble, 2008 Second International Symposium on Intelligent Information Technology A lication, Shanghai. (2008) 871-875.10.1109/IITA.2008.
[23] Neha Jain, Shishir Kumar, Amit Kumar, Pourya Shamsolmoali, Masoumeh Zareapoor, Hybrid  deep neural networks for face emotion recognition, Pattern Recognition Letters. 115(2018). 101-106. https://.org/10.1016/j.patrec.2018.04.010.
[24] L. Ma and K. Khorasani, Facial expression recognition using constructive feedforward neural networks, in IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(2004).1588-1595.10.1109/TSMCB.2004.825930.
[25] P. Liu, S. Han, Z. Meng and Y. Tong, Facial Expression Recognition via a Boosted Deep Belief Net- work, 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus. 2014.1805-1812 . 10.1109/CVPR.2014.233.
[26] I. Mpiperis, S. Malassiotis, V. Petridis and M. G. Strintzis, 3D facial expression recognition using swarm intelligence, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, Las Vegas, NV. (2008). 2133-2136 . 10.1109/ICASSP.2008.4518064.
[27] C. Mayer, M. Wimmer, M. Eggers and B. Radig, Facial Expression Recognition with 3D Deformable Models, 2009 Second International Conferences on Advances in Computer-Human Interactions, Cancun. (2009). 26-31 . 10.1109/ACHI.2009.33.
[28] Y. V. Venkatesh, A. K. Kassim and O. V. R. Murthy, Resampling A roach to Facial Expression Recogni- tion Using 3D Meshes, 2010 20th International Conference on Pattern Recognition, Istanbul.( 2010)3772- 3775. 10.1109/ICPR.2010.91.
[29] Y. Tie and L. Guan, A Deformable 3-D Facial Expression Model for Dynamic Human Emotional State Recognition, in IEEE Transactions on Circuits and Systems for Video Technology, 23(2013). 142-157. 10.1109/TCSVT.2012.2203210
[30] H. Chen, C. Huang and C. Fu, Hybrid-Boost Learning for Multi-Pose Face Detection and Facial Expres- sion Recognition, 2007 IEEE International Conference on Multimedia and Expo, Beijing.( 2007). 671-674.10.1109/ICME.2007.4284739
[31] Yubo Wang, Haizhou Ai, Bo Wu and Chang Huang, Real time facial expression recognition with Ad- aBoost, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., Cambridge. 3(2004). 926-929. 10.1109/ICPR.2004.1334680.
[32] Y. Cheon and D. Kim, A Natural Facial Expression  Recognition  Using  Differential-AAM  and  k-NNS, 2008 Tenth IEEE International Symposium on Multimedia, Berkeley, CA. (2008).220- 227.10.1109/ISM.2008.121
[33] L. Zhang and D. Tjondronegoro, Facial Expression Recognition Using Facial Movement Features, in IEEE Transactions on Affective Computing. 2(2011) .219-229. 10.1109/T-AFFC.2011.13
[34] Kazmi, S.B., Qurat-ul-Ain Arfan Jaffar, M. Soft Comput .(2012) 16: 369. https:// .org/10.1007/s00500- 011-0721-4
[35] Michel, Phili El Kaliouby, Rana. Real time facial expression recognition in video using su ort vector machines. ICMI ’03 Proceedings of the 5th international conference on Multimodal interfaces . Vancouver, British Columbia, Canada. (2003). 258-264 .10.1145/958432.958479.
[36] Zhang, Xiao Mahoor, Mohammad Mavadati, Seyedmohammad. Facial expression recognition using lp-norm MKL multiclass-SVM. Machine Vision and A lications. 26 (2015). 467-483. 10.1007/s00138-015- 0677-y.
[37] D. Datcu and L. J. M. Rothkrantz, Automatic recognition of facial expressions using Bayesian belief networks, 2004 IEEE International Conference on Systems, Man and Cybernetics . The Hague 3( 2004). 2209-2214. : 10.1109/ICSMC.2004.1400656
[38] Q.  Zhen,  D.  Huang,  Y.  Wang  and  L.  Chen,  Muscular  Movement  Model-Based  Automatic  3D/4D Facial Expression Recognition, in IEEE Transactions on Multimedia, 18(2016) .1438-1450. 10.1109/TMM.2016.2557063
[39] Qayyum, Huma Majid, Muhammad  Anwar,  Syed  Khan,  Bilal,  Facial  Expression  Recognition  Using Stationary Wavelet Transform Features . Mathematical Problems in Engineering. (2017).1-9. : 10.1155/2017/9854050.
[40] Pu,  Xiaorong  Fan,  Ke   Chen,  Xiong   Ji,  Luping   Zhou,  Zhihu.  Facial  expression  recognition   from image sequences using twofold random forest classifier. Neurocomputing. 168(2015).1173-1180.10.1016/j.neucom.2015.05.005.
[41] V. Gomathi, K. Ramar, A. S. Jeevakumar, Human Facial Expression Recognition using MANFIS Model , International Journal of Computer Science and Engineering, 3(2009) .93-97.
[42] Zhan Yong-zhao, Ye Jing-fu, Niu De-jiao and Cao Peng, Facial expression recognition based on Gabor wavelet transformation and elastic templates matching, Third International Conference on Image and Graphics (ICIG’04), Hong Kong, China.(2004) .254-257. 10.1109/ICIG.2004.63
[43] Yurtkan, Kamil Demirel, Hasan, Feature selection for improved 3D facial expression recognition . Pattern Recognition Letters.38. (2013).26-33. 10.1016/j.patrec.2013.10.026.
[44] Xiaoyi Feng, Facial expression recognition based on local binary patterns and coarse-to-fine classification, The Fourth International Conference onComputer and Information Technology, 2004. CIT ’04., Wuhan, China(2004) . 178-183. 10.1109/CIT.2004.1357193.
[45] D. Huang, M. Ardabilian, Y. Wang and L. Chen, 3-D Face Recognition Using eLBP-Based Facial Description and Local Feature Hybrid Matching, in IEEE Transactions on Information Forensics and Security . 7(2012). 1551-1565. 10.1109/TIFS.2012.2206807
[46] Yu, Kaimin Wang, Zhiyong Hagenbuchner,  Markus  Feng,  David  Dagan  Feng,  Spectral  embed-  ding based facial expression recognition with multiple features, Neurocomputing. 129(2014). 136–145. 10.1016/j.neucom.2013.09.046.
[47] A. Ramirez Rivera, J. A. Rojas Castillo and O. Chae, Recognition of face expressions using Local Principal Texture Pattern, 2012 19th IEEE International Conference on Image Processing, Orlando, FL. (2012). 2609-2612. 10.1109/ICIP.2012.6467433.
[48] K. Mistry, L. Zhang, S. C. Neoh, C. P. Lim and B. Fielding, A Micro-GA Embedded PSO Feature Selec- tion A roach to Intelligent Facial Emotion Recognition, in IEEE Transactions on Cybernetics .47(2017). 1496-1509.
[49] Q. Mao, M. Dong, Z. Huang and Y. Zhan, Learning Salient Features  for Speech Emotion Recogni-  tion Using Con utional Neural Networks, in IEEE Transactions on Multimedia. 16(2014). 2203-2213. 10.1109/TMM.2014.2360798
[50] Huang, Zhengwei  Dong,  Ming  Mao,  Qirong  Zhan,  Yongzhao,  Speech Emotion Recognition Us-  ing CNN. MM 2014 - Proceedings of the 2014 ACM Conference on Multimedia . (2014). 801-804 . 10.1145/2647868.2654984.
[51] E. H. Kim, K. H. Hyun, S. H. Kim and Y. K. Kwak, Speech Emotion Recognition Using Eigen-FFT       in Clean and Noisy Environments, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication, Jeju. (2007) . 689-694. 10.1109/ROMAN.2007.4415174.
[52] Y. Attabi and P. Dumouchel, Anchor Models for Emotion Recognition from Speech, in IEEE Transactions on Affective Computing. 4(2013). 280-290 . 10.1109/T-AFFC.2013.17.
[53] J. Liu, C. Chen, J. Bu, M. You and J. Tao, Speech Emotion Recognition using an Enhanced Co-Training Algorithm, 2007 IEEE International Conference on Multimedia and Expo, Beijing, (2007) .999-1002 . 10.1109/ICME.2007.4284821.
[54] Yashaswi Alva M, Nachamai M and J. Paulose, A comprehensive survey on features and methods for speech emotion detection, 2015 IEEE International Conference on Electrical, Computer and Communi- cation Technologies (ICECCT), Coimbatore.(2015).1-6. 10.1109/ICECCT.2015.7226047.
[55] Yogesh C.K., M. Hariharan, Ruzelita Ngadiran, Abdul Hamid Adom, Sazali Yaacob, Chawki Berkai, Kemal Polat, A new hybrid PSO assisted biogeography-based optimization for emotion and stress recog- nition from speech signal, Expert Systems with A lications, 69(2017). 149-158.
[56] Paithane, A. N.. Human Emotion Recognition using Electrocardiogram Signals. , International Confer- ence on Pervasive Computing (ICPC). (2014). 10.1109/PERVASIVE.2015.7087042.
[57] M, Muruga an Khairunizam, Wan Yaacob, Sazali Selvaraj, Jerritta, Electrocardiogram-based emotion recognition system using empirical mode decomposition and discrete Fourier transform. Expert Systems. 31 (2013). 10.1111/exsy.12014.
[58] By Mingmin Zhao, Fadel Adib, Dina Katabi , Communications of the ACM,61 No. 9(2018) . 91-100.  :10.1145/3236621.
[59] By Mingmin Zhao, Fadel Adib, Dina Katabi , Communications of the ACM,61 No. 9(2018) . 91-100.  :10.1145/3236621  59.  Ferdinando,  Hany    Se  ¨anen,  Tapio    Alasaarela,  Esko,  2017,  Enhanc- ing Emotion Recognition from ECG Signals using Supervised Dimensionality Reduction. 112-118 . 10.5220/0006147801120118.
[60]   Ferdinando,  Hany    Se  ¨anen,  Tapio    Alasaarela,  Esko,  2017,  Enhanc- ing Emotion Recognition from ECG Signals using Supervised Dimensionality Reduction. 112-118 . 10.5220/0006147801120118.
[61] Duan RN., Wang XW., Lu BL, EEG-Based Emotion Recognition in Listening Music by Using Su ort Vector Machine and Linear Dynamic System. In: Huang T., Zeng Z., Li C., Leung C.S. (eds) Neural Information Processing,Lecture Notes in Computer Science. (2012). 7666. Springer, Berlin, Heidelberg.
[62] Y. Liu, C. Wu, Y. Kao and Y. Chen, Single-trial EEG-based emotion recognition using kernel Eigen-emotion pattern and adaptive su ort vector machine, 2013 35th Annual International Confer- ence of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka. (2013). 4306-4309. 10.1109/EMBC.2013.6610498.
[63] Suwicha Jirayucharoensak, Setha Pan-Ngum, and Pasin Israsena 2014. EEG-Based Emotion Recognition Using Deep Learning Network with Principal Component Based Covariate Shift Adaptation, The Scientific World Journal,627892. https:// .org/10.1155/2014/627892.
[64] Mehmood, Raja Majid Lee, Hyo Jong., EEG based Emotion Recognition from Human Brain using Hjorth Parameters and SVM. International Journal of Bio-Science and Bio-Technology, 7(2015). 23-32. 10.14257/ijbsbt.2015.7.3.03.
[65] Tauseef Sohaib, Ahmad  Qureshi, Shahnawaz  Hagelb¨ack, Johan  Hilborn, Olle  Jerˇci´c, Petar. Evaluating Classifiers for Emotion Recognition Using EEG . Found. Augment. Cognit. Lecture Notes Comput. Sci.. 8027(2013). 492-501. 10.1007/978-3-642-39454-6 53.
[66] Gao, Yongbin et al. Deep learninig of EEG signals for emotion recognition. 2015 IEEE International Conference on Multimedia Expo Workshops (ICMEW) . (2015).1-5.
[67] J. J. Bird, L. J. Manso, E. P. Ribiero,  A. Ekart,  and D. R. Faria,  A study on mental state classifi-  cation using eeg-based brain-machine interface, in 9th International Conference on Intelligent Systems, IEEE(2018).
[68] Y.-S. Seol, D.-J. Kim, and H.-W. Kim. — Emotion recognition from text using knowledge- based ANN, in proceedings of IC-ISCC. (2008) 1569-1572.
[69] Z. Teng, F. Ren and S. Kuroiwa, Emotion Recognition from Text based on the Rough Set Theory and the Su ort Vector Machines, 2007 International Conference on Natural Language Processing and Knowledge Engineering, Beijing. (2007). 36-41. 10.1109/NLPKE.2007.4368008.
[70] Wu, Chung-Hsien et al. Emotion recognition from text using semantic labels and separable mixture models. ACM Trans. Asian Lang. Inf. Process. 5 (2006).165-183. 10.1145/1165255.1165259.
[71] N. Majumder, S. Poria, A. Gelbukh and E. Cambria, Deep Learning-Based Document Modeling for Personality Detection from Text, in IEEE Intelligent Systems, 32(2017) .74-79. 10.1109/MIS.2017.23
[72] M. Pantic and I. Patras, Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences, in IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 36(2006).433-449 . 10.1109/TSMCB.2005.859075.
[73] I. Mpiperis, S. Malassiotis, V. Petridis and M. G. Strintzis, 3D facial expression recognition using swarm intelligence, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, Las Vegas, NV(2008). .2133-2136. 10.1109/ICASSP.2008.4518064.
[74] O. Rudovic, M. Pantic and I. Patras, Coupled Gaussian processes for pose-invariant facial expression recognition, in IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(2013) .1357-1369. 10.1109/TPAMI.2012.233.
[75] W. Zheng, Multi-View Facial Expression Recognition Based on Group Sparse Reduced-Rank Regression, in IEEE Transactions on Affective Computing, 5(2014) . 71-85. 10.1109/TAFFC.2014.2304712.
[76] W. Liu, S. Li and Y. Wang, Automatic Facial Expression Recognition Based on Local Binary Patterns of Local Areas, 2009 WASE International Conference on Information Engineering, Taiyuan, Chanx. (2009). 197-200. 10.1109/ICIE.2009.36.
[77] Li, Huibin Ding, Huaxiong Huang, di Wang, Yunhong Zhao, xi Morvan, J. - M.  Chen, Liming, 2015, An Efficient Multimodal 2D + 3D Feature-based A roach to Automatic Facial Expression Recognition. Computer Vision and Image Understanding. 140. 10.1016/j.cviu.2015.07.005.
[78] Wang, Xun et al. A New Facial Expression Recognition Method Based on Geometric Alignment and LBP Features. 2014 IEEE 17th International Conference on Computational Science and Engineering. (2014). 1734-1737.
[79] R. Srivastava and S. Roy, 3D facial expression recognition using residues, TENCON 2009,2009 IEEE Region 10 Conference, Singapore. (2009).1-5. 10.1109/TENCON.2009.5395856.
[80] Antoine Bechara, The role of emotion in decision-making: Evidence from neurological patients with orbitofrontal damage ,Brain and Cognition, 55(2004). 30-40. https:// .org/10.1016/j.bandc.2003.04.001.
[81]  Lakshmanaprabu SK, Shankar K, Deepak Gupta, Ashish Khanna, Joel J. P. C. Rodrigues, Plácido R. Pinheiro, Victor Hugo C. de Albuquerque, “Ranking Analysis for Online Customer Reviews of Products Using Opinion Mining with Clustering”, Complexity, June 2018, Article ID 3569351.  https://doi.org/10.1155/2018/356935.
[82] Aditya Khamparia, Deepak Gupta, Nguyen Gia Nhu, Ashish Khanna, Babita Shukla, Prayag Tiwari, “Sound Classification Using Convolutional Neural Network and Tensor Deep Stacking Network”, IEEE Access, doi: 10.1109/ACCESS.2018.2888882.
[83] Anvita Saxena, Kaustubh Tripathi, Ashish Khanna, Deepak Gupta, Shirish Sundaram, “Emotion Detection through EEG Signals using FFT and Machine learning techniques”. International Conference on Innovative Computing and Communications (ICICC2019). Advances in Intelligent Systems and Computing, Springer. [Accepted]
[84] Deepak Gupta, Nimish Verma, Mayank Sehgal, Nitesh, “Feature selection using multiobjective grey wolf optimization algorithm”, 2019, Innovative Computing and Communication. Vol.1 (July 2019) 15–18.
[85] Haag A., Goronzy S., Schaich P., Williams J. (2004) Emotion Recognition Using Bio-sensors: First Steps towards  an  Automatic  System.  In:  Andr´e  E.,  Dybkjær  L.,  Minker  W.,  Heisterkamp  P.  (eds)  Affective Dialogue Systems. Lecture Notes in Computer Science. ADS 2004. 3068. Springer, Berlin, Heidelberg, : https:// .org/10.1007/978-3-540-24842-2 4.
[86] K. Takahashi, Remarks on emotion recognition from bio-potential signals, Proc. 2nd Int. Conf. Auton. Robots Agents, 1315 (2004). 186-191. 10.1.1.125.2544.