Hybrid attention-based deep learning network for emotion recognition by ECG signal

Vaezi, Mahtab; Nasri, Mehdi; Azimifar, Farhad; Mosleh, Mahdi

doi:10.57647/j.mjee.2025.1902.29

Document Type : Reseach Article

Authors

¹ Department of Biomedical Engineering, Isfahan (Khorasgan) Branch, Islamic Azad University, Isfahan, Iran.

² Department of Biomedical Engineering Khomeinishahr Branch, Islamic Azad University, Isfahan, Iran.

³ Efficiency and Smartization of Energy Systems Research Center, Khomeinishahr Branch, Islamic Azad University, Isfahan, Iran.

⁴ Department of Computer Engineering, Isfahan (Khorasgan) Branch, Islamic Azad University, Isfahan, Iran.

10.57647/j.mjee.2025.1902.29

Abstract

Emotions play an important role in our daily activities, decision-making, and artificial intelligence needs to identify emotions to interact constructively with its audience. In this paper, an intelligent method for two-dimensional emotion recognition is proposed. The ECG signal available in the DREAMER database has been used to recognize emotions because of the high correlation of this signal with emotions and easy recording. First step for valence and arousal recognition, the ECG signal is entered into the deep learning network, which is a combination of CNN and LSTM. CNN performs feature extraction and LSTM performs data classification. The attention mechanism aims to optimize the weights and improve the performance of the network, overseeing the proposed deep learning network. Using the proposed method, valence and emanation were identified with 95% and 94% accuracy, respectively. The proposed hybrid network is very suitable for high-dimensional data, and the use of the attention mechanism helps to improve the performance of the network by preventing overfit and getting stuck in local optimal.

Keywords

References

[1] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, "Multimodal machine learning: A survey and taxonomy," IEEE transactions on pattern analysis machine intelligence, vol. 41, no. 2, pp. 423-443, 2018.

https://doi.org/10.1109/TPAMI.2018.2798607

[2] N. Ahmed, Z. Al Aghbari, and S. Girija, "A systematic survey on multimodal emotion recognition using learning algorithms," Intelligent Systems with Applications, vol. 17, p. 200171, 2023.

https://doi.org/10.1016/j.iswa.2022.200171

[3] S.-W. Byun, J.-H. Kim, and S.-P. Lee, "Multi-modal emotion recognition using speech features and text-embedding," Applied Sciences, vol. 11, no. 17, p. 7967, 2021.

https://doi.org/10.3390/app11177967

[4] Z. Lian, Y. Li, J.-H. Tao, J. Huang, and M.-Y. Niu, "Expression analysis based on face regions in real-world conditions," International Journal of Automation Computing, vol. 17, pp. 96-107, 2020.

https://doi.org/10.1007/s11633-019-1176-9

[5] J. A. Russell, "A circumplex model of affect," Journal of personality social psychology, vol. 39, no. 6, p. 1161, 1980.

https://doi.org/10.1037/h0077714

[6] A. Geetha, T. Mala, D. Priyanka, and E. Uma, "Multimodal Emotion Recognition with deep learning: advancements, challenges, and future directions," Information Fusion, vol. 105, p. 102218, 2024.

https://doi.org/10.1016/j.inffus.2023.102218

[7] C. E. Izard, "The many meanings/aspects of emotion: Definitions, functions, activation, and regulation," Emotion Review, vol. 2, no. 4, pp. 363-370, 2010.

https://doi.org/10.1177/1754073910374661

[8] C. Kauschke, D. Bahn, M. Vesker, and G. Schwarzer, "The role of emotional valence for the processing of facial and verbal stimuli—positivity or negativity bias?," Frontiers in psychology, vol. 10, p. 444703, 2019.

https://doi.org/10.3389/fpsyg.2019.01654

[9] S. Kim and S.-P. Lee, "A BiLSTM–Transformer and 2D CNN Architecture for Emotion Recognition from Speech," Electronics, vol. 12, no. 19, p. 4034, 2023.

https://doi.org/10.3390/electronics12194034

[10] D. Avola et al., "Spatio-Temporal Image-Based Encoded Atlases for EEG Emotion Recognition," International Journal of Neural Systems, vol. 34, no. 5, p. 2450024, 2024.

http://dx.doi.org/10.1142/S0129065724500242

[11] Y. Chen, H. Zhang, J. Long, and Y. Xie, "Temporal shift residual network for EEG-based emotion recognition: a 3D feature image sequence approach," Multimedia Tools Applications, vol. 83, no. 15, pp. 45739-45759, 2024.

https://doi.org/10.1007/s11042-023-17142-7

[12] N. Kumari and R. Bhatia, "Saliency map and deep learning based efficient facial emotion recognition technique for facial images," Multimedia Tools Applications, vol. 83, no. 12, pp. 36841-36864, 2024.

https://doi.org/10.1007/s11042-023-16220-0

[13] M. K. Chowdary, T. N. Nguyen, and D. J. Hemanth, "Deep learning-based facial emotion recognition for human–computer interaction applications," Neural Computing Applications, vol. 35, no. 32, pp. 23311-23328, 2023.

https://doi.org/10.1007/s00521-021-06012-8

[14] K. Korovai, D. Zhelezniakov, O. Yakovchuk, O. Radyvonenko, N. Sakhnenko, and I. Deriuga, "Handwriting Enhancement: Recognition-Based and Recognition-Independent Approaches for On-device Online Handwritten Text Alignment," IEEE Access, 2024.

https://doi.org/10.1109/ACCESS.2024.3412433

[15] S. Zhang, Y. Yang, C. Chen, X. Zhang, Q. Leng, and X. Zhao, "Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospects," Expert Systems with Applications, vol. 237, p. 121692, 2024.

https://doi.org/10.1016/j.eswa.2023.121692

[16] P. Chakraborty, S. Ahmed, M. A. Yousuf, A. Azad, S. A. Alyami, and M. A. Moni, "A human-robot interaction system calculating visual focus of human’s attention level," IEEE Access, vol. 9, pp. 93409-93421, 2021.

https://doi.org/10.1109/ACCESS.2021.3091642

[17] S. Mishra, N. Bhatnagar, P. P, and S. T. R, "Speech emotion recognition and classification using hybrid deep CNN and BiLSTM model," Multimedia Tools Applications, vol. 83, no. 13, pp. 37603-37620, 2024.

https://doi.org/10.1007/s11042-023-16849-x

[18] M. Aslan, M. Baykara, and T. B. Alakuş, "Analysis of brain areas in emotion recognition from EEG signals with deep learning methods," Multimedia Tools Applications, vol. 83, no. 11, pp. 32423-32452, 2024.

https://doi.org/10.1007/s11042-023-16696-w

[19] M. Vaezi, M. Nasri, F. Azimifar, and M. Mosleh, "BMPA-DSL: Binary Marine Predators Algorithm to Identify Driver’s Different Levels of Stress," presented at the 2023 14th International Conference on Information and Knowledge Technology (IKT), 2023.

https://doi.org/10.1109/IKT62039.2023.10433031

[20] A. Dessai and H. Virani, "Multimodal and Multidomain Feature Fusion for Emotion Classification Based on Electrocardiogram and Galvanic Skin Response Signals," Sci, vol. 6, no. 1, p. 10, 2024.

https://doi.org/10.3390/sci6010010

[21] J. Cheng et al., "Emotion recognition from multi-channel EEG via deep forest," IEEE Journal of Biomedical Health Informatics, vol. 25, no. 2, pp. 453-464, 2020.

https://doi.org/10.1109/JBHI.2020.2995767

[22] M. Vaezi, M. Nasri, F. Azimifar, and M. Mosleh, "Driver Assistance System for Stress Recognition by Handcrafted Feature Extraction and Convolutional Neural Network," in 2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP), 2024, pp. 1-5: IEEE.

https://doi.org/10.1109/AISP61396.2024.10475225

[23] T. Fan et al., "A new deep convolutional neural network incorporating attentional mechanisms for ECG emotion recognition," Computers in Biology Medicine, vol. 159, p. 106938, 2023.

https://doi.org/10.1016/j.compbiomed.2023.106938

[24] P. Chakraborty, M. A. Yousuf, and S. Rahman, "Predicting level of visual focus of human’s attention using machine learning approaches," in Proceedings of International Conference on Trends in Computational and Cognitive Engineering: Proceedings of TCCE 2020, 2021, pp. 683-694: Springer.

https://doi.org/10.1007/978-981-33-4673-4_56

[25] M. J. Al Dujaili, A. Ebrahimi-Moghadam, and A. Fatlawi, "Speech emotion recognition based on SVM and KNN classifications fusion," International Journal of Electrical Computer Engineering, vol. 11, no. 2, p. 1259, 2021.

https://doi.org/0.11591/ijece.v11i2.pp1259-1264

[26] X. Liu, X. Cheng, and K. Lee, "GA-SVM-based facial emotion recognition using facial geometric features," IEEE Sensors Journal, vol. 21, no. 10, pp. 11532-11542, 2020.

https://doi.org/10.1109/JSEN.2020.3028075

[27] M. A. Hasnul, N. A. Ab. Aziz, and A. Abd. Aziz, "Augmenting ECG Data with Multiple Filters for a Better Emotion Recognition System," Arabian Journal for Science Engineering, pp. 1-22, 2023.

https://doi.org/10.1007/s13369-022-07585-9

[28] Q. Gao, C.-h. Wang, Z. Wang, X.-l. Song, E.-z. Dong, and Y. Song, "EEG based emotion recognition using fusion feature extraction method," vol. 79, no. 37, pp. 27057-27074, 2020.

https://doi.org/10.1007/s11042-020-09354-y

[29] X. Wang, J. Zhang, C. He, H. Wu, and L. Cheng, "A novel emotion recognition method based on the feature fusion of single-lead EEG and ECG signals," IEEE Internet of Things Journal, 2023.

https://doi.org/10.1109/JIOT.2023.3320269

[30] S. Wang, J. Qu, Y. Zhang, and Y. Zhang, "Multimodal emotion recognition from EEG signals and facial expressions," IEEE Access, vol. 11, pp. 33061-33068, 2023.

https://doi.org/10.1109/ACCESS.2023.3263670

[31] M. Vaezi and M. Nasri, "AS3-SAE: Automatic Sleep Stages Scoring using Stacked Autoencoders," Frontiers in Biomedical Technologies‏, pp. 400-416, 2023.

https://doi.org/10.18502/fbt.v10i4.13722

[32] S. Katsigiannis and N. Ramzan, "DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices," IEEE journal of biomedical health informatics, vol. 22, no. 1, pp. 98-107, 2017.

https://doi.org/10.1109/JBHI.2017.2688239

[33] S. N. M. S. Ismail, N. A. A. Aziz, and S. Z. Ibrahim, "A comparison of emotion recognition system using electrocardiogram (ECG) and photoplethysmogram (PPG)," Journal of King Saud University-Computer Information Sciences, vol. 34, no. 6, pp. 3539-3558, 2022.

https://doi.org/10.1016/j.jksuci.2022.04.012

[34] M. U. Hossain, M. A. Rahman, M. M. Islam, A. Akhter, M. A. Uddin, and B. K. Paul, "Automatic driver distraction detection using deep convolutional neural networks," Intelligent Systems with Applications, vol. 14, p. 200075, 2022.

https://doi.org/10.1016/j.iswa.2022.200075

[35] R. Yamashita, M. Nishio, R. K. G. Do, and K. Togashi, "Convolutional neural networks: an overview and application in radiology," Insights into imaging, vol. 9, pp. 611-629, 2018.

https://doi.org/10.1007/s13244-018-0639-9

[36] Q. Zhang, T. Gao, X. Liu, and Y. Zheng, "Public environment emotion prediction model using LSTM network," Sustainability, vol. 12, no. 4, p. 1665, 2020.

https://doi.org/10.3390/su12041665

[37] Z. Liu, W. Zhou, and H. Li, "AB-LSTM: Attention-based bidirectional LSTM model for scene text detection," ACM Transactions on Multimedia Computing, Communications, Applications, vol. 15, no. 4, pp. 1-23, 2019.

https://doi.org/10.1145/3356728

[38] S. M. Alarcao and M. J. Fonseca, "Emotions recognition using EEG signals: A survey," IEEE Transactions on Affective Computing, vol. 10, no. 3, pp. 374-393, 2017.

https://doi.org/10.1109/TAFFC.2017.2714671

[39] Y.-L. Hsu, J.-S. Wang, W.-C. Chiang, and C.-H. Hung, "Automatic ECG-based emotion recognition in music listening," IEEE Transactions on Affective Computing, vol. 11, no. 1, pp. 85-99, 2017.

https://doi.org/10.1109/TAFFC.2017.2781732

[40] S. Nita, S. Bitam, M. Heidet, and A. Mellouk, "A new data augmentation convolutional neural network for human emotion recognition based on ECG signals," Biomedical Signal Processing Control, vol. 75, p. 103580, 2022.

https://doi.org/10.1016/j.bspc.2022.103580

[41] C. M. T. Khan, N. A. Ab Aziz, J. E. Raja, S. W. B. Nawawi, and P. Rani, "Evaluation of machine learning algorithms for emotions recognition using electrocardiogram," Emerging Science Journal, vol. 7, no. 1, pp. 147-161, 2022.

http://dx.doi.org/10.28991/ESJ-2023-07-01-011

Majlesi Journal of Electrical Engineering

Hybrid attention-based deep learning network for emotion recognition by ECG signal

References

References

Volume 19, Issue 2
June 2025
Pages 1-8

Hybrid attention-based deep learning network for emotion recognition by ECG signal

References

References

Volume 19, Issue 2June 2025Pages 1-8

Volume 19, Issue 2
June 2025
Pages 1-8