Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information
: In multi-modal emotion aware frameworks, it is essential to estimate the emotional features then fuse them to different degrees. This basically follows either a feature-level or decision-level strategy. In all likelihood, while features from several modalities may enhance the classification perfor...
محفوظ في:
| المؤلف الرئيسي: | |
|---|---|
| مؤلفون آخرون: | , |
| منشور في: |
2019
|
| الموضوعات: | |
| الوصول للمادة أونلاين: | https://bspace.buid.ac.ae/handle/1234/3015 https://doi.org/10.3390/info10070239. |
| الوسوم: |
إضافة وسم
لا توجد وسوم, كن أول من يضع وسما على هذه التسجيلة!
|
| _version_ | 1862980616573681664 |
|---|---|
| author | M. Ghoniem, Rania |
| author2 | D. Algarni, Abeer Shaalan, Khaled |
| author2_role | author author |
| author_facet | M. Ghoniem, Rania D. Algarni, Abeer Shaalan, Khaled |
| author_role | author |
| dc.creator.none.fl_str_mv | M. Ghoniem, Rania D. Algarni, Abeer Shaalan, Khaled |
| dc.date.none.fl_str_mv | 2019 2025-05-14T09:45:23Z 2025-05-14T09:45:23Z |
| dc.identifier.none.fl_str_mv | Rania M. Ghoniem, Abeer D. Algarni and Khaled Shaalan (2019) “Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information,” Information, 10(7), p. 239. https://bspace.buid.ac.ae/handle/1234/3015 https://doi.org/10.3390/info10070239. |
| dc.language.none.fl_str_mv | en |
| dc.publisher.none.fl_str_mv | MDPI |
| dc.relation.none.fl_str_mv | Informationv10 n7 (20190701): 239 |
| dc.subject.none.fl_str_mv | multi-modal emotion aware systems; speech processing; EEG signal processing; hybrid classification models |
| dc.title.none.fl_str_mv | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information |
| dc.type.none.fl_str_mv | Article |
| description | : In multi-modal emotion aware frameworks, it is essential to estimate the emotional features then fuse them to different degrees. This basically follows either a feature-level or decision-level strategy. In all likelihood, while features from several modalities may enhance the classification performance, they might exhibit high dimensionality and make the learning process complex for the most used machine learning algorithms. To overcome issues of feature extraction and multi-modal fusion, hybrid fuzzy-evolutionary computation methodologies are employed to demonstrate ultra-strong capability of learning features and dimensionality reduction. This paper proposes a novel multi-modal emotion aware system by fusing speech with EEG modalities. Firstly, a mixing feature set of speaker-dependent and independent characteristics is estimated from speech signal. Further, EEG is utilized as inner channel complementing speech for more authoritative recognition, by extracting multiple features belonging to time, frequency, and time–frequency. For classifying unimodal data of either speech or EEG, a hybrid fuzzy c-means-genetic algorithm-neural network model is proposed, where its fitness function finds the optimal fuzzy cluster number reducing the classification error. To fuse speech with EEG information, a separate classifier is used for each modality, then output is computed by integrating their posterior probabilities. Results show the superiority of the proposed model, where the overall performance in terms of accuracy average rates is 98.06%, and 97.28%, and 98.53% for EEG, speech, and multi-modal recognition, respectively. The proposed model is also applied to two public databases for speech and EEG, namely: SAVEE and MAHNOB, which achieve accuracies of 98.21% and 98.26%, respectively |
| id | budr_4d658e489efdc3423353e77f3d6b59a9 |
| identifier_str_mv | Rania M. Ghoniem, Abeer D. Algarni and Khaled Shaalan (2019) “Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information,” Information, 10(7), p. 239. |
| language_invalid_str_mv | en |
| network_acronym_str | budr |
| network_name_str | The British University in Dubai repository |
| oai_identifier_str | oai:bspace.buid.ac.ae:1234/3015 |
| publishDate | 2019 |
| publisher.none.fl_str_mv | MDPI |
| repository.mail.fl_str_mv | |
| repository.name.fl_str_mv | |
| repository_id_str | |
| spelling | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain InformationM. Ghoniem, RaniaD. Algarni, AbeerShaalan, Khaledmulti-modal emotion aware systems; speech processing; EEG signal processing; hybrid classification models: In multi-modal emotion aware frameworks, it is essential to estimate the emotional features then fuse them to different degrees. This basically follows either a feature-level or decision-level strategy. In all likelihood, while features from several modalities may enhance the classification performance, they might exhibit high dimensionality and make the learning process complex for the most used machine learning algorithms. To overcome issues of feature extraction and multi-modal fusion, hybrid fuzzy-evolutionary computation methodologies are employed to demonstrate ultra-strong capability of learning features and dimensionality reduction. This paper proposes a novel multi-modal emotion aware system by fusing speech with EEG modalities. Firstly, a mixing feature set of speaker-dependent and independent characteristics is estimated from speech signal. Further, EEG is utilized as inner channel complementing speech for more authoritative recognition, by extracting multiple features belonging to time, frequency, and time–frequency. For classifying unimodal data of either speech or EEG, a hybrid fuzzy c-means-genetic algorithm-neural network model is proposed, where its fitness function finds the optimal fuzzy cluster number reducing the classification error. To fuse speech with EEG information, a separate classifier is used for each modality, then output is computed by integrating their posterior probabilities. Results show the superiority of the proposed model, where the overall performance in terms of accuracy average rates is 98.06%, and 97.28%, and 98.53% for EEG, speech, and multi-modal recognition, respectively. The proposed model is also applied to two public databases for speech and EEG, namely: SAVEE and MAHNOB, which achieve accuracies of 98.21% and 98.26%, respectivelyMDPI2025-05-14T09:45:23Z2025-05-14T09:45:23Z2019ArticleRania M. Ghoniem, Abeer D. Algarni and Khaled Shaalan (2019) “Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information,” Information, 10(7), p. 239.https://bspace.buid.ac.ae/handle/1234/3015https://doi.org/10.3390/info10070239.enInformationv10 n7 (20190701): 239oai:bspace.buid.ac.ae:1234/30152025-05-14T09:48:34Z |
| spellingShingle | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information M. Ghoniem, Rania multi-modal emotion aware systems; speech processing; EEG signal processing; hybrid classification models |
| title | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information |
| title_full | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information |
| title_fullStr | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information |
| title_full_unstemmed | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information |
| title_short | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information |
| title_sort | Multi-Modal Emotion Aware System Based on Fusion of Speech and Brain Information |
| topic | multi-modal emotion aware systems; speech processing; EEG signal processing; hybrid classification models |
| url | https://bspace.buid.ac.ae/handle/1234/3015 https://doi.org/10.3390/info10070239. |