Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes
While recognition of sign language alphabets and isolated words has matured in recent years, recognition of sign language sentences, or continuous signing, is still a research topic of interest in computer vision, especially in signer independent mode of recognition. Existing state-of-the-art soluti...
Saved in:
| Main Author: | |
|---|---|
| Other Authors: | , , , |
| Format: | article |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/11073/25705 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1864513442435563520 |
|---|---|
| author | Almaazmi, Mariam |
| author2 | Elkadi, Salma Elsayed, Lamis Salman, Lina Shanableh, Tamer |
| author2_role | author author author author |
| author_facet | Almaazmi, Mariam Elkadi, Salma Elsayed, Lamis Salman, Lina Shanableh, Tamer |
| author_role | author |
| dc.creator.none.fl_str_mv | Almaazmi, Mariam Elkadi, Salma Elsayed, Lamis Salman, Lina Shanableh, Tamer |
| dc.date.none.fl_str_mv | 2024-10-28T09:47:13Z 2024-10-28T09:47:13Z 2024 |
| dc.identifier.none.fl_str_mv | https://hdl.handle.net/11073/25705 10.1109/access.2024.3485131 2169-3536 |
| dc.language.none.fl_str_mv | en_US |
| dc.publisher.none.fl_str_mv | Institute of Electrical and Electronics Engineers (IEEE) |
| dc.relation.none.fl_str_mv | https://doi.org/10.1109/ACCESS.2024.3485131 |
| dc.rights.none.fl_str_mv | https://creativecommons.org/licenses/by-nc-nd/4.0/ |
| dc.source.none.fl_str_mv | IEEE Access 1 1 |
| dc.subject.none.fl_str_mv | Computer vision Continuous sign language recognition Deep learning Recurrent neural networks Video analysis |
| dc.title.none.fl_str_mv | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes |
| dc.type.none.fl_str_mv | Postprint Peer-Reviewed info:eu-repo/semantics/publishedVersion info:eu-repo/semantics/article |
| description | While recognition of sign language alphabets and isolated words has matured in recent years, recognition of sign language sentences, or continuous signing, is still a research topic of interest in computer vision, especially in signer independent mode of recognition. Existing state-of-the-art solutions in the continuous Arabic Sign Language Recognition (ArSLR) are promising; however, when implemented in signer independent mode, the accuracies drop noticeably. In this paper, we propose a solution for recognizing continuous Arabic signing in signer dependent and independent modes through the use of motion images with positioning information. Initially, sign videos are converted into several motion images, each emphasizing a different segment of the sentence. This is achieved by a weighted sum of residual images after applying optical flow and motion compensation. Each motion image is composed of the whole sentence video, hence putting the emphasized segment into context in terms of previous and successive sign words. Thereafter, hand-crafted features are calculated from each resultant image, including numerical summaries of the horizontal, vertical and diagonal profiles. With such features, the architecture used for model generation and testing is simplified, where it consists of a single Bi-LSTM layer followed by dropout, softmax, and classification layers. This paper makes use of a recent Arabic Sign Language dataset known as ArabSign. The dataset is composed of 6 signers and 93 words arranged into 50 sentences with 30 repetitions each. Experimental results revealed that the proposed solution is suitable for signer dependent and signer independent modes of continuous sign language recognition. Using a Leave-One-Signer-Out policy, the proposed solution achieved word-recognition rates of 99.8% and 75.3%, respectively. These results noticeably surpass relevant state-of-the-art solutions. |
| format | article |
| id | aus_605527dffbc3787e21fca7784d73904e |
| identifier_str_mv | 10.1109/access.2024.3485131 2169-3536 |
| language_invalid_str_mv | en_US |
| network_acronym_str | aus |
| network_name_str | aus |
| oai_identifier_str | oai:repository.aus.edu:11073/25705 |
| publishDate | 2024 |
| publisher.none.fl_str_mv | Institute of Electrical and Electronics Engineers (IEEE) |
| repository.mail.fl_str_mv | |
| repository.name.fl_str_mv | |
| repository_id_str | |
| rights_invalid_str_mv | https://creativecommons.org/licenses/by-nc-nd/4.0/ |
| spelling | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent ModesAlmaazmi, MariamElkadi, SalmaElsayed, LamisSalman, LinaShanableh, TamerComputer visionContinuous sign language recognitionDeep learningRecurrent neural networksVideo analysisWhile recognition of sign language alphabets and isolated words has matured in recent years, recognition of sign language sentences, or continuous signing, is still a research topic of interest in computer vision, especially in signer independent mode of recognition. Existing state-of-the-art solutions in the continuous Arabic Sign Language Recognition (ArSLR) are promising; however, when implemented in signer independent mode, the accuracies drop noticeably. In this paper, we propose a solution for recognizing continuous Arabic signing in signer dependent and independent modes through the use of motion images with positioning information. Initially, sign videos are converted into several motion images, each emphasizing a different segment of the sentence. This is achieved by a weighted sum of residual images after applying optical flow and motion compensation. Each motion image is composed of the whole sentence video, hence putting the emphasized segment into context in terms of previous and successive sign words. Thereafter, hand-crafted features are calculated from each resultant image, including numerical summaries of the horizontal, vertical and diagonal profiles. With such features, the architecture used for model generation and testing is simplified, where it consists of a single Bi-LSTM layer followed by dropout, softmax, and classification layers. This paper makes use of a recent Arabic Sign Language dataset known as ArabSign. The dataset is composed of 6 signers and 93 words arranged into 50 sentences with 30 repetitions each. Experimental results revealed that the proposed solution is suitable for signer dependent and signer independent modes of continuous sign language recognition. Using a Leave-One-Signer-Out policy, the proposed solution achieved word-recognition rates of 99.8% and 75.3%, respectively. These results noticeably surpass relevant state-of-the-art solutions.American University of SharjahInstitute of Electrical and Electronics Engineers (IEEE)2024-10-28T09:47:13Z2024-10-28T09:47:13Z2024PostprintPeer-Reviewedinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/articlehttps://hdl.handle.net/11073/2570510.1109/access.2024.34851312169-3536IEEE Access11en_UShttps://doi.org/10.1109/ACCESS.2024.3485131https://creativecommons.org/licenses/by-nc-nd/4.0/oai:repository.aus.edu:11073/257052024-10-28T11:44:35Z |
| spellingShingle | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes Almaazmi, Mariam Computer vision Continuous sign language recognition Deep learning Recurrent neural networks Video analysis |
| status_str | publishedVersion |
| title | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes |
| title_full | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes |
| title_fullStr | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes |
| title_full_unstemmed | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes |
| title_short | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes |
| title_sort | Motion Images with Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent Modes |
| topic | Computer vision Continuous sign language recognition Deep learning Recurrent neural networks Video analysis |
| url | https://hdl.handle.net/11073/25705 |