**Enhancing Sign Language Learning with Augmented Reality** Hawa Mohd Ekhsan 1\*, Mohamad Daniel Muhamad Khairi 2, Jiwa Noris Hamid 3 1,2,3Faculty of Computer and Mathematical Sciences, Universiti Teknologi Mara Perlis Branch, Arau Campus, 02600 Arau, Perlis, Malaysia. Corresponding author: \* hawame@uitm.edu.my Received Date: \*date Accepted Date: \*date Published Date: \*date **HIGHLIGHTS** - People's awareness of learning sign language is still low because they think it is unnecessary to learn it. - Even though sign language is for impaired people, it is also essential for normal people to learn to convey information and communicate efficiently. - This application is targeted at normal people as a sign language learning tool, and augmented reality will make the learning process effective and exciting. ABSTRACT *Sign language is the communication language used by people with disabilities, especially deaf and hearing-impaired people. The communication between normal and disabled people using sign language will help them carry out their daily activities. Unfortunately, normal people are not aware of the importance of sign language because they are not directly dealing with disabled persons. Besides, some normal people found that sign language is difficult to learn. This study focuses on developing a mobile application for sign language with augmented reality features. This application is targeted at normal people as a sign language learning tool, and augmented reality will make the learning process effective and exciting. The methodology for this study is the ADDIE model that consists of five phases, namely Analysis, Design, Development, Implementation, and Evaluation. This application will assist the users in learning sign language interactively and interestingly. The evaluation of the application was done by the expert in the related background and the normal people. The usability test result revealed that the sign language application is usable for normal people to know and learn the basic sign language. In conclusion, the sign language application is an interesting application for normal people to use and learn sign language.* *Keywords: Sign language, augmented reality, mobile application* # INTRODUCTION Sign language is a language that uses hand signs and other motions, including facial expressions and body gestures. This visual language is designed to aid deaf or hearing-impaired people and does not have a standard written form (Bragg, 2020). Many deaf or hearing-impaired people worldwide use sign language as their primary communication channel. However, some normal people use sign languages, such as the teacher who teaches the deaf students or the family members of the deaf people. As the number of deaf or hearing-impaired people increases, sign language is essential for normal people as it can assist them to communicate and remove the barriers with deaf people. Normal people with sign language expertise also can build their careers as sign language interpreters. There are many ways to learn sign language, such as using a book or paper-based approach, online classes, online videos, and mobile applications. The conventional way of learning by paper-based materials is less effective as the hand movements are not obvious. The online courses and videos have a limitation where the youngsters are less interested. Currently, the mobile application for learning sign language has become a trend (Hafit et al., 2019), and it can be a great resource to a broader range of users. Mobile application is flexible because the learning process can be done at any time without pressure to complete it. Thus, using the mobile application is seen as the best approach, but the app should be added with attractive features to attract children and teenagers. The development of the mobile application for sign language can be enhanced with Augmented Reality (AR). AR is a real-time direct or indirect look of a physical, real-world environment that can be enhanced by adding virtual computer-generated information to it. Today, many mobile applications have been integrated with AR to enhance capabilities and user experience. According to Almutairi and Al-Megren (2017), deaf children's visual literacy can be developed with the support of AR. Since AR provides the way to learn sign language in a digital environment and a more interactive way, this study has explored this approach. Thus, this project is aimed to develop a mobile application that can be used to learn letters and numbers using sign language with AR features. The AR feature provides the three-dimensional (3D) model contents when the user scans the sign language materials. # RELATED WORKS The use of mobile applications for learning sign language may be convenient for the user as it allows the user to interact with the mobile application and can increase understanding. Research conducted by Razalli, Mamat, Razali, M-Yassin, Lakulu, Hashim, and Ariffin (2021) has developed a mobile application for prayer learning that focuses on the hearing impaired community. This research has identified that the design of the mobile prayer application should include features such as graphics, animation, voice, video, text (size, font, colour), and arrangement. The feedbacks from 276 respondents indicate that the application helps hearing-impaired people to perform their prayer and can attract them to learn effectively and systematically. Combining augmented reality and sign language application can help in educational development. Almutairi and Al-Megren (2017) have developed an AR application for reading and writing skills among Arabic children with hearing problems who have difficulties adapting to the lessons. The deaf children have been taught thirteen new words using two approaches: using words and sign language. The Sign Language Teaching Model (SLTM) is used with two levels of education; the first level is learning the correct use of a sign in conjunction with their visual and written representations, and the second level is using verbalizing the words by imitating face, mouth and tongue movements. The finding from this paper shows that the results of the participants who learned through the Augmented Reality application completed more tasks successfully than participants who learned new words through traditional approaches. These findings encourage the use of AR in and out of the classroom to support the development of hearing-impaired children's literacy. Another research conducted by Deb et al. (2018) uses marked-based AR to develop the sign language teaching aid. The modelling method that the research use for modeling and rigging is the hand model by using the software blender, then the hand model has been transferred to the Unity platform. The researcher also invited ten children to test the AR application with 3D animated sign gestures on the mobile system. The findings from this research paper show that the children involved in the testing were highly excited and enthusiastic. 4 out of 10 boys were involved in these experiments were able to correctly reproduce the sign language with their hands' gesture while the three others did not give a specific response. AR also has been used in Soogund and Joseph (2019) research as a sign language translator. This application has been used as the translator tool for deaf and hearing-impaired children to learn English and sign language. This sign language translator application was developed as an educational tool for deaf and hearing-impaired children. By using this application, the children will be able to learn English using signs, while hearing users will learn signs using English. This paper aims to shrink the communication barriers between hearing and non-hearing people. The researcher also hopes the children can have a better education and adapt to society. The researcher states that the children are happy to use the application because the animated character attracts the children to be fascinated and interested in learning. # METHODOLOGY The development model for this study is the ADDIE Instructional Design Model. The ADDIE model is a basic model that can be applied to learning solutions since it is simple and has become the basis of other instructional design models (Saidin et al., 2016). This ADDIE model has five phases: Analysis, Design, Development, Implementation, and Evaluation, as illustrated in Figure 1. ![C:\\Users\\fujitsu\\Documents\\uitm sem5\\chapter 1 csp600\\addie\_model.PNG](62410c59df80d_media/media/image1.png) **Figure 1:** The Phases in ADDIE Model The Analysis phase involved several activities related to problem identification, setting the research goals and objectives, a market survey on existing tools, and other research requirements. In the Design phase, the flowchart has been created to show the flow of the mobile app processes. Then, the storyboard was sketched to illustrate the look and design of the mobile app. The Development phase is the essential part of this research since the researcher has to put a lot of effort and time into developing the 3D model, target image, mobile app interfaces, and sound. The 3D model has been created using Blender software, while Adobe Photoshop has been used to create the target images. The target images are saved in Vuforia, and the 3D models are imported to Unity3d software. The mobile app has been evaluated to the real users in the Implementation phase through Expert Review and Usability Testing. The Expert Review has been conducted to get feedback from experts in related fields. It is important to know whether the mobile app design process was successful, which part of the app can be improved, and how well it works. The feedback from respondents was analyzed to get the research findings. The final step is the Evaluation phase to ensure the mobile app and its contents achieve the learning objectives. **MODULES AND SCREEN SAMPLES** The target image would be used as the mark for the AR application to recognize a 3D model that will appear after the user scans it. Figure 2 shows some of the target images representing letter and sign images. ![](62410c59df80d_media/media/image2.png) ![](62410c59df80d_media/media/image3.png) **Figure 2:** Samples of target images This application has 3 modules namely “Belajar Isyarat Tangan” (Learn SignLanguage), “Belajar Huruf” (Learn Alphabet) and “Belajar Nombor” (Learn Number) as shown in Figure 3. ![](62410c59df80d_media/media/image4.jpeg) ![](62410c59df80d_media/media/image5.jpeg) **Figure 3:** The home page and the module page In the "Belajar Isyarat Tangan” module, the user needs to scan the target image either alphabet or number, and the mobile app will display the sign language in 3D form, as shown in Figure 4. ![](62410c59df80d_media/media/image6.png)![](62410c59df80d_media/media/image7.png)![](62410c59df80d_media/media/image8.png) **Figure 4:** “Belajar Isyarat Tangan” Module In “Belajar Huruf” and “Belajar Nombor" modules, the user will scan the sign target image, and then the mobile app will display the alphabet or number scanned by the app as shown in Table 1. **Table 1:** Sample of Target Image and Its Scanned Image | Target Image | Scanned Image | | ------------------------------------------ | ------------------------------------------ | | ![](62410c59df80d_media/media/image9.png) | ![](62410c59df80d_media/media/image10.png) | | ![](62410c59df80d_media/media/image11.png) | ![](62410c59df80d_media/media/image12.png) | **FINDINGS AND DISCUSSIONS** Usability evaluation has been conducted by inviting participants from different sign language knowledge, gender, computer literacy level, and roles (family member, teacher, or public people) to use the mobile application. The participants were then asked to answer a questionnaire designed to gather their feedback on the usability of the mobile application. The respondents need to answer the questionnaire based on the Likert Scale, as shown in Table 2 for the Expert Review and Usability Testing. **Table 2:** Likert Scale Range | Range | Description | | ----- | ----------------- | | 1 | Strongly Disagree | | 2 | Disagree | | 3 | Neutral | | 4 | Agree | | 5 | Strongly Agree | Expert Review has been conducted by three experts from the multimedia and computer science field to identify usability problems related to the user interface design. The experts were required to perform several tasks related to the user interface, content, and usefulness. Then, the experts have given their feedback on whether they can successfully do the task or failed to do it. The tasks are listed in Table 3, which shows that all of the experts successfully did the tasks given. **Table 3:** Tasks for Expert Review and the feedbacks
No. Task

Feedback

Success (/), Failed (X)
Respondent 1 Respondent 2 Respondent 3
1 Click the "Mula” button to start the application. / / /
2 Click the Help button (?) to learn how to use the application. / / /
3 Click the “Belajar isyarat tangan” module to learn the letters or numbers by scanning the target image. / / /
4 Choose and scan one target image from the provided image in folder “Isyarat Tangan” / / /
5 Click button “Huruf” at the bottom of the application to learn sign language for letters. / / /
6 Choose and scan one target image for “Huruf” from the provided image in folder “hurufScan” / / /
7 Click button “Nombor” at the bottom of the application to learn sign language for numbers. / / /
8 Choose and scan one target image for “Nombor" from the provided image in the folder "NomborScan” / / /
9 Click the button to go back to the homepage. / / /
10 Click the "Keluar" button to exit from the application. / / /
11 Click “Ya” to exit the application and “Tidak” to stay in the application. / / /
The experts also evaluate the main criteria of the application that are related to the user interface, content, and usefulness. The mean score for each criterion is summarized in Table 4. **Table 4:** Mean score for each criterion by experts | Criteria | Mean Score | | ---------------- | ---------- | | Interface design | 4.4 | | Content | 4.6 | | Usefulness | 4.6 | For the interface design feedback, all of the experts agreed that the interface of the sign language application was appealed with a mean score is 4.4. In terms of the content, all experts agreed (mean score 4.6) that the content is simple, understandable, and 3D models are easy to understand. All experts also agreed that this application is simple to use and easy to learn, which remarks the usefulness with a mean score of 4.6. On the other hand, usability testing has been performed to determine the responses from real users and whether the project met the objectives. A total of 17 respondents were involved in this testing, but only four knew how to communicate using sign language. Two respondents have good knowledge of sign language, while the other two are at a moderate level. The feedback from the usability evaluation is presented in Table 5. **Table 5:** Mean scores for Usability Testing | Criteria | Mean score | | ---------------- | ---------- | | Interface design | 3.7 | | Content | 3.9 | | Usefulness | 3.8 | The mean scores in Table 5 indicate that the mobile app needs to be improved to make it more useful to users. The important part that should be focused on is the 3D image modelling to ensure that the images are realistic and perfectly represent the real-world object. In this case, the 3D model for hand sign language is imperfect and needs to be more visually impressive. **CONCLUSION AND RECOMMENDATIONS** This application is designed and developed for normal people with a more engaging approach to ensure that learning can be done at the user's pace and that the user can learn anytime and anywhere. This approach would also increase the users’ experience when using the application as they can improve their self-directed learning skills. However, based on the findings, this application needs some improvements specifically in 3D modeling for hand signs and alphabets requiring high-quality, detailed, and smoother 3D images. This improvement is crucial mainly in hand signs since users need a more precise guide in learning sign language concerning the placement of each finger. The 3D models for hand signs should be able to be rotated in different angles, and all edges are in curved shape form that represents the human hand naturally and realistically. Hence, one of the challenges in this study is to ensure that sign language's real and virtual representations are efficiently affected by each other. In conclusion, this mobile application provides users with several positive implications since it can be used as a self-learning tool and attract normal people to learn sign language. The learning process can be improved by including more interactivity and engagement features such as a quiz module to test the user’s understanding of the sign language. This application would benefit parents, family members, the community, and the hearing-impaired people to communicate with each other effectively. Besides that, learning sign language may enhance the career, such as educators, social workers, counselors, translators, or other service providers. **REFERENCES** Almutairi, A., & Al-Megren, S. (2017). Augmented reality for the literacy development of deaf children: A preliminary investigation. *Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2017)*, 359–360. Bragg, D. (2020). *Sign Language Interfaces : Discussing the Field ’ s Biggest Challenges. [Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems](https://dl.acm.org/doi/proceedings/10.1145/3334480) (CHI EA '20)*. 1–5. Deb, S., Suraksha, & Bhattacharya, P. (2018). Augmented Sign Language Modeling(ASLM) with interaction design on smartphone - an assistive learning and communication tool for inclusive classroom. *Procedia Computer Science*, *125*, 492–500. Hafit, H., Xiang, C. W., Yusof, M. M., Wahid, N., & Kassim, S. (2019). Malaysian Sign Language Mobile Learning Application : A recommendation app to communicate with hearing-impaired communities. *International Journal of Electrical and Computer Engineering (IJECE)*, *9*(6), 5512. Razalli, A. R., Mamat, N., Razali, N., Yasin, M. H. M., Lakulu, M., Hashim, A. T. M., & Ariffin, A. (2021). Development of Prayer Mobile Application Software for The Hearing Impaired (Deaf) Based on Malaysian Sign Language. *International Journal of Academic Research in Business and Social Sciences*, 11(6), 1108–1122. Saidin, N. F., Halim, N. D. A., & Yahaya, N. (2016). Designing Mobile Augmented Reality (MAR) for Learning Chemical Bonds. In *Proceedings of the 2nd International Colloquium of Art and Design Education Research (i-CADER 2015)* (pp. 367–377). Springer Singapore. Soogund, N.-U.-N., & Joseph, M. H. (2019). SignAR: A Sign Language Translator Application with Augmented Reality using Text and Image Recognition. *2019 IEEE International Conference on Intelligent Techniques in Control, Optimization and Signal Processing (INCOS 2019)*, *3*(1), 1–5.