**Enhancing Sign Language Learning with Augmented Reality**

\*\*Double blind review, please do not include authors information in this version \*\*

Received Date: \*date

Accepted Date: \*date

Published Date: \*date

**HIGHLIGHTS (Arial, 11)**

  - The awareness of people to learn sign language is still low because they think that there is no necessity to learn the language.

  - Even though sign language is for impairment people, but it is also important for normal people to learn it to convey information and communicate efficiently.

  - This application is targeted for normal people as a sign language learning tool and augmented reality will make the learning process become effective and interesting.

ABSTRACT

*Sign language is the communication language used by people with disabilities especially deaf and hearing-impaired people. The communication between normal and disabilities people using the sign language will help them to carry out their daily activities. Unfortunately, normal people are not aware of the importance of sign language because they are not directly dealing with the disabilities person. Besides, some of the normal people found that the sign language is difficult to learn. This study focuses on the development of mobile application for sign language with augmented reality feature. This application is targeted for normal people as a sign language learning tool and augmented reality will make the learning process become effective and interesting. The methodology for this study is ADDIE model that consists of five phases namely Analysis, Design, Development, Implementation, and Evaluation. This application will assist the users to learn sign language in interactive and interesting way. The evaluation of the application was done by the expert in related background and also the normal people. The result of usability test revealed that the sign language application is usable for the normal people to know and learn the basic sign language. In conclusion, the sign language application is an interesting application for the normal people to use and learn the sign language.*

*Keywords: Sign language, augmented reality, mobile application*

# INTRODUCTION

Sign language is a language that uses hand signs and other motions, including facial expressions and body gestures. This language is a visual language designed to aid deaf or hearing impaired people, and do not have a standard written form (Bragg, 2020). Many deaf or hearing impaired people around the world use sign language as their primary channel of communication.

However, there are also normal people that use sign language such as teacher who teach the deaf students or the family members of the deaf people. As the number of deaf or hearing-impaired people increases, sign language is important for normal people as it can assist them to communicate and remove the barriers with deaf people. Normal people with sign language expertise also can built their career as a sign language interpreter.

There are many ways to learn sign language such as using book or paper-based approach, online classes, online videos, and mobile application. The conventional way of learning by paper-based materials are less effective as the hand movements are not obvious to see. The online courses and videos has limitation where the youngsters are less interested in that way. Currently, mobile application for learning sign language has become a trend (Hafit et al., 2019) and it can be a great resource to a broader range of users. Mobile application is more flexible because the learning process can be done at any time without pressure to complete it. Thus, the use of mobile application is seen to be the best approach but in order to attract children and teenagers, the app should be added with attractive features.

The development of mobile application for sign language can be enhanced with Augmented Reality (AR). AR is a real time direct or indirect look of a physical real-world environment that can been enhanced by adding virtual computer-generated information to it. Today, many mobile applications have been integrated with AR to enhance capabilities and user experience. According to Almutairi and Al-Megren (2017), the deaf children visual literacy can be developed with the support of AR. Since AR provides the way to learn sign language in digital environment and in more interactive way, this approach has been explored in this study.

Thus, this project is aimed to develop a mobile application that can be used to learn letters and numbers using sign language with AR feature. The AR feature provides the three dimensional (3D) model contents when user scan the sign language materials.

# RELATED WORKS

The use of mobile application for learning sign language may be convenient for user as it allows user to interact with the mobile application and can increase understanding. A research conducted by Razalli, Mamat, Razali, M-Yassin, Lakulu, Hashim and Ariffin (2021) has developed a mobile application for prayer learning that focuses on the hearing impaired community. This research has identified that the design of the prayer mobile application should include features such as graphics, animation, voice, video, text (size, font, colour) and arrangement. The feedbacks from 276 respondents indicate that the application helps the hearing impaired people to perform their prayer and can attract them to learn in an effective and systematic manner.

The combination of augmented reality and the sign language application can help in educational development. Almutairi and Al-Megren (2017) have developed an AR application for reading and writing skills among Arabic children with hearing problems where they have difficulties in adapting the lessons. The deaf children have been taught with thirteen new words using two approaches which are using word and using sign language. Sign Language Teaching Model (SLTM) is used with two levels of education; first level is learning of the correct use of a sign in conjunction with their visual and written representations, and second level is using verbalizes the words by doing imitation of face, mouth and tongue movements. The finding from this paper shows that the results of the participants who learned through the Augmented Reality application completed more tasks successfully than participants who learned new words through traditional approaches. These findings encourage the use of AR in and out of the classroom to support the development of hearing impaired children's literacy.

Another research conducted by Deb et al., (2018) uses marked-based AR for development of the sign language teaching aid. The modelling method that the research use to modeling and rigging the hand model by using the software blender, then the hand model has been transferred to Unity platform. The researcher also invited 10 children to test the AR application with 3D animated sign gesture on mobile system. The findings from this research paper show that the children that involved in the testing were extremely excited and enthusiastic. 4 out of 10 boys were involved in these experiments were able to correctly reproduce the sign language with their hands gesture while the 3 others did not give a specific response.

AR also has been used in Soogund and Joseph (2019) research as a sign language translator. This application has been used as the translator tool for the deaf and hearing- impaired children to learn English and sign language. This sign language translator application was developed as the education tool for the deaf and hearing-impaired children. By using this application, the children will be able to learn English using signs while hearing users will learn signs using English. The objective of this paper is to shrink the communication barriers between the hearing and non-hearing people. The researcher also hope the children can have a better education and can adapt in the society. The researcher state that the children are happy to use the application because the animated character attracts the children to fascinated and interested to learn.

# METHODOLOGY

The development model for this study is the ADDIE Instructional Design Model. The ADDIE model is a basic model that can be applied to learning solutions of any type since it is simple and become the basis of other instructional design models (Saidin et al., 2016). This ADDIE model has five phases namely Analysis, Design, Development, Implementation, and Evaluation as illustrated in Figure 1.

![C:\\Users\\fujitsu\\Documents\\uitm sem5\\chapter 1 csp600\\addie\_model.PNG](622a7379c3059_media/media/image1.png)

**Figure 1:** The Phases in ADDIE Model

The Analysis phase involved several activities that related to problem identification, setting the research goals and objectives, market survey on existing tools and other research requirements. In Design phase, the flowchart has been created to show the flow of the mobile app processes. Then, the storyboard has been sketched to illustrate the look and design of the mobile app.

The Development phase is the most important part in this research since the researcher have to put a lot of efforts and time to develop the 3D model, target image, mobile app interfaces, including the sound. The 3D model has been created using Blender software, while Adobe Photoshop has been used to create the target images. The target images are saved in Vuforia and the 3D models are imported to Unity3d software.

The mobile app has been evaluated to the real users in the Implementation phase through Expert Review and Usability Testing. The Expert Review has been conducted to get feedback from the experts in related fields. It is important to know whether the mobile app design process was a success, which part of the app can be improved and how well the app works. The feedback from respondents were analyzed to get the research findings. The final step is Evaluation phase that intended to ensure the mobile app and its contents achieved the learning objectives.

**MODULES AND SCREEN SAMPLES**

The target image would be used as the mark for AR application to recognize 3D model that will appear after user scan it. Figure 2 shows some of the target images that represent letter and sign image.

![](622a7379c3059_media/media/image2.png) ![](622a7379c3059_media/media/image3.png)

**Figure 2:** Samples of target images

This application has 3 modules namely “Belajar Isyarat Tangan” (Learn SignLanguage), “Belajar Huruf” (Learn Alphabet) and “Belajar Nombor” (Learn Number) as shown in Figure 3.

![](622a7379c3059_media/media/image4.jpeg) ![](622a7379c3059_media/media/image5.jpeg)

**Figure 3:** The home page and the module page

In “Belajar Isyarat Tangan” module, user needs to scan the target image either alphabet or number and the mobile app will display the sign language in 3D form as shown in Figure 4.

![](622a7379c3059_media/media/image6.png)![](622a7379c3059_media/media/image7.png)![](622a7379c3059_media/media/image8.png)

**Figure 4:** “Belajar Isyarat Tangan” Module

In “Belajar Huruf” and “Belajar Nombor” modules, user will scan the sign target image, then the mobile app will display the alphabet or number scanned by the app as shown in Table 1.

**Table 1:** Sample of Target Image and Its Scanned Image

| Target Image                               | Scanned Image                              |
| ------------------------------------------ | ------------------------------------------ |
| ![](622a7379c3059_media/media/image9.png)  | ![](622a7379c3059_media/media/image10.png) |
| ![](622a7379c3059_media/media/image11.png) | ![](622a7379c3059_media/media/image12.png) |

**FINDINGS AND DISCUSSIONS**

Usability evaluation has been conducted by inviting participants from a mixture of different sign language knowledge, gender, computer literacy level and roles (family member, teacher or public people) to use the mobile application. The participants were then asked to answer a questionnaire that was designed to gather their feedback on the usability of the mobile application.

For the Expert Review and Usability Testing, the respondents need to answer the questionnaire based on the Likert Scale as shown in Table 2.

**Table 2:** Likert Scale Range

| Range | Description       |
| ----- | ----------------- |
| 1     | Strongly Disagree |
| 2     | Disagree          |
| 3     | Neutral           |
| 4     | Agree             |
| 5     | Strongly Agree    |

Expert Review has been conducted by three experts from multimedia and computer science field that is aimed to identify usability problems that related to the user interface design. The experts were required to perform several tasks and accomplished it. Then, the experts have given their feedback either they can successfully do the task or failed to do it. Based on the feedback, all of the experts are successfully did the tasks given.

For the interface design feedback, all of the experts agreed that the interface of the sign language application was appealed. In term of the content of the application, all experts agreed that the content is simple, understandable, and the 3D models are easy to understand. Furthermore, all experts agreed that this application is simple to use and easy to learn.

On the other hand, usability testing has been performed to find out the responds from real users and either the project met the objectives. A total of 17 respondents involved in this testing but only four respondents knew how to communicate using the sign language. There are two respondents that have a good knowledge in sign language while the other two are in moderate level. The feedback from usability evaluation is presented in Table 3.

**Table 3:** Mean scores for Usability Testing

| Criteria         | Mean score |
| ---------------- | ---------- |
| Interface design | 3.7        |
| Content          | 3.9        |
| Usefulness       | 3.8        |

The mean scores in Table 3 indicates that the mobile app need to be improved to make it more usable to users. The important part that should be focused is the 3D images modelling to ensure that the images are realistic and perfectly represent the real world object. In this case, the 3D model for hand sign language is imperfect and need to be more visually impressive.

**CONCLUSION AND RECOMMENDATIONS**

This mobile application provides users with several positive implications since it can be used as a self-learning and can attract normal people to learn the sign language. The learning process can be improved by including more interactivity and engagement such as quiz module to test the user understanding of the sign language. This would also increase the user experience when using the application.

**REFERENCES**

Almutairi, A., & Al-Megren, S. (2017). Augmented reality for the literacy development of deaf children: A preliminary investigation. *Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2017)*, 359–360.

Bragg, D. (2020). *Sign Language Interfaces : Discussing the Field ’ s Biggest Challenges. [Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems](https://dl.acm.org/doi/proceedings/10.1145/3334480) (CHI EA '20)*. 1–5.

Deb, S., Suraksha, & Bhattacharya, P. (2018). Augmented Sign Language Modeling(ASLM) with interaction design on smartphone - an assistive learning and communication tool for inclusive classroom. *Procedia Computer Science*, *125*, 492–500.

Hafit, H., Xiang, C. W., Yusof, M. M., Wahid, N., & Kassim, S. (2019). Malaysian Sign Language Mobile Learning Application : A recommendation app to communicate with hearing-impaired communities. *International Journal of Electrical and Computer Engineering (IJECE)*, *9*(6), 5512.

Razalli, A. R., Mamat, N., Razali, N., Yasin, M. H. M., Lakulu, M., Hashim, A. T. M., & Ariffin, A. (2021). Development of Prayer Mobile Application Software for The Hearing Impaired (Deaf) Based on Malaysian Sign Language. *International Journal of Academic Research in Business and Social Sciences*, 11(6), 1108–1122.

Saidin, N. F., Halim, N. D. A., & Yahaya, N. (2016). Designing Mobile Augmented Reality (MAR) for Learning Chemical Bonds. In *Proceedings of the 2nd International Colloquium of Art and Design Education Research (i-CADER 2015)* (pp. 367–377). Springer Singapore.

Soogund, N.-U.-N., & Joseph, M. H. (2019). SignAR: A Sign Language Translator Application with Augmented Reality using Text and Image Recognition. *2019 IEEE International Conference on Intelligent Techniques in Control, Optimization and Signal Processing (INCOS 2019)*, *3*(1), 1–5.