Aplikasi Permainan untuk Edukasi Bahasa Isyarat Menggunakan Computer Vision

Abstract

Learning sign language remains a challenge, particularly for beginners who often struggle to understand gestures without direct feedback. Conventional learning methods such as books or instructional videos on YouTube are often considered monotonous and fail to maintain long-term learning interest. An innovative approach is the integration of technology through educational games, which combine entertainment, challenge, and interactivity to create a more engaging and adaptive learning process. This research developed an RPG-based educational game application to support the learning of Indonesian Sign Language (BISINDO) using computer vision technology. The system integrates the MediaPipe framework to detect hand movements and extract spatial features, which are then fed to a Random Forest Classifier algorithm to classify gestures into alphabet letters. Each classified letter is used as an input action in the game, with data communication handled through the UDP protocol. The evaluation was carried out using the MEEGA (Model for the Evaluation of Educational Games) method (Petri et al., 2016), involving 40 users directly and observing how the game assisted in the process of learning sign language. The test results showed a questionnaire score of 4.3825 (Good) out of 5, which proves that the game is able to provide an interactive learning experience and significantly improve users' understanding of sign language