Hand gesture recognition for user interaction in Augmented Reality game

Authors

  • Nguyễn Thị Thanh Tâm Khoa Đa phương tiện
  • Dương Doãn Tùng Faculty of Electrical and Electronic Engineering, PHENIKAA University

Keywords:

Hand Gesture Recognition, Human-Computer Interaction, Augmented Reality, Data Fusion, Transfer Learning, Deep Learning

Abstract

This paper presents a hand gesture recognition system for an interactive augmented reality game, utilizing skeletal and image data to improve accuracy. We collected a comprehensive dataset of hand gestures comprising RGB images and skeletal coordinates for five distinct gestures. A Late Fusion model, which combines skeletal data with RGB image information, was proposed and achieved a test accuracy of 88.20%. This model was successfully integrated into a Unity 3D game, allowing players to control in-game actions through intuitive hand gestures. Experimental results demonstrate the effectiveness of the proposed approach in enhancing user interaction and delivering a highly responsive gaming experience in AR environments.

Author Biography

Dương Doãn Tùng, Faculty of Electrical and Electronic Engineering, PHENIKAA University

Duong Doan Tung is a third-year student at the Faculty of Electrical and Electronic Engineering, Phenikaa University, Vietnam, studying toward a Bachelor of Engineering with a specialization in Robotics and Artificial Intelligence. His research interests include Computer Vision, Reinforcement Learning, and Human-Robot Interaction.

Downloads

Published

2025-06-27

Issue

Vol. 1 No. 3 (2024): Journal of Science and Technology on Information and Communications

Section

Computer Science