ASL Real-time Detection from Video/Image
Web and mobile application for real-time recognition of ASL signs from a camera or image.
ASL Real-time Detection recognizes American Sign Language letters from a webcam, mobile camera, or imported image. The system detects up to two hands, extracts their landmarks, applies a classification model, and returns the prediction with its confidence level.
Problem addressed
Sign language recognition from a camera requires a system capable of accurately detecting hands, interpreting their positions, and providing stable predictions in real-time on the web and on mobile.
Objectives
Architecture
The FastAPI backend loads a MediaPipe Hand Landmarker detector and a TorchScript classifier at startup. For each received image, it resizes the image, detects up to two hands, and extracts 21 three-dimensional landmarks per hand. The landmarks are normalized with respect to the wrist and then transformed into a 137-feature vector. The classifier then applies a prediction, a softmax calculation, an argmax, and a confidence threshold set to 0.55. The React/Vite frontend allows the use of a webcam or the import of an image. The Expo mobile application allows the use of the camera, gallery, WebSocket stream, text composition, and translation to Arabic.