Okeha/sign-language-translator-backend
Real-time SL detection system featuring WebSocket streaming, VideoMAE fine-tuned on WLASL, and LLM-driven semantic disambiguation. Includes a full pipeline for 3D motion capture (JSON/Three.js) and optimized inference for CUDA and DirectML.
What's novel
Real-time SL detection system featuring WebSocket streaming, VideoMAE fine-tuned on WLASL, and LLM-driven semantic disambiguation. Includes a full pipeline for 3D motion capture (JSON/Three.js) and optimized inference for CUDA and DirectML.
Code Analysis
14 files read · 3 roundsA FastAPI backend that uses a fine-tuned VideoMAE model to classify ASL glosses from webcam video frames in real-time over WebSocket, then uses a pluggable LLM (cloud via OpenRouter or local via HuggingFace) to convert gloss sequences into natural English sentences, with a Streamlit frontend and Med
Strengths
The project has genuine end-to-end depth: a real VideoMAE fine-tuning pipeline with data augmentation, stratified dataset splitting, and class balancing; a well-structured FastAPI service with WebSocket streaming, session management, and a clean factory pattern for pluggable LLM backends; and a MediaPipe motion capture pipeline that produces 3D bone-mapping data for a frontend avatar. The architecture is coherent and the README accurately describes the system.
Weaknesses
Test coverage is essentially nonexistent (only a trivial data_test.py and a few scattered test files), the ChatMemory class is duplicated verbatim across both sentence service files, the /glosses/to-sentence endpoint still contains a placeholder that just joins glosses with spaces instead of calling the LLM, and the training code has hardcoded paths and a hardcoded NUM_CLASSES=282 that doesn't match the dynamic label count derived from the dataset.
Score Breakdown
Signal breakdown
Innovation
Craft
Traction
Scope
Evidence
Commits
28
Contributors
1
Files
2269
Active weeks
11
Repository
Language
Python
Stars
2
Forks
0
License
—