Sign Language Recognition Fusion Prototype
Budget: $30 – $250 USD
I am building a proof-of-concept Sign Language Recognition system on my Mac-based lab machine and need a fast, end-to-end implementation that reaches solid Accuracy on the AUTSL dataset. You will have online access to the full 38 k-video corpus, so no shipping drives or USBs; simply script the download and preprocessing.
Here is the workflow I want you to own, code, and document:
• Data preparation – download AUTSL, isolate the RGB stream for every clip, then extract frame-level hand-body keypoints with MediaPipe (OpenPose is fine if you prefer).
• Dual-branch network – an RGB pathway built around a 3D-ResNet (or a comparable spatiotemporal CNN) and a Skeleton pathway driven by either LSTM layers or a Temporal Convolutional stack.
• Mid-level fusion – combine the two streams before the classifier so they jointly vote on the final sign.
• Robustness enhancement – implement Modality Dropout during training to simulate missing channels and toughen the model against scenarios where keypoints fail or footage is blurry.
• Evaluation – report Accuracy clearly; you can mention other metrics in logs, but Accuracy is the headline figure I will compare to baselines.
• Delivery – clean, runnable Python (PyTorch preferred) with comments, a short README explaining the commands, and saved weights. Everything should execute on macOS within three days of project start.
I will test by running your scripts, reproducing preprocessing, and verifying that inference works with either or both modalities present. If the code runs error-free and meets the stated Accuracy target we agree upon, the milestone is complete.
Here is the workflow I want you to own, code, and document:
• Data preparation – download AUTSL, isolate the RGB stream for every clip, then extract frame-level hand-body keypoints with MediaPipe (OpenPose is fine if you prefer).
• Dual-branch network – an RGB pathway built around a 3D-ResNet (or a comparable spatiotemporal CNN) and a Skeleton pathway driven by either LSTM layers or a Temporal Convolutional stack.
• Mid-level fusion – combine the two streams before the classifier so they jointly vote on the final sign.
• Robustness enhancement – implement Modality Dropout during training to simulate missing channels and toughen the model against scenarios where keypoints fail or footage is blurry.
• Evaluation – report Accuracy clearly; you can mention other metrics in logs, but Accuracy is the headline figure I will compare to baselines.
• Delivery – clean, runnable Python (PyTorch preferred) with comments, a short README explaining the commands, and saved weights. Everything should execute on macOS within three days of project start.
I will test by running your scripts, reproducing preprocessing, and verifying that inference works with either or both modalities present. If the code runs error-free and meets the stated Accuracy target we agree upon, the milestone is complete.