ISL Pose-to-Text Pipeline, Training and Optimizations -- 2

Project Description

I am building a Machine Learning pipeline to translate Indian Sign Language (ISL) poses into text/sentences sign language translation. The dataset consists of pose landmarks stored as .npy files (MediaPipe-style, ~1600 dimensions per frame), with aligned text sentences tokenized using SentencePiece.

My current training setup is based on a Transformer seq2seq model in PyTorch, but I am facing issues with:

Slow BLEU growth and weak generalization (BLEU barely improves over many epochs).
Normalization strategy (per-sample standardization may be removing motion cues; need body-relative normalization).
Training stability (AMP caused exploding gradients; gradient clipping and schedulers need review).
Limited hardware (RTX 3050 Ti, 4 GB VRAM) – need gradient accumulation, mixed precision, and efficient training strategies.

Urgently, I need an experienced ML/DL engineer or researcher who can:

Audit my training code (PyTorch, Transformer seq2seq, tokenization, collate, schedulers, AMP usage).
Fix and improve the pipeline – better preprocessing, normalization, clipping, AMP usage, schedulers, logging.
Provide a reproducible training recipe that achieves faster BLEU growth and stable convergence.
Suggest hyperparameters and model configs suitable for small-GPU training (RTX 3050), and advise if cloud GPUs are required.
Optionally, recommend or prepare final scripts for cloud training or real-time deployment.

Deliverables

Audit ML pipeline script and data with top fixes.
Improved training scripts (PyTorch) with clear README and reproducible results.
Overfit test demonstration (small subset reaching near-100% accuracy).
Hyperparameter/config suggestions for best performance on my hardware.
(Optional) Cloud GPU training recommendation with cost/time estimate.

Ideal Candidate
Strong background in PyTorch, Transformer models, and training stability.
Experience with sign language recognition/translation, pose-based NLP, or seq2seq models.
Practical knowledge of mixed precision (AMP), gradient clipping, schedulers, and reproducibility.
Ability to communicate clearly and document improvements.


Please share:
Relevant past work (esp. sign language / pose translation).
Your proposed approach and plan.
Timeline and milestone-based cost (audit-only / audit+fixes / audit+experiments).