Adapt Bittensor CLM Model Tuning script to work with multiple GPUs
Budget: $30 – $250 USD
I need for someone to adapt the current Bittensor CLM Model Tuning script in order to work with multiple GPUs.
The Bittensor version of the script was adapted from Hugging Face's Transformers/language-modeling code.
Currently the script doesn't work with GPU parallelism and I need someone to make it work with 2/4/6/8 GPUs in parallel in order to be able to train larger models or to train them faster.
The script utilizes the datasets hosted on Bittensor's IPFS Genesis Dataset.
Models that will be trained with it will be: https://huggingface.co/EleutherAI/gpt-neo-1.3B and https://huggingface.co/EleutherAI/gpt-neo-2.7B
The script can be found here:
https://github.com/opentensor/clm_model_tuning
Details about it here:
https://docs.bittensor.com/nested/FineTuning.html
Bittensor dataset here:
https://docs.bittensor.com/nested/TheDataset.html
The Bittensor version of the script was adapted from Hugging Face's Transformers/language-modeling code.
Currently the script doesn't work with GPU parallelism and I need someone to make it work with 2/4/6/8 GPUs in parallel in order to be able to train larger models or to train them faster.
The script utilizes the datasets hosted on Bittensor's IPFS Genesis Dataset.
Models that will be trained with it will be: https://huggingface.co/EleutherAI/gpt-neo-1.3B and https://huggingface.co/EleutherAI/gpt-neo-2.7B
The script can be found here:
https://github.com/opentensor/clm_model_tuning
Details about it here:
https://docs.bittensor.com/nested/FineTuning.html
Bittensor dataset here:
https://docs.bittensor.com/nested/TheDataset.html
Related categories:
Python
Machine Learning (ML)
Artificial Intelligence
Programming
Performance Tuning