Optimize Multi-GPU Compute on Sagemaker
Budget: $25 – $50 USD
I have a basic setup on AWS using Sagemaker SDK. I am using one of the large models ( 3B parameters or more) from Huggingface and try to optimize for memory and fine-tuning of such model.
If you have experience in this kind of optimization, it will be of great help.
I want to use the T5 model and its large variants (3B - 11B params) in this case. My goal is to fine-tune it using my own data. I have gone through documentations on model distributions on HF and Sagemaker. They seem to have different ways to do it. Since these are standard models, can you provide a projection on the following:
1. what GPU instance is required
2. What is the estimated speed up using HF's accelerate or deepspeed?
As far as deliverables, I have the following objectives:
1. able to run t5 large models from HF on one of their existing datasets, e.g. cnn_dailymail, xsum.
minimum: https://huggingface.co/google/flan-t5-large
preferred: https://huggingface.co/google/flan-t5-xl
2. training data minimum dimension: 1024 tokens input, 1024 tokens output. currently, in HF examples, they use 1024 input, 128 output. So my training data requirement will have a huge impact on memory footprint.
3. the training should proceed with reasonable speed, e.g. < 5s/iteration.
4. Model checkpointing and reloading for continued fine-tuning is supported.
5. Timeline: 2 weeks.
Let me know if you have any questions.
If you have experience in this kind of optimization, it will be of great help.
I want to use the T5 model and its large variants (3B - 11B params) in this case. My goal is to fine-tune it using my own data. I have gone through documentations on model distributions on HF and Sagemaker. They seem to have different ways to do it. Since these are standard models, can you provide a projection on the following:
1. what GPU instance is required
2. What is the estimated speed up using HF's accelerate or deepspeed?
As far as deliverables, I have the following objectives:
1. able to run t5 large models from HF on one of their existing datasets, e.g. cnn_dailymail, xsum.
minimum: https://huggingface.co/google/flan-t5-large
preferred: https://huggingface.co/google/flan-t5-xl
2. training data minimum dimension: 1024 tokens input, 1024 tokens output. currently, in HF examples, they use 1024 input, 128 output. So my training data requirement will have a huge impact on memory footprint.
3. the training should proceed with reasonable speed, e.g. < 5s/iteration.
4. Model checkpointing and reloading for continued fine-tuning is supported.
5. Timeline: 2 weeks.
Let me know if you have any questions.