ETL Specialist for Databricks on AWS
Budget: $50 – $75 USD
I need a Databricks developer with expertise in ETL (Extract, Transform, Load) processes. The project primarily revolves around manipulating data sourced from cloud storage, specifically Amazon S3.
We are using Databricks (NON-community edition) cluster with 8 workers (8 GB memory, 2 cores each)
The pyspark notebook pulls data out from a Postgres database and tries to write to S3 using partitions.
Problems we are facing
- The write job (writing dataframe to S3) gets stuck at the last task
- I am writing to S3 using 24 partitions of data. So I can see 24 tasks in Spark UI.
- I can see that 22 of these tasks finish in less than 2 secs but the last two tasks take forever
- Eventually, the whole job times out
- However, we are able to see the data in S3. Why? Is this complete data?
What I am looking for
- To get the data into S3 by doing appropriate spark configurations or other changes to code
- Providing some guidance on the cluster configuration
We are using Databricks (NON-community edition) cluster with 8 workers (8 GB memory, 2 cores each)
The pyspark notebook pulls data out from a Postgres database and tries to write to S3 using partitions.
Problems we are facing
- The write job (writing dataframe to S3) gets stuck at the last task
- I am writing to S3 using 24 partitions of data. So I can see 24 tasks in Spark UI.
- I can see that 22 of these tasks finish in less than 2 secs but the last two tasks take forever
- Eventually, the whole job times out
- However, we are able to see the data in S3. Why? Is this complete data?
What I am looking for
- To get the data into S3 by doing appropriate spark configurations or other changes to code
- Providing some guidance on the cluster configuration