Create a AWS EMR serverless template, having pyspark and all other python datascience dependencies
Budget: ₹830 – ₹1,170 INR
Hi, I am looking for someone who knows how to create a Serverless template to launch AWS EMR Cluster with Pyspark and Datascience dependencies. Basically need to know how bootstrap initialization works. When I run the command example - serverless deploy then the template should install the EMR Serverless Cluster that has Pyspark 3.2.0 and Python 3.9, Pandas and other datascience dependencies installed.
Currently I am doing it via AWS CLI and could able to trigger a lambda function to submit spark jobs when a file arives in the AWS S3.
My objectives: 1) I am accessing data from a S3 bucket.
2) when a file comes in the bucket it should fire the lambda and submit the job. Spark job preprocess the data and it should stored the preprocessed data into the Target S3 location.
I could able to do all these, however I could not able to install the python dependencies so my job is failing.
Currently I am doing it via AWS CLI and could able to trigger a lambda function to submit spark jobs when a file arives in the AWS S3.
My objectives: 1) I am accessing data from a S3 bucket.
2) when a file comes in the bucket it should fire the lambda and submit the job. Spark job preprocess the data and it should stored the preprocessed data into the Target S3 location.
I could able to do all these, however I could not able to install the python dependencies so my job is failing.