Create a AWS EMR serverless template, having pyspark and all other python datascience dependencies -- 2
Budget: ₹830 – ₹1,170 INR
Hi, I am looking for someone who knows how to create a Serverless template to launch AWS EMR Cluster with Pyspark and Datascience dependencies and also deploy a Lambda functions. Basically need to know how bootstrap initialization works. When I run the command example - serverless deploy then the template should install the EMR Serverless Cluster that has Pyspark 3.2.0 and Python 3.9, Pandas and other datascience dependencies installed.
Currently I am doing it via AWS CLI and could able to trigger a lambda function to submit spark jobs when a file arives in the AWS S3.
My objectives: 1) I am accessing data from a S3 bucket.
2) when a file comes in the bucket it should fire the lambda and submit the job. Spark job preprocess the data and it should stored the preprocessed data in the Target S3 location.
I could able to do all these, however, I could not able to install the python dependencies so my job is failing.
SKILLS Needed:- Serverless Frameworks, AWS EMR Cluster, Python, Pyspark, must know how to install dependencies while creating the servers via serverless template or AWS SAM Template.
If you have done this job previously then only bid.
Included three screenshot - diagram.png shows my objective, emr_cluster_1 and 2 shows I got bootstrap error for your reference.
Currently I am doing it via AWS CLI and could able to trigger a lambda function to submit spark jobs when a file arives in the AWS S3.
My objectives: 1) I am accessing data from a S3 bucket.
2) when a file comes in the bucket it should fire the lambda and submit the job. Spark job preprocess the data and it should stored the preprocessed data in the Target S3 location.
I could able to do all these, however, I could not able to install the python dependencies so my job is failing.
SKILLS Needed:- Serverless Frameworks, AWS EMR Cluster, Python, Pyspark, must know how to install dependencies while creating the servers via serverless template or AWS SAM Template.
If you have done this job previously then only bid.
Included three screenshot - diagram.png shows my objective, emr_cluster_1 and 2 shows I got bootstrap error for your reference.