AWS Glue ETL Pipeline
Budget: ₹1,500 – ₹12,500 INR
I have a collection of CSV files already sitting in S3 and I want a Glue-based pipeline that processes them end-to-end. The job breaks down into three clear steps:
1. Use an AWS Glue crawler to catalogue every incoming CSV in my bucket and keep the schema current whenever new files arrive.
2. Load the catalogued data into my existing AWS database (I’ll share the engine, endpoint, and credentials once we start).
3. Run the necessary transformations—filtering, cleansing, re-mapping columns, or any other tweaks we define together—and write the final result back to S3 as fresh CSV files.
I’m after a lean, reusable solution that can be kicked off on demand or by trigger, logs its run in CloudWatch, and finishes quickly. Please build the Glue job in PySpark or Glue Studio, include any IAM roles or policies the pipeline needs, and hand over the script, a brief setup guide, and a short screen share or notes that prove everything works.
Acceptance criteria
• Crawler auto-discovers new files and updates the Data Catalog
• Transformed data lands in the target database with the correct schema
• New CSVs appear in the specified S3 prefix with the expected structure
• All steps succeed without manual intervention and are visible in CloudWatch
If you’ve built similar Glue ETL flows before, this should be straightforward and quick. Let’s move fast—looking forward to your expertise.
1. Use an AWS Glue crawler to catalogue every incoming CSV in my bucket and keep the schema current whenever new files arrive.
2. Load the catalogued data into my existing AWS database (I’ll share the engine, endpoint, and credentials once we start).
3. Run the necessary transformations—filtering, cleansing, re-mapping columns, or any other tweaks we define together—and write the final result back to S3 as fresh CSV files.
I’m after a lean, reusable solution that can be kicked off on demand or by trigger, logs its run in CloudWatch, and finishes quickly. Please build the Glue job in PySpark or Glue Studio, include any IAM roles or policies the pipeline needs, and hand over the script, a brief setup guide, and a short screen share or notes that prove everything works.
Acceptance criteria
• Crawler auto-discovers new files and updates the Data Catalog
• Transformed data lands in the target database with the correct schema
• New CSVs appear in the specified S3 prefix with the expected structure
• All steps succeed without manual intervention and are visible in CloudWatch
If you’ve built similar Glue ETL flows before, this should be straightforward and quick. Let’s move fast—looking forward to your expertise.