Data Ingestion from S3 to RDS with Fallback to AWS Glue using Dockerized Python Application

Title: Data Ingestion from S3 to RDS with Fallback to AWS Glue using Dockerized
Python Application
Objective:
Develop a Dockerized Python application that automates the process of:
 Reading data from an Amazon S3 bucket
 Pushing it to an RDS (MySQL-compatible) database
 Automatically falling back to AWS Glue if the RDS database is unavailable or
the push operation fails
This project helps you integrate multiple AWS services (S3, RDS, Glue), work with
data pipelines, and use Docker for packaging and deployment.
Project Tasks:
1. Python Script Development:
 Create a Python script that:
o Reads a CSV file from an S3 bucket
o Parses it using pandas
o Tries to upload the data to an RDS MySQL-compatible database
o If the RDS upload fails, falls back to:
 Creating a table in AWS Glue Data Catalog
 Registering the dataset location in S3
2. Dockerfile Creation:
 Build a Dockerfile that:
o Uses a Python 3.9+ base image
o Installs all necessary libraries (boto3, pandas, sqlalchemy, pymysql)
o Copies the Python script into the container
o Runs the script on container startup
3. Requirements File:
 Create a requirements.txt with all necessary Python dependencies:
4. Image Build and Container Run:
 Build the Docker image:
 Run the container using environment variables for AWS credentials:
Configuration Parameters (Replace with actual values):
 S3 Bucket Name
 CSV File Key
 RDS DB Endpoint, Username, Password, DB Name, Table Name
 Glue Database Name, Table Name, and S3 Location
Deliverables:
1. Python script and Dockerfile stored in a GitHub repository
2. Working Docker image and container logs showing:
o Successful push to RDS or fallback to Glue
3. Screenshot of:
o Records inserted into RDS or
o Table created in AWS Glue Catalog

4. Summary Report (1–2 pages) explaining:
o Data flow (S3 → RDS → Glue fallback)
o AWS services used
o Docker setup
o Challenges faced and how they were solved