Build and Finalise AWS Data Pipeline and Backend Architecture
Budget: $8 – $15 USD
We are seeking an experienced AWS backend developer to help us design, implement, and finalize a robust data pipeline that meets both functional and operational requirements. This pipeline will enable scheduled ingestion, transformation, and analysis of API data and provide a RESTful API to dynamically serve data to our website.
Objective
To create a cost-effective, highly available, and secure AWS-based backend architecture that ensures seamless data ingestion, transformation, and real-time interaction with our website.
Key Responsibilities
Data Ingestion
Schedule Python scripts using AWS Lambda to fetch data from APIs.
Handle API rate limits, errors, and retry mechanisms.
Store raw data temporarily in Amazon S3.
Data Parsing
Clean and structure raw data for further processing.
Ensure data validity and store parsed data in S3.
Data Transformation
Enrich and reshape parsed data for analysis or database upload.
Store intermediate results in S3 for staging.
Data Upload to SQL Database
Save transformed data into Amazon RDS (PostgreSQL/MySQL) with optimized database inserts.
Ensure data integrity and performance.
Data Analysis and SQL Transformations
Execute SQL procedures and queries in RDS for additional analysis and derived insights.
Store final results in the database for website access.
RESTful API Development
Build a secure RESTful API using AWS API Gateway and Lambda to provide real-time data to the website.
Optimize database queries to minimize latency and maximize efficiency.
Non-Functional Requirements
Availability: Ensure high availability and fault tolerance for all components, particularly the API.
Scalability: Design for future growth in data volume and API traffic.
Cost Optimization: Use serverless and managed services to reduce idle costs.
Security: Implement IAM roles and secure database credentials with AWS Secrets Manager.
Monitoring: Use CloudWatch for tracking failures and performance metrics, with alerting for critical issues.
Skills and Experience Required
Proven expertise with AWS services, including Lambda, S3, RDS, API Gateway, EventBridge, and Secrets Manager.
Strong proficiency in Python for scripting and data processing.
Experience in designing and implementing RESTful APIs.
Knowledge of SQL for database management and data analysis.
Familiarity with CloudWatch for monitoring and troubleshooting.
Ability to optimize backend workflows for performance, scalability, and cost-efficiency.
Project Details
Start Date: asap
Location: Remote
This project offers an exciting opportunity to build a foundational system critical to our operations. If you have the skills and experience to deliver this solution, we’d love to hear from you. Please submit your portfolio or examples of similar work with your application.
Objective
To create a cost-effective, highly available, and secure AWS-based backend architecture that ensures seamless data ingestion, transformation, and real-time interaction with our website.
Key Responsibilities
Data Ingestion
Schedule Python scripts using AWS Lambda to fetch data from APIs.
Handle API rate limits, errors, and retry mechanisms.
Store raw data temporarily in Amazon S3.
Data Parsing
Clean and structure raw data for further processing.
Ensure data validity and store parsed data in S3.
Data Transformation
Enrich and reshape parsed data for analysis or database upload.
Store intermediate results in S3 for staging.
Data Upload to SQL Database
Save transformed data into Amazon RDS (PostgreSQL/MySQL) with optimized database inserts.
Ensure data integrity and performance.
Data Analysis and SQL Transformations
Execute SQL procedures and queries in RDS for additional analysis and derived insights.
Store final results in the database for website access.
RESTful API Development
Build a secure RESTful API using AWS API Gateway and Lambda to provide real-time data to the website.
Optimize database queries to minimize latency and maximize efficiency.
Non-Functional Requirements
Availability: Ensure high availability and fault tolerance for all components, particularly the API.
Scalability: Design for future growth in data volume and API traffic.
Cost Optimization: Use serverless and managed services to reduce idle costs.
Security: Implement IAM roles and secure database credentials with AWS Secrets Manager.
Monitoring: Use CloudWatch for tracking failures and performance metrics, with alerting for critical issues.
Skills and Experience Required
Proven expertise with AWS services, including Lambda, S3, RDS, API Gateway, EventBridge, and Secrets Manager.
Strong proficiency in Python for scripting and data processing.
Experience in designing and implementing RESTful APIs.
Knowledge of SQL for database management and data analysis.
Familiarity with CloudWatch for monitoring and troubleshooting.
Ability to optimize backend workflows for performance, scalability, and cost-efficiency.
Project Details
Start Date: asap
Location: Remote
This project offers an exciting opportunity to build a foundational system critical to our operations. If you have the skills and experience to deliver this solution, we’d love to hear from you. Please submit your portfolio or examples of similar work with your application.