Cloud Infrastructure And Data Pipeline Review for Optimization
Budget: $250 – $750 USD
Project Overview:
We are looking for an experienced Data Engineering Consultant / Cloud Architect to review our current infrastructure on both Azure and AWS. The goal is to identify weaknesses, bottlenecks, and risks, and then provide actionable recommendations to improve performance, reliability, scalability, and cost-efficiency of our data pipelines.
Scope of Work:
Infrastructure Assessment
Review our AWS and Azure cloud environments, including storage, networking, compute, and security configurations.
Evaluate existing data pipeline architecture, including ETL/ELT workflows, orchestration tools, and integrations with other systems.
Identify areas of risk such as misconfigurations, single points of failure, and data security concerns.
Data Pipeline Review
Analyze the current ingestion, transformation, and loading processes.
Identify performance bottlenecks and data quality issues.
Review orchestration tools (e.g., Azure Data Factory, AWS Glue, Step Functions, Airflow, etc.) and suggest optimizations.
Optimization & Recommendations
Provide detailed documentation outlining issues found and prioritized recommendations.
Suggest improvements for:
Scalability and elasticity
Cost optimization
Security and compliance (IAM roles, encryption, access control)
Monitoring and observability
Recommend tools and services to improve reliability and efficiency.
Optional – Implementation Support (if agreed upon)
Assist our team with implementing the approved recommendations and best practices.
Provide guidance on future-proofing the architecture.
Required Skills & Experience:
Strong experience with AWS (S3, Glue, Redshift, Lambda, IAM, etc.) and Azure (Data Factory, Synapse, Data Lake, Functions, etc.).
Deep understanding of data engineering and cloud-native architectures.
Knowledge of ETL/ELT best practices, data pipeline orchestration, and modern data stack tools.
Experience in security and cost optimization across multi-cloud environments.
Excellent documentation and communication skills.
Deliverables:
A comprehensive report detailing:
Current state analysis of AWS and Azure environments
List of identified issues and risks
Suggested improvements with priority levels
A recommended high-level architecture for scaling and securing the pipelines
We are looking for an experienced Data Engineering Consultant / Cloud Architect to review our current infrastructure on both Azure and AWS. The goal is to identify weaknesses, bottlenecks, and risks, and then provide actionable recommendations to improve performance, reliability, scalability, and cost-efficiency of our data pipelines.
Scope of Work:
Infrastructure Assessment
Review our AWS and Azure cloud environments, including storage, networking, compute, and security configurations.
Evaluate existing data pipeline architecture, including ETL/ELT workflows, orchestration tools, and integrations with other systems.
Identify areas of risk such as misconfigurations, single points of failure, and data security concerns.
Data Pipeline Review
Analyze the current ingestion, transformation, and loading processes.
Identify performance bottlenecks and data quality issues.
Review orchestration tools (e.g., Azure Data Factory, AWS Glue, Step Functions, Airflow, etc.) and suggest optimizations.
Optimization & Recommendations
Provide detailed documentation outlining issues found and prioritized recommendations.
Suggest improvements for:
Scalability and elasticity
Cost optimization
Security and compliance (IAM roles, encryption, access control)
Monitoring and observability
Recommend tools and services to improve reliability and efficiency.
Optional – Implementation Support (if agreed upon)
Assist our team with implementing the approved recommendations and best practices.
Provide guidance on future-proofing the architecture.
Required Skills & Experience:
Strong experience with AWS (S3, Glue, Redshift, Lambda, IAM, etc.) and Azure (Data Factory, Synapse, Data Lake, Functions, etc.).
Deep understanding of data engineering and cloud-native architectures.
Knowledge of ETL/ELT best practices, data pipeline orchestration, and modern data stack tools.
Experience in security and cost optimization across multi-cloud environments.
Excellent documentation and communication skills.
Deliverables:
A comprehensive report detailing:
Current state analysis of AWS and Azure environments
List of identified issues and risks
Suggested improvements with priority levels
A recommended high-level architecture for scaling and securing the pipelines