Multi-Cloud Databricks Deployment Mentor

I’m looking for an experienced data engineer who can guide me end-to-end as I build and launch a full data pipeline on Databricks across AWS, Azure, and Google Cloud.

What I need help with
• Designing and refining ingestion, processing, and storage layers that suit a multi-cloud strategy
• Standing up the necessary cloud infrastructure (IAM, networking, storage, compute, CI/CD pipelines) in each provider
• Tuning and troubleshooting Databricks clusters, jobs, and notebooks for cost-efficiency and speed

How we’ll work
We’ll meet in live sessions (screen share or pair-programming) where you walk me through architecture decisions, best practices, and hands-on tasks. Between sessions I’ll implement the work; you’ll review code, Terraform templates, notebooks, and deployment steps, then suggest improvements.

Deliverables
1. Structured session plan covering ingestion, processing, storage, infrastructure, and optimization topics
2. Architecture diagram and reference Terraform/equivalent scripts for AWS, Azure, and GCP
3. Sample Databricks notebooks and job configs illustrating best practices
4. Checklist I can follow for final production rollout
5. Follow-up support (chat/email) to resolve blockers until the pipeline is live

You should be fluent with Delta Lake, Spark, Databricks Workflows, and each cloud’s security and networking services. Practical mentoring experience is a plus—I’m aiming to learn as much as ship the solution.