SQL View to PySpark Notebook

Project Description

I'm looking for an experienced Azure Databricks / PySpark developer to help convert an existing SQL Server view into a Databricks PySpark transformation.

Project Overview

I have:

An existing SQL view with multiple CTEs.
An existing Databricks Asset Bundle notebook that will be used as the template.
A source Delta table already available in Unity Catalog.

The task is to replace the existing business transformation with a new PySpark implementation while preserving the existing notebook framework.

Source

The source is a Delta table in Unity Catalog.

The notebook should:

Read data from the source table.
Apply filters.
Replicate the SQL CTE logic in PySpark.
Perform required aggregations and calculations.
Produce the final dataframe.
Preserve the notebook's existing audit and validation framework.
Write the output Delta table.
Create a SQL View on top of the Delta table.
SQL Logic

The SQL contains:

Multiple CTEs
Window selection logic
Aggregations
CASE expressions
GROUP BY
Date filtering
Calculated measures

The objective is to produce the same output as the SQL view using PySpark DataFrame APIs.

Existing Notebook

The notebook already contains:

Logging framework
Metadata handling
Schema validation
Audit column generation
Exception handling
Delta write logic

These should remain unchanged.

Only the business transformation needs to be replaced.

Deliverables
Complete PySpark implementation.
Clean, optimized, production-ready code.
Logic matching the SQL view.
Compatible with Databricks Runtime.
Delta table creation.
SQL View creation.
Assistance with testing if required.
Required Skills
Azure Databricks
PySpark
Spark SQL
Delta Lake
SQL Server
DataFrame API
Window Functions
Databricks Asset Bundles (preferred)
Nice to Have
Experience with Unity Catalog
Production ETL development
Azure Data Factory knowledge
What I'll Provide
Existing SQL View
Existing Databricks notebook
Business logic