Convert SAS Data Pipeline to PySpark

I have an existing SAS program that handles end-to-end data processing for a single SQL Database source. The code cleans raw tables, applies a series of transformations, then produces several aggregated outputs that feed downstream reports. I now need the entire workflow re-implemented in PySpark running on Azure Databricks so I can retire the SAS environment and take advantage of Databricks’ scalability.

You will receive:
• The original .sas files with inline comments that explain each step
• A data-dictionary of the SQL tables involved
• Sample input/output datasets to verify parity

What I’m expecting from you:
1. A well-structured Databricks notebook (or .py files) that reproduces the SAS logic for data cleaning, transformation, and aggregation.
2. Connection code that pulls directly from the same SQL Database and writes the final tables back, using best practices for credentials and Spark JDBC.
3. Clear in-line comments so my team can maintain the pipeline going forward.
4. A short hand-off session where we review the notebook, run through the full load, and confirm the outputs match the SAS results record-for-record.

Acceptance criteria
– All row counts and aggregation values match the SAS outputs (±0 where rounding is not involved).
– Runtime in Databricks is comparable or faster than the current SAS batch.
– Code follows PySpark and Databricks style conventions and can be scheduled as a job without manual intervention.

If you have deep experience translating SAS data pipelines into PySpark and are comfortable working inside Azure Databricks, I’d love to work with you on this migration.