PySpark Code Unit Testing for Quality
Budget: ₹1,500 – ₹12,500 INR
I'm in need of a professional with extensive PySpark unit testing experience. I have PySpark code that loads data from Oracle ODS to ECS (S3 bucket). The goal is to write unit test cases that will achieve at least 80% coverage in SonarQube quality.
You will focus primarily on testing for:
- Data validation and integrity
- Error handling and exceptions
The ideal candidate should:
- Be proficient in using PyTest, as this is our preferred testing framework
- Have a comprehensive understanding of PySpark
- Be able to deliver immediately
Please note, the main focus of this project is not on the data transformations that the PySpark code performs (which includes data cleaning and filtering, data aggregation and summarization, as well as data joining and merging), but rather on ensuring the code's quality and reliability through rigorous testing.
You will focus primarily on testing for:
- Data validation and integrity
- Error handling and exceptions
The ideal candidate should:
- Be proficient in using PyTest, as this is our preferred testing framework
- Have a comprehensive understanding of PySpark
- Be able to deliver immediately
Please note, the main focus of this project is not on the data transformations that the PySpark code performs (which includes data cleaning and filtering, data aggregation and summarization, as well as data joining and merging), but rather on ensuring the code's quality and reliability through rigorous testing.