Pyspark Data Cleaning and Sorting
Budget: £20 – £250 GBP
This vital task entails cleaning and sorting two CSV files of approximately 100,000 rows and second one of about 1.5million rows using pyspark (Python) in Jupyter Notebook(s). The project consists of several key tasks:
Read in both datasets and then:
- Standardizing data to ensure consistency
- Removal of duplicate entries
- Filtering columns we need
- Handling and filling missing values
- Aggregating data on certain groupings as output
Important requirement:
I also need unit tests to be written for the code at the end.
Ideal Skills:
Candidates applying for this project should be adept with Pyspark in Python and have experience in data cleaning and manipulation. Experience with working on datasets of similar size would also be preferable. Attention to detail in ensuring the accuracy of the cleaned data will be of paramount importance.
Price: This is a simple task so please bid responsibly. It is also time sensitive so I believe it should take no more than 1 day. I will choose based on price/review and if interest please message me so I can share with you details.
Read in both datasets and then:
- Standardizing data to ensure consistency
- Removal of duplicate entries
- Filtering columns we need
- Handling and filling missing values
- Aggregating data on certain groupings as output
Important requirement:
I also need unit tests to be written for the code at the end.
Ideal Skills:
Candidates applying for this project should be adept with Pyspark in Python and have experience in data cleaning and manipulation. Experience with working on datasets of similar size would also be preferable. Attention to detail in ensuring the accuracy of the cleaned data will be of paramount importance.
Price: This is a simple task so please bid responsibly. It is also time sensitive so I believe it should take no more than 1 day. I will choose based on price/review and if interest please message me so I can share with you details.