Pyspark unit test cases
Budget: ₹1,500 – ₹12,500 INR
There is a simple project on Python testing.
Pyspark unit test cases need to be developed.
Roughly 10 functions and likely to take a few hours
for total work for experienced developer.
Any IDE like intellij or Pycharm will do.
Unit test library name is "Unittest" , using "mock".
Details of what the function does, variable details, and expected results.
Functions will involve pyspark dataframes and udfs (user defined functions).
Input1 (variable1) is json text data with multiple nested key value pairs.
Input2 is a list of keys from a json file (variable2) ,
output will be these attributes in json format (variable3).
Some functions will involve returning results.
Some are only to call other functions without a return value.
Set of python pyspark functions involving dataframes, uds,
general pyspark operations such as join, select, python lambdas,
reading and writing files from and to S3,
handling different spark data types like integers, arrays, strings
and different data formats such as csv, json.
Pyspark unit test cases need to be developed.
Roughly 10 functions and likely to take a few hours
for total work for experienced developer.
Any IDE like intellij or Pycharm will do.
Unit test library name is "Unittest" , using "mock".
Details of what the function does, variable details, and expected results.
Functions will involve pyspark dataframes and udfs (user defined functions).
Input1 (variable1) is json text data with multiple nested key value pairs.
Input2 is a list of keys from a json file (variable2) ,
output will be these attributes in json format (variable3).
Some functions will involve returning results.
Some are only to call other functions without a return value.
Set of python pyspark functions involving dataframes, uds,
general pyspark operations such as join, select, python lambdas,
reading and writing files from and to S3,
handling different spark data types like integers, arrays, strings
and different data formats such as csv, json.