Reading and Writing Parquet file with nested datatype using Pyspark
Budget: $2 – $8 USD
Please find the images attached
Read the parquet file line by line , column by column, each and every column value will be passed to another function will return some value, with that new value the string has to be replaced in the current column value and and write the records ( with changed values) to new parquet file......while writing we have to make sure that order of the records, schema structure everything should be same ( apart from changed values)
For ex: in the Sample Data.jpeg ,we see properties.eye for all old names James, Michael,Robert , Washington...
for old_name --> James , create a function by name transformer() and if we pass properties.eye ---> brown should replace with black,
for old_name --> Michael if we pass properties.eye ---> null should replace with black
the changes should be appear in the new new parquet file by name target.parquet with same schema structure , order of columns,order of records
Note:- sample data is just for input data, logic should be dynamic , parquet file schema will not be the same all the time.....our code should read the parquet file schema dynamically and and create the parquet file with changed data ( xxx) ....the rows, schema and columns should be same
Code Snippet for sample data
dataDictionary = [
('James',{'hair':'black','eye':'brown'}, ("James","","Smith")),
('Michael',{'hair':'brown','eye': None}, ("Michael","Rose","")),
('Robert',{'hair':'red','eye':'black'}, ("Robert","","Williams")),
('Washington',{'hair':'grey','eye':'grey'}, ("Maria","Anne","Jones"))
]
schema = StructType([
StructField('old_name', StringType(), True),
StructField('properties', MapType(StringType(),StringType()),True),
StructField('name', StructType([
StructField('firstname', StringType(), True),
StructField('middlename', StringType(), True),
StructField('lastname', StringType(), True)
]))
])
Sample data screen shot has the sample data
Sample schema screen shot has the schema details
Read the parquet file line by line , column by column, each and every column value will be passed to another function will return some value, with that new value the string has to be replaced in the current column value and and write the records ( with changed values) to new parquet file......while writing we have to make sure that order of the records, schema structure everything should be same ( apart from changed values)
For ex: in the Sample Data.jpeg ,we see properties.eye for all old names James, Michael,Robert , Washington...
for old_name --> James , create a function by name transformer() and if we pass properties.eye ---> brown should replace with black,
for old_name --> Michael if we pass properties.eye ---> null should replace with black
the changes should be appear in the new new parquet file by name target.parquet with same schema structure , order of columns,order of records
Note:- sample data is just for input data, logic should be dynamic , parquet file schema will not be the same all the time.....our code should read the parquet file schema dynamically and and create the parquet file with changed data ( xxx) ....the rows, schema and columns should be same
Code Snippet for sample data
dataDictionary = [
('James',{'hair':'black','eye':'brown'}, ("James","","Smith")),
('Michael',{'hair':'brown','eye': None}, ("Michael","Rose","")),
('Robert',{'hair':'red','eye':'black'}, ("Robert","","Williams")),
('Washington',{'hair':'grey','eye':'grey'}, ("Maria","Anne","Jones"))
]
schema = StructType([
StructField('old_name', StringType(), True),
StructField('properties', MapType(StringType(),StringType()),True),
StructField('name', StructType([
StructField('firstname', StringType(), True),
StructField('middlename', StringType(), True),
StructField('lastname', StringType(), True)
]))
])
Sample data screen shot has the sample data
Sample schema screen shot has the schema details
Related categories:
PySpark