I want to connect with a Pyspark developer who can help me in writing pyspark code to ingest XML files dynamically to Hive tables
Budget: $250 – $750 SGD
Autocreation of hive tables:
get the attribute name,data type,size from xsd
mapping data type to hive compatibility data type
specify table partition column
create table structure script
connecting to hive and exeute the script
Create a XML parser using pyspark to
Parsing Manifest file:
Write a script to store data in table
Combining Manifest file with Master data:
generate metadata list of parameter reuire to run the loading
XML Validation:
Create a script to read XML/XSD from metadata table
Import XML packages
XML filename,formatutf-8,filesize validation
XML against XSD validation
XML validation Fail and sucess scenario
Write a wrapper shell script for job invocation
XML to table:
Read xml file and load to hive table
get the attribute name,data type,size from xsd
mapping data type to hive compatibility data type
specify table partition column
create table structure script
connecting to hive and exeute the script
Create a XML parser using pyspark to
Parsing Manifest file:
Write a script to store data in table
Combining Manifest file with Master data:
generate metadata list of parameter reuire to run the loading
XML Validation:
Create a script to read XML/XSD from metadata table
Import XML packages
XML filename,formatutf-8,filesize validation
XML against XSD validation
XML validation Fail and sucess scenario
Write a wrapper shell script for job invocation
XML to table:
Read xml file and load to hive table