Unstructured Data Transformation Script
Budget: $10 – $30 AUD
I'm looking for a Data Engineer who can write a script that transforms unstructured data from .txt and .xls files into structured MySQL database entries.
Key Requirements:
- The script should be able to handle both .txt, .csv and .xls files.
- The unstructured data it processes will be uploaded by the user through the site.
- The transformed data should be suitable for entry into a MySQL database autonomously
- The script should perform data validation to ensure that only clean and accurate data are transformed.
- Provide detailed mappings of how unstructured data is to be transformed into structured data.
- Implement error handling to manage issues like file corruption or incorrect formats.
- Include functionality to schedule the script to run automatically at specified intervals.
- Add logging capabilities to track the script’s operations and identify any transformation issues.
- Provide configuration options to customize data transformation rules as needed.
- Set a maximum file size limit that the script can handle to avoid performance issues.
- Create a simple user interface for uploading files and tracking the transformation process.
- Ensure the script can handle increasing amounts of data without significant performance loss.
Ideal Skills:
- Proficiency in scripting languages such as Python or R.
- Strong experience with MySQL.
- Prior experience in writing data transformation scripts.
The script should perform simple parsing of data. The data validation process should be fully automated without any manual intervention required. The user interface should be simple and minimal for easy file uploads and process tracking.
Use only basic validation processes to clean and transform the data. Log basic operations like start and end times, and status of each file processed. The script should have an option to be run manually by the user on demand. Use the Pandas library for data transformation tasks. when user input is required
Key Requirements:
- The script should be able to handle both .txt, .csv and .xls files.
- The unstructured data it processes will be uploaded by the user through the site.
- The transformed data should be suitable for entry into a MySQL database autonomously
- The script should perform data validation to ensure that only clean and accurate data are transformed.
- Provide detailed mappings of how unstructured data is to be transformed into structured data.
- Implement error handling to manage issues like file corruption or incorrect formats.
- Include functionality to schedule the script to run automatically at specified intervals.
- Add logging capabilities to track the script’s operations and identify any transformation issues.
- Provide configuration options to customize data transformation rules as needed.
- Set a maximum file size limit that the script can handle to avoid performance issues.
- Create a simple user interface for uploading files and tracking the transformation process.
- Ensure the script can handle increasing amounts of data without significant performance loss.
Ideal Skills:
- Proficiency in scripting languages such as Python or R.
- Strong experience with MySQL.
- Prior experience in writing data transformation scripts.
The script should perform simple parsing of data. The data validation process should be fully automated without any manual intervention required. The user interface should be simple and minimal for easy file uploads and process tracking.
Use only basic validation processes to clean and transform the data. Log basic operations like start and end times, and status of each file processed. The script should have an option to be run manually by the user on demand. Use the Pandas library for data transformation tasks. when user input is required