Data Engineering
Budget: ₹600 – ₹1,500 INR
Refer to the problem statement in this github repo and come out with a Git lib with python script, docker envrionment and explanation for the problem statement. https://github.com/findmypast/recruitment-test-data-engineering
Problem
There are a sequence of steps that we would like you to complete. We hope this won't take more than a couple of hours of your time.
Fork the git repo to your own Github account.
Devise a database schema to hold the data in the people and places CSV files, and apply it to the MySQL database. You may apply this schema via a script, via the MySQL command-line client, or via a GUI client.
Create a Docker image for loading the CSV files, places.csv and people.csv, into the tables you have created in the database. Make sure the appropriate config is in the docker compose file. Your data ingest process can be implemented in any way that you like, as long as it runs within a Docker container. You may implement this via programme code in a language of your choice, or via the use of ETL tools.
Create a Docker image for outputting a summary of content in the database. You may implement this using a programming language of your choice. The output must be in JSON format, and be written to a file in the data folder called data/summary_output.json. It should consist of a list of the countries, and a count of how many people were born in that country. We have supplied a sample output data/sample_output.json to compare your file against.
Share a link to your cloned github repo with us so we can review your code ahead of your interview.
We have provided an example schema and code that shows how to handle a simple data ingest and output.
Details of how to run and connect to the database are below, together with how to use the example schema and code.
Notes on completing these tasks
There is no right way to do this. We are interested in the choices that you make, how you justify them, and your development process.
Consider how normalized your schema should be, and whether or not you should be using foreign keys to join tables.
When you create a container, make sure that you add the container config to the docker compose.yml file, and add your Dockerfile and code to the images folder.
Make sure that your code is executable, and if you are working in a scripting language, make sure that your script has an appropriate “hash-bang” line (as featured in our example scripts).
Most of the example code uses ORM libraries to connect to the database. This is not essential for the purpose of this test: your code should connect to the database and your queries should be implemented in whatever way you are most comfortable with.
Consider what kind of error handling and testing is appropriate.
All data input, storage, and output should be in UTF-8. Expect multi-byte characters in the data.
The MySQL database storage is ephemeral; it will not persist, so make sure all schema and data queries are repeatable.
You may find it easier to work with a subset of the data when developing your ingest
Problem
There are a sequence of steps that we would like you to complete. We hope this won't take more than a couple of hours of your time.
Fork the git repo to your own Github account.
Devise a database schema to hold the data in the people and places CSV files, and apply it to the MySQL database. You may apply this schema via a script, via the MySQL command-line client, or via a GUI client.
Create a Docker image for loading the CSV files, places.csv and people.csv, into the tables you have created in the database. Make sure the appropriate config is in the docker compose file. Your data ingest process can be implemented in any way that you like, as long as it runs within a Docker container. You may implement this via programme code in a language of your choice, or via the use of ETL tools.
Create a Docker image for outputting a summary of content in the database. You may implement this using a programming language of your choice. The output must be in JSON format, and be written to a file in the data folder called data/summary_output.json. It should consist of a list of the countries, and a count of how many people were born in that country. We have supplied a sample output data/sample_output.json to compare your file against.
Share a link to your cloned github repo with us so we can review your code ahead of your interview.
We have provided an example schema and code that shows how to handle a simple data ingest and output.
Details of how to run and connect to the database are below, together with how to use the example schema and code.
Notes on completing these tasks
There is no right way to do this. We are interested in the choices that you make, how you justify them, and your development process.
Consider how normalized your schema should be, and whether or not you should be using foreign keys to join tables.
When you create a container, make sure that you add the container config to the docker compose.yml file, and add your Dockerfile and code to the images folder.
Make sure that your code is executable, and if you are working in a scripting language, make sure that your script has an appropriate “hash-bang” line (as featured in our example scripts).
Most of the example code uses ORM libraries to connect to the database. This is not essential for the purpose of this test: your code should connect to the database and your queries should be implemented in whatever way you are most comfortable with.
Consider what kind of error handling and testing is appropriate.
All data input, storage, and output should be in UTF-8. Expect multi-byte characters in the data.
The MySQL database storage is ephemeral; it will not persist, so make sure all schema and data queries are repeatable.
You may find it easier to work with a subset of the data when developing your ingest