Data Engineer Task
Budget: ₹1,500 – ₹12,500 INR
You
will need to get the most up to date information, scrape it, clean it and save the values in the database.
Then you will write a function that queries the database and checks for matches.
1. Set up a database model with the following attributes:
● id (unique value)
● meti_id - the id given by meti
● country_code - the iso 3166-1 alpha -2 format
(as in https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#LA)
● entity_name
● parent_id - in case of alias, the id of the original name
● entity_type (type of WMD)
● created_at - the timestamp of when this value is saved in the db
Make sure that each of the attributes has the correct type
2. In https://www.meti.go.jp/english/policy/external_economy/trade_control/index.html navigate via
python to the section of End User List and choose the latest update
3. In the new page, there is a pdf file with the sanction list, open and parse it
4. Scrape all values as follow:
● No. is the meti_id
● Country or Region should be converted to country code
● Company or organization is the entity_name. Convert it to lowercase and save the value. If a
value contains brackets, separate the value in the bracket and save it as an alias. In a similar
way, if an entity name is separated by a slash, save the first one as the original and the other
as an alias. If that happens to an alias, just save both as separate aliases
● If there are multiple brackets, save them as multiple aliases
● If there is a latin version and a non-latin version, save the non-latin version as an alias
● If the brackets appear in the middle of a name, there is no need to separate it
● Also Known As - All aliases should have the parent id of the main entity name
● For entity_type, save only the latin letters
5. Once all values are cleaned, save them in the db, this method would be used as a cron job on a daily
basis
6. Write a function that gets a name and check whether it exists in the db (1:1 match, regardless of
lower/uppercase). If yes, return all the attributes of this name from the db and if it has aliases, return
also a list with all the aliases (names only, no need for the other attributes of the aliases)
will need to get the most up to date information, scrape it, clean it and save the values in the database.
Then you will write a function that queries the database and checks for matches.
1. Set up a database model with the following attributes:
● id (unique value)
● meti_id - the id given by meti
● country_code - the iso 3166-1 alpha -2 format
(as in https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#LA)
● entity_name
● parent_id - in case of alias, the id of the original name
● entity_type (type of WMD)
● created_at - the timestamp of when this value is saved in the db
Make sure that each of the attributes has the correct type
2. In https://www.meti.go.jp/english/policy/external_economy/trade_control/index.html navigate via
python to the section of End User List and choose the latest update
3. In the new page, there is a pdf file with the sanction list, open and parse it
4. Scrape all values as follow:
● No. is the meti_id
● Country or Region should be converted to country code
● Company or organization is the entity_name. Convert it to lowercase and save the value. If a
value contains brackets, separate the value in the bracket and save it as an alias. In a similar
way, if an entity name is separated by a slash, save the first one as the original and the other
as an alias. If that happens to an alias, just save both as separate aliases
● If there are multiple brackets, save them as multiple aliases
● If there is a latin version and a non-latin version, save the non-latin version as an alias
● If the brackets appear in the middle of a name, there is no need to separate it
● Also Known As - All aliases should have the parent id of the main entity name
● For entity_type, save only the latin letters
5. Once all values are cleaned, save them in the db, this method would be used as a cron job on a daily
basis
6. Write a function that gets a name and check whether it exists in the db (1:1 match, regardless of
lower/uppercase). If yes, return all the attributes of this name from the db and if it has aliases, return
also a list with all the aliases (names only, no need for the other attributes of the aliases)