laravel 9 ETL job
Budget: €250 – €750 EUR
To interface with the "Heads of Medicines Agencies" portal and use the following API to extract data
https://mri.cts-mrp.eu/portal/v1/odata/ProductSearch?$filter=(UpdatedAt%20ge%202022-12-27T18%3A54%3A09%2B08%3A00)
%20and%20(UpdatedAt%20le%202023-01-27T18%3A54%3A09%2B08%3A00)%20and%20(domainKey%20eq%20%27h%27)&$expand=activeSubstances,
atcCodes,doseForms,rms&$count=true&$skip=10&$top=10
The inital run should start from 1980-01-01 to now, subsequent runs should start from the last run date.
The last run date should be stored in settings table.
Max query is 1000, so skip and top can be used to page through data, the json carries the number of records available in "@odata.count".
Data is somewhat dirty with records stored by approval id instead of name and dose, with it being common for names to be mixed with dose form as well, this needs to be cleaned so there is a single drug in each name and the dose forms are stored within the doseForm table. doseForms, unitDescription, and name - all need cleaning. Samples in json are attached.
There are two types of "domainkey" and it appears if domainkey is missing then it will default to "h", so two runs are needed, both "h" and "v".
Data needs to be then updated in the tables and any data existing the in the table but not in the HMA API needs to be marked deleted (it may be possible that there is a way to collect deleted products from the API, but I can't see it)
This needs to run as a laravel 9 job with a mechanism to send alerts/errors to a log.
Fixed price quotes only please and I prefer to pay in euros
https://mri.cts-mrp.eu/portal/v1/odata/ProductSearch?$filter=(UpdatedAt%20ge%202022-12-27T18%3A54%3A09%2B08%3A00)
%20and%20(UpdatedAt%20le%202023-01-27T18%3A54%3A09%2B08%3A00)%20and%20(domainKey%20eq%20%27h%27)&$expand=activeSubstances,
atcCodes,doseForms,rms&$count=true&$skip=10&$top=10
The inital run should start from 1980-01-01 to now, subsequent runs should start from the last run date.
The last run date should be stored in settings table.
Max query is 1000, so skip and top can be used to page through data, the json carries the number of records available in "@odata.count".
Data is somewhat dirty with records stored by approval id instead of name and dose, with it being common for names to be mixed with dose form as well, this needs to be cleaned so there is a single drug in each name and the dose forms are stored within the doseForm table. doseForms, unitDescription, and name - all need cleaning. Samples in json are attached.
There are two types of "domainkey" and it appears if domainkey is missing then it will default to "h", so two runs are needed, both "h" and "v".
Data needs to be then updated in the tables and any data existing the in the table but not in the HMA API needs to be marked deleted (it may be possible that there is a way to collect deleted products from the API, but I can't see it)
This needs to run as a laravel 9 job with a mechanism to send alerts/errors to a log.
Fixed price quotes only please and I prefer to pay in euros