Python script to scrape data from websites and populate SQL database
Budget: $15 – $25 CAD
Details
- Scrape details from a site based on xpath or css selector
- URLs for scraping will be provided in a separate list (list population will be automated using a separate service in the future)
- Design functionality to be able to add new domains for scraping
-- Each new domain can have different xpath for each data point that is being scraped
-- Provide ideas on how to better structure the app so that future onboarding of new domains is easy
- Scraping should be throttled to ensure that server is not blocking the requests
- Ensure that the script can run both locally and on a server environment
- Provide functionality to change user-agent and have capacity to use proxy servers
- Have a command line menu to change script settings (ex. change user-agent, start-stop service, change throttling services and print the updates in the console as the script runs).
- Identify and extract certain images on the scraped url and save them in a certain filenaming convention.
- Scraped data will be saved to SQL database in the future. SQL database is being designed separately (details will be provided soon).
- Create conditions to ensure that scraped values are all saved in a similar format (ex. certain fields are all uppercase, price is saved in a proper format etc)
- While script testing scraped data can be saved as CSV file
- Take full size screenshots of the scraped url and save it in a certain filename with variables.
- Send email notifications for errors or for successful script completion
Candidate should also be experienced in deploying the script/app to server environment (ex. heroku)
More details will be discussed during the project implementation.
- Scrape details from a site based on xpath or css selector
- URLs for scraping will be provided in a separate list (list population will be automated using a separate service in the future)
- Design functionality to be able to add new domains for scraping
-- Each new domain can have different xpath for each data point that is being scraped
-- Provide ideas on how to better structure the app so that future onboarding of new domains is easy
- Scraping should be throttled to ensure that server is not blocking the requests
- Ensure that the script can run both locally and on a server environment
- Provide functionality to change user-agent and have capacity to use proxy servers
- Have a command line menu to change script settings (ex. change user-agent, start-stop service, change throttling services and print the updates in the console as the script runs).
- Identify and extract certain images on the scraped url and save them in a certain filenaming convention.
- Scraped data will be saved to SQL database in the future. SQL database is being designed separately (details will be provided soon).
- Create conditions to ensure that scraped values are all saved in a similar format (ex. certain fields are all uppercase, price is saved in a proper format etc)
- While script testing scraped data can be saved as CSV file
- Take full size screenshots of the scraped url and save it in a certain filename with variables.
- Send email notifications for errors or for successful script completion
Candidate should also be experienced in deploying the script/app to server environment (ex. heroku)
More details will be discussed during the project implementation.