A webscraper for websites -- 3
Budget: $250 – $750 CAD
I am looking for someone to write basic web scrapper for few particular websites. It should be python application.
Goal
Create current state (snapshot) of fragrance section of the given websites.
Here are those websites I want to
https://www.sephora.com/
https://shop.shoppersdrugmart.ca/ (Perfumes category of products)
Fields needed:
keywords or tags, images, title, quantity and price (all quantities and their prices), description
Requirements:
Website scrapper should have a way to specify keywords as optional parameter. For example, if I search for Dior; it should scrap all products tagged with this keyword on the website. On Sephora, it should scrap all products (all pages) from following link:
https://www.sephora.com/ca/en/brand/dior/fragrance
If the keyword parameter is missing, it should scrap all the products under fragrance category.
We should be able to schedule crawling.
When a crawling session starts, it should create a separate folder under given bucket with timestamp in the directory name. This is to make sure that if we have two instances of crawler running, they should not produce inconsistent results.
During a single crawl, it should not have duplicated entries provided there are no duplicates on the original website.
Output format: S3 supported.
Those are high-level requirements from my side but don’t hesitate to share your suggestions if the solution can be implemented in better way.
Ideal skills and experience:
- Proficient in web scraping techniques and tools
- Familiarity with scraping specific websites
- Ability to scrape both text and images
- Experience with storing data in the S3 format
Note: The above links are from Canada and if you are in different region, you might need VPN. For testing your code, we can only provide non-sudo ssh access to a machine with Canadian IP.
Goal
Create current state (snapshot) of fragrance section of the given websites.
Here are those websites I want to
https://www.sephora.com/
https://shop.shoppersdrugmart.ca/ (Perfumes category of products)
Fields needed:
keywords or tags, images, title, quantity and price (all quantities and their prices), description
Requirements:
Website scrapper should have a way to specify keywords as optional parameter. For example, if I search for Dior; it should scrap all products tagged with this keyword on the website. On Sephora, it should scrap all products (all pages) from following link:
https://www.sephora.com/ca/en/brand/dior/fragrance
If the keyword parameter is missing, it should scrap all the products under fragrance category.
We should be able to schedule crawling.
When a crawling session starts, it should create a separate folder under given bucket with timestamp in the directory name. This is to make sure that if we have two instances of crawler running, they should not produce inconsistent results.
During a single crawl, it should not have duplicated entries provided there are no duplicates on the original website.
Output format: S3 supported.
Those are high-level requirements from my side but don’t hesitate to share your suggestions if the solution can be implemented in better way.
Ideal skills and experience:
- Proficient in web scraping techniques and tools
- Familiarity with scraping specific websites
- Ability to scrape both text and images
- Experience with storing data in the S3 format
Note: The above links are from Canada and if you are in different region, you might need VPN. For testing your code, we can only provide non-sudo ssh access to a machine with Canadian IP.