Python Web Scraping - Add features on existing working code °° Only for available and skilled Freelancer °°
Budget: €250 – €750 EUR
Introduction :
I have a market price comparison website . Users can compare price of targeted product from different shops (this is front-end part).
There is a scrapping back-end "engine" system for ensure valid data information provided to front-end website
This engine is coded in Python and retrieve all needed information (size-price, sku, product description…) into MongoDB database.
Back-end scrapping engine run 24/7, never crash, never stop (or website data will not up to date).
Works Steps :
1/ Backup actual scrapping infrastructure then migrate to new servers (from 2 olds windows servers to 4 new fresh windows servers, cause we need to speed up a lot scrapping engine process).
2/ Ensure scrapping engine works to retrieve needed data from all targeted shops as it actually works for both DB collection name
.
3/ Scrapping issue adjustments :
• Add logical / request to submit / force (send request to scrapping engine) : Dedicated Unique SKU CRON Task update, and also Batch (multiple) SKU CRON Task update
• « Last Price Value » must updated as soon as new product is added into MongoDB (not later when cron task will scrape this data), otherwise we need to wait a long time to get « Last Sale Price » value displayed on front-end page.
• Scrapped Size missmatch on some website :
« restock » website got weird size on some product like and also some wrong product scrapped / redirect link
• Add Calendar page (detailed into attached document) to product scrapping and create corresponding products pages. Then also display into website homepage, all next products to release. For this calendar frontend section we must only display : product image, product name, release date, last sale price. Also clicking redirect to product page.
4/ Add retail market to scrapping engine and create « retail » market collection into mongoDb
5/ Add some extra Shops website to scrapping engine
I have a market price comparison website . Users can compare price of targeted product from different shops (this is front-end part).
There is a scrapping back-end "engine" system for ensure valid data information provided to front-end website
This engine is coded in Python and retrieve all needed information (size-price, sku, product description…) into MongoDB database.
Back-end scrapping engine run 24/7, never crash, never stop (or website data will not up to date).
Works Steps :
1/ Backup actual scrapping infrastructure then migrate to new servers (from 2 olds windows servers to 4 new fresh windows servers, cause we need to speed up a lot scrapping engine process).
2/ Ensure scrapping engine works to retrieve needed data from all targeted shops as it actually works for both DB collection name
.
3/ Scrapping issue adjustments :
• Add logical / request to submit / force (send request to scrapping engine) : Dedicated Unique SKU CRON Task update, and also Batch (multiple) SKU CRON Task update
• « Last Price Value » must updated as soon as new product is added into MongoDB (not later when cron task will scrape this data), otherwise we need to wait a long time to get « Last Sale Price » value displayed on front-end page.
• Scrapped Size missmatch on some website :
« restock » website got weird size on some product like and also some wrong product scrapped / redirect link
• Add Calendar page (detailed into attached document) to product scrapping and create corresponding products pages. Then also display into website homepage, all next products to release. For this calendar frontend section we must only display : product image, product name, release date, last sale price. Also clicking redirect to product page.
4/ Add retail market to scrapping engine and create « retail » market collection into mongoDb
5/ Add some extra Shops website to scrapping engine