Create Python App and Restful API to find/save/update same products from different e-commerce websites
Budget: $100 – $400 USD
I need to have an App and RESTful API to scraping some websites and find prices for same product.
YOU NEED TO KNOW: you no need to start from scratching. some part of app is developed. you need to improve the current app and implement update-task of application. as well as you need to fix some issues and implement API.
*** This app has two part. one is application that will find/save/update products. other one is API that can communicate between user and MongoDB ***
Application should develop separate by API. means, you need to have two separate python projects.
API backend will deploy on separate server.
Application instances will deploy on different servers to scrap different categories parallelly.
All websites you need to scrap located in us.
We have Amazon, Best Buy, Newegg.
Scenario:
This API should pick a product in Amazon (Coca-Cola 1.5 Lt sugar free) from many products that amazon provides based on category URL which we provided in db.
Now, API should looking for same product in Best Buy, Newegg, and Costco by using search query as well.
When same product found in other e-commerce websites, API should save the product details and all prices in MongoDB collection.
Name, prices, images addresses, brand, stock status, stock count, etc... of the product in all scraped e-commerce websites should save in MongoDB.
Many product data should save in db per every website.
All image addresses of product in all website should store in MongoDB collection separated by website name.
For example related to above line:
We have 4 e-commerce websites.
Scraped product has 5 images in every online shopping websites.
In this case we need to save 20 image address in MongoDB.
This web app should scrap the whole category and do same job for all products one-by-one.
This task should run every period which we specify in db.
App should logics to make sure of same product finding in different websites
App should compare some part of product listing in different e-commerce websites.
* Product Title
* Product Description
* Product Images
We should be able to modify similarity score for app to use that value to compare the title, description, and image of products to make sure of found same product.
Example:
We will specify 80% similarity for title, 60% for description, 60% for images.
If app meet above scores, It will consider that found product as the same product.
comparison logic should be improvable.
Means, We should have a method in API that user can tell the app whether provided result is accurate or not to rate the quality of product finding (app comparison logic improvement)
Ex: If user stated YES, we can continue with above score value. if user stated NO, app should increase the score by 1 (80+1, 60+1, 60+1)
First website that you need to start finding a product from, is Amazon.
We should make sure of same product comparison from different websites.
Be aware that Title, images, and description of product can be different in different websites.
Product comparison accuracy is most important part of project.
If we compare two different product, it will be useless.
API functions should response many type of data as JSON.
We should be able to specify the threads of find-task and update-task in DB.
Ex: App should read the threads count from db and run threads as many as we defined in db.
DATABASE STRUCTURE IS PREPARED BY ME AND WILL PROVIDE.
PROXY API WILL PROVIDE TO YOU.
FREELANCER SHALL TO USE ClickUp AS TASK MANAGEMENT TOOL.
FIRST OF ALL, READ PROJECT DETAILS AND TELL ME HOW YOU WANT TO DO THIS JOB ?
WHAT IS YOUR PLAN TO AVOID BLOCKING THE APP FROM CRAWLING BY E-COMMERCE WEBSITES WHILE WE NEED TO UPDATE ALL PRODUCT PRICES MANY TIMES IN A DAY ?
WE NEED TO FIND NEW PRODUCT AND UPDATE CURRENT PRODUCT DETAILS IN DB AT LEAST 6 TIMES IN A DAY
*** We defined 5 steps for this project. Milestones will create one-by-one during these steps. Milestone will never release until project end ***
YOU NEED TO KNOW: you no need to start from scratching. some part of app is developed. you need to improve the current app and implement update-task of application. as well as you need to fix some issues and implement API.
*** This app has two part. one is application that will find/save/update products. other one is API that can communicate between user and MongoDB ***
Application should develop separate by API. means, you need to have two separate python projects.
API backend will deploy on separate server.
Application instances will deploy on different servers to scrap different categories parallelly.
All websites you need to scrap located in us.
We have Amazon, Best Buy, Newegg.
Scenario:
This API should pick a product in Amazon (Coca-Cola 1.5 Lt sugar free) from many products that amazon provides based on category URL which we provided in db.
Now, API should looking for same product in Best Buy, Newegg, and Costco by using search query as well.
When same product found in other e-commerce websites, API should save the product details and all prices in MongoDB collection.
Name, prices, images addresses, brand, stock status, stock count, etc... of the product in all scraped e-commerce websites should save in MongoDB.
Many product data should save in db per every website.
All image addresses of product in all website should store in MongoDB collection separated by website name.
For example related to above line:
We have 4 e-commerce websites.
Scraped product has 5 images in every online shopping websites.
In this case we need to save 20 image address in MongoDB.
This web app should scrap the whole category and do same job for all products one-by-one.
This task should run every period which we specify in db.
App should logics to make sure of same product finding in different websites
App should compare some part of product listing in different e-commerce websites.
* Product Title
* Product Description
* Product Images
We should be able to modify similarity score for app to use that value to compare the title, description, and image of products to make sure of found same product.
Example:
We will specify 80% similarity for title, 60% for description, 60% for images.
If app meet above scores, It will consider that found product as the same product.
comparison logic should be improvable.
Means, We should have a method in API that user can tell the app whether provided result is accurate or not to rate the quality of product finding (app comparison logic improvement)
Ex: If user stated YES, we can continue with above score value. if user stated NO, app should increase the score by 1 (80+1, 60+1, 60+1)
First website that you need to start finding a product from, is Amazon.
We should make sure of same product comparison from different websites.
Be aware that Title, images, and description of product can be different in different websites.
Product comparison accuracy is most important part of project.
If we compare two different product, it will be useless.
API functions should response many type of data as JSON.
We should be able to specify the threads of find-task and update-task in DB.
Ex: App should read the threads count from db and run threads as many as we defined in db.
DATABASE STRUCTURE IS PREPARED BY ME AND WILL PROVIDE.
PROXY API WILL PROVIDE TO YOU.
FREELANCER SHALL TO USE ClickUp AS TASK MANAGEMENT TOOL.
FIRST OF ALL, READ PROJECT DETAILS AND TELL ME HOW YOU WANT TO DO THIS JOB ?
WHAT IS YOUR PLAN TO AVOID BLOCKING THE APP FROM CRAWLING BY E-COMMERCE WEBSITES WHILE WE NEED TO UPDATE ALL PRODUCT PRICES MANY TIMES IN A DAY ?
WE NEED TO FIND NEW PRODUCT AND UPDATE CURRENT PRODUCT DETAILS IN DB AT LEAST 6 TIMES IN A DAY
*** We defined 5 steps for this project. Milestones will create one-by-one during these steps. Milestone will never release until project end ***