Build Big Data Web Crawler / Scraper | Machine Learning (Multiproxy / Multithreaded)
Budget: $250 – $750 USD
I have a list of 200 Million websites I'd like to have categorized into various industries
1. For example, www.shein.com would be classified as "fashion" as determined by the machine learning classification model you create. The classification model would also need to categorize and rate its other top 5 potential categories. All the results should be stored in a database for eventual client use. The scraper should so be able to update/remove the entries and keep track of the dates such as when it was last crawled
2. The crawler will also need to look for and try to identify 10 or so technologies used on the website. IE Shopify, WordPress, woo-commerce, (Similar to BuiltWith)
3. The crawler should have the official social media links on all the pages associated with the brand IE Facebook, Instagram, and Youtube.
I'm looking for someone that can develop the entire pipeline and train the machine learning that is required.
in your proposal include the technologies you would be using to accomplish the task, Also the cloud services (if used) The tech stack to be more than sufficient and quick as possible to handle the entire job.
1. For example, www.shein.com would be classified as "fashion" as determined by the machine learning classification model you create. The classification model would also need to categorize and rate its other top 5 potential categories. All the results should be stored in a database for eventual client use. The scraper should so be able to update/remove the entries and keep track of the dates such as when it was last crawled
2. The crawler will also need to look for and try to identify 10 or so technologies used on the website. IE Shopify, WordPress, woo-commerce, (Similar to BuiltWith)
3. The crawler should have the official social media links on all the pages associated with the brand IE Facebook, Instagram, and Youtube.
I'm looking for someone that can develop the entire pipeline and train the machine learning that is required.
in your proposal include the technologies you would be using to accomplish the task, Also the cloud services (if used) The tech stack to be more than sufficient and quick as possible to handle the entire job.