New Features for Golang web product scraper (Some Angular stuff to)
Budget: $750 – $1,500 USD
Current Source Code available to those who have read through the description and are interested but need additional information for bidding.
Project Summary:
I have a semifunctional web scraping project based on Golang (using gin-gonic, gocolly, and mongodb) for the backend and scraping, and Angular (currenly version 11) for the consumer facing frontend and the control frontend to the scraper. The scraper uses a series of "navigation elements" in a "navigation flow" that describes how the scraper traverses the product site to extract product data. It also currently does some amount of parsing and data normalization as well as send updated data about scraped products and assets to the consumer facing backend. I am looking to bring the project up to date and working again, as well as add additional features.
Who I am looking for:
I am looking for a solo or small team that has solid golang skills and abiltity in web scraping. Also at least enough knowledge of frontend/angular in order to launch frontend controller in order to test/debug code, strongly prefer enough angular skills to improve/add small features to assist in scraper activities. Near fluent to fluent English.
Current Project Status/Description:
Project is in a semi broken state, Angular is an old version that probably needs some work to get the app running, do not have a complete mongodb image of functioning app with navigation flows or other needed data. Have a script that partially populates data that can serve as an example of what data should look like but is likely outdated and some work needed to restore it to a state capable of scraping.
Scraper Backend - Uses a recursive function to navigate a product site by use of a series of data objects that provide actions, css selectors, and additional metadata for help in parsing. Also provides all the endpoints for interacting with the Scraper Control Frontend in order to make new "navigation flows" and do additional editing on scraped items and provide additional data for automated parsing and normalization.
Scraper Control Backend - Four main parts:
1. Navigation Flow Editor - Editor to create and edit navigation flow objects that describe how the scraper navigates and extracts data.
2. Label Tag Editor (deprecation/rework needed) - Creates and edits labeltags which are objects that assist in providing data to assist in parsing and providing filterable metadata. Overly broad concept, that needs rework and probably to be split into several more granular concepts.
3. Master Item Editor - Editor to create and edit "master items". When products are scraped they are scraped as "store items" which do not have fully filled out or normalized data. The editor can be prepopulated by the backend with data from a store item, and then further edited to ensure high quality data that can be searched and filtered for on the consumer facing site.
4. Item Matcher - A two panel ui which allows drag and drop from a list of store items on the right to a list of master items on the left to associate the store items to a master item. If a master item does not exist for a store item the master item editor can be launched from a store item.
5. Control Panel - Panel for running tasks such as scraping, or asset collection, as well as sending data to the consumer backend.
Consumer Frontend - End user facing frontend in Angular that provides a store front like interface allowing for navigating and filtering of products. All visible products are master items and the store items show up on the master item they are associated with.
Consumer Backend - Golang server that serves up frontend app and assets and provides api endpoints to access products from.
Work to be done/Features:
Note: While all work outlined is needed to be done some of the Angular/Frontend specific stuff either I may complete or will find someone else to do.
0A. Getting Project "Running" - Update Angular to latest or near latest angular version and have all backends and frontends building and running.
0B. Demonstrate Scraping - Understand Scraper and its UI enough to create a navigation flow, scrape a website, clean up a few items and update the consumer backend to show the products on the Consumer Frontend.
1. Extend the scraper to run with two different scraping frameworks, currently working on gocolly but also need to run using chromedp (chromium devtool protocols essentially golang puppeteer), in order to work with all sites some of which require executing javascript to navigate and bypass bot checks.
2. Extend Product categories - currently only written to scrape one type of product, want to include other types of products as well as parts, and (later) product accessories. Mostly this will be just creating new struct types with different fields and expanding that parsers and filter functionality.
3A. Get MVP of database (yes this will have quite a bit of "data entry") expand scraper to include 5 sites, ~10 manufactures, ~20 product categories. Scrape products from sites, and create master items from said items.
3B. Review scraping process look for improvements, and if the products objects, parsing, and navigation flows can be improved.
4A. Add authentication and ability for partial access of Scraper Backend/Frontend to allow third party contractors to use to have them running the Master Item Editor and Item Matcher. Have these Edits pushed to a "staging area".
4B. Add logic and interface to allow review of third party edits / item creations prior to moving the data from the staging area to the production ready area.
That is all I need at this time, but I have a lot more foreseeable work available depending on results of how this goes. Also, If you are knowledgeable about named entity recognition, natural language processing, or just have other ideas based on AI or machine learning or just interface or process ideas how to make this work better definitely room to expand the body of work.
Project Summary:
I have a semifunctional web scraping project based on Golang (using gin-gonic, gocolly, and mongodb) for the backend and scraping, and Angular (currenly version 11) for the consumer facing frontend and the control frontend to the scraper. The scraper uses a series of "navigation elements" in a "navigation flow" that describes how the scraper traverses the product site to extract product data. It also currently does some amount of parsing and data normalization as well as send updated data about scraped products and assets to the consumer facing backend. I am looking to bring the project up to date and working again, as well as add additional features.
Who I am looking for:
I am looking for a solo or small team that has solid golang skills and abiltity in web scraping. Also at least enough knowledge of frontend/angular in order to launch frontend controller in order to test/debug code, strongly prefer enough angular skills to improve/add small features to assist in scraper activities. Near fluent to fluent English.
Current Project Status/Description:
Project is in a semi broken state, Angular is an old version that probably needs some work to get the app running, do not have a complete mongodb image of functioning app with navigation flows or other needed data. Have a script that partially populates data that can serve as an example of what data should look like but is likely outdated and some work needed to restore it to a state capable of scraping.
Scraper Backend - Uses a recursive function to navigate a product site by use of a series of data objects that provide actions, css selectors, and additional metadata for help in parsing. Also provides all the endpoints for interacting with the Scraper Control Frontend in order to make new "navigation flows" and do additional editing on scraped items and provide additional data for automated parsing and normalization.
Scraper Control Backend - Four main parts:
1. Navigation Flow Editor - Editor to create and edit navigation flow objects that describe how the scraper navigates and extracts data.
2. Label Tag Editor (deprecation/rework needed) - Creates and edits labeltags which are objects that assist in providing data to assist in parsing and providing filterable metadata. Overly broad concept, that needs rework and probably to be split into several more granular concepts.
3. Master Item Editor - Editor to create and edit "master items". When products are scraped they are scraped as "store items" which do not have fully filled out or normalized data. The editor can be prepopulated by the backend with data from a store item, and then further edited to ensure high quality data that can be searched and filtered for on the consumer facing site.
4. Item Matcher - A two panel ui which allows drag and drop from a list of store items on the right to a list of master items on the left to associate the store items to a master item. If a master item does not exist for a store item the master item editor can be launched from a store item.
5. Control Panel - Panel for running tasks such as scraping, or asset collection, as well as sending data to the consumer backend.
Consumer Frontend - End user facing frontend in Angular that provides a store front like interface allowing for navigating and filtering of products. All visible products are master items and the store items show up on the master item they are associated with.
Consumer Backend - Golang server that serves up frontend app and assets and provides api endpoints to access products from.
Work to be done/Features:
Note: While all work outlined is needed to be done some of the Angular/Frontend specific stuff either I may complete or will find someone else to do.
0A. Getting Project "Running" - Update Angular to latest or near latest angular version and have all backends and frontends building and running.
0B. Demonstrate Scraping - Understand Scraper and its UI enough to create a navigation flow, scrape a website, clean up a few items and update the consumer backend to show the products on the Consumer Frontend.
1. Extend the scraper to run with two different scraping frameworks, currently working on gocolly but also need to run using chromedp (chromium devtool protocols essentially golang puppeteer), in order to work with all sites some of which require executing javascript to navigate and bypass bot checks.
2. Extend Product categories - currently only written to scrape one type of product, want to include other types of products as well as parts, and (later) product accessories. Mostly this will be just creating new struct types with different fields and expanding that parsers and filter functionality.
3A. Get MVP of database (yes this will have quite a bit of "data entry") expand scraper to include 5 sites, ~10 manufactures, ~20 product categories. Scrape products from sites, and create master items from said items.
3B. Review scraping process look for improvements, and if the products objects, parsing, and navigation flows can be improved.
4A. Add authentication and ability for partial access of Scraper Backend/Frontend to allow third party contractors to use to have them running the Master Item Editor and Item Matcher. Have these Edits pushed to a "staging area".
4B. Add logic and interface to allow review of third party edits / item creations prior to moving the data from the staging area to the production ready area.
That is all I need at this time, but I have a lot more foreseeable work available depending on results of how this goes. Also, If you are knowledgeable about named entity recognition, natural language processing, or just have other ideas based on AI or machine learning or just interface or process ideas how to make this work better definitely room to expand the body of work.