Web data extraction
Budget: $750 – $1,500 USD
FTE Data extraction specification
Goal:
To create a web data extraction software to extract data at scale from any website The software should also be responsible and respect the crawling websites. The software should be able to run under proxy IPs and be easy to install and manage. The goal is to extract data from 100+ websites globally (India and US to begin with).
Requirements:
the time to configure new websites or altering existing web sites can be done in minutes with minimal understanding of the software.
An user interface to manage the meta data for the websites to be crawled. This would include the site name, URL, frequency etc.
Some of the management functions would include adding new websites, deactivating, activating websites, changes to existing URLs, changing the attributes etc...
For each website, I should be able to define the attributes for each item (Item could be product or service) that I want to extract. For example, website id, title, description, pricing, shipping, and image for a product on Amazon.
Each element should be cleansed to be free of HTML and bad characters
All abusive words should be removed.
Should be able to define the categories or sections to be extracted. For ex, I should be able to extract only toys category from Amazon
Should be able to extract products based on specific attributes. For example, top seller, the products that are shown on specific tabs or quadrants of a web site.
Each extracted item should be uniquely identified. The identifier should produce the same key each time the item is extracted from the same web site. For example, the identifier for the item iPhone12 on Amazon should not change unless the name of the product changes.
The output format should support both CSV and JSON.
The software should be configurable to store the output to any desired locations including any cloud providers, disk etc..
The code should be written in Python mostly (and Perl minimally if required).
Code should be commented to make maintenance easier.
Number of parallel threads should be configurable.
A UI is required to check sample data for configured and to be configured web sites.
There should be a way to track the status of the data extraction pipe at a granular level.
Success criteria:
Ability to extract data from 10 web sites as defined by me.
Ability to change the elements to be extracted.
Demonstrate the ability to run as proxy and extract data from 1-3 desired web sites.
Code review to ensure I understand the software.
Output should be verified and approved for both JSON and CSV.
Pricing quote:
Would like to get a pricing quote with and without the UIs for the configuration manager (1) and (11).
Output format sample:
{
{“site id”: 3232},
{“date time” : “dd mon yyy hh24:mi:ss”}
{“content”:
{“row id” : “343-34-er-34”, “item name” : “x”, “item desc”: “abc”, “item price” : 20.45, “currency: “Indian”}
}
}
Fields to capture in POC:
Product Name
Product Description
Price
Site Category
Brand
Product attributes (color, weight, etc…)
The complete list of required fields will be given after the POC and the completion of the project will be determined based on the result.
Goal:
To create a web data extraction software to extract data at scale from any website The software should also be responsible and respect the crawling websites. The software should be able to run under proxy IPs and be easy to install and manage. The goal is to extract data from 100+ websites globally (India and US to begin with).
Requirements:
the time to configure new websites or altering existing web sites can be done in minutes with minimal understanding of the software.
An user interface to manage the meta data for the websites to be crawled. This would include the site name, URL, frequency etc.
Some of the management functions would include adding new websites, deactivating, activating websites, changes to existing URLs, changing the attributes etc...
For each website, I should be able to define the attributes for each item (Item could be product or service) that I want to extract. For example, website id, title, description, pricing, shipping, and image for a product on Amazon.
Each element should be cleansed to be free of HTML and bad characters
All abusive words should be removed.
Should be able to define the categories or sections to be extracted. For ex, I should be able to extract only toys category from Amazon
Should be able to extract products based on specific attributes. For example, top seller, the products that are shown on specific tabs or quadrants of a web site.
Each extracted item should be uniquely identified. The identifier should produce the same key each time the item is extracted from the same web site. For example, the identifier for the item iPhone12 on Amazon should not change unless the name of the product changes.
The output format should support both CSV and JSON.
The software should be configurable to store the output to any desired locations including any cloud providers, disk etc..
The code should be written in Python mostly (and Perl minimally if required).
Code should be commented to make maintenance easier.
Number of parallel threads should be configurable.
A UI is required to check sample data for configured and to be configured web sites.
There should be a way to track the status of the data extraction pipe at a granular level.
Success criteria:
Ability to extract data from 10 web sites as defined by me.
Ability to change the elements to be extracted.
Demonstrate the ability to run as proxy and extract data from 1-3 desired web sites.
Code review to ensure I understand the software.
Output should be verified and approved for both JSON and CSV.
Pricing quote:
Would like to get a pricing quote with and without the UIs for the configuration manager (1) and (11).
Output format sample:
{
{“site id”: 3232},
{“date time” : “dd mon yyy hh24:mi:ss”}
{“content”:
{“row id” : “343-34-er-34”, “item name” : “x”, “item desc”: “abc”, “item price” : 20.45, “currency: “Indian”}
}
}
Fields to capture in POC:
Product Name
Product Description
Price
Site Category
Brand
Product attributes (color, weight, etc…)
The complete list of required fields will be given after the POC and the completion of the project will be determined based on the result.