Python Web Scraper for Indonesian E-Grocery
Budget: $250 – $750 USD
I'm looking for a proficient web scraper to collect data from an e-grocery website
Key Requirements:
Create an executable scraper in Python that can extract the SKU (product) list & their metadata from the following Indonesian online grocery website protected by cloudflare
The website will be provided via direct message
The scraper must be IP-block tolerant, meaning if the scraper gets blocked, execution can be continued from the point where the last row of data was successfully extracted.
Scraper functions:
get_product_urls
Input:
latitude
longitude
Output:
List of URLs of ALL products available (being sold) in the whole website when users access the website from location = (latitude, longitude)
Benchmark: there should be around ~7000 products
get_product_metadata
Input:
latitude
longitude
product_url
Output:
list of metadata of product from product_url, accessed from (latitude, longitude)
Mandatory metadata to store (please see inspect element details of the sample product URL provided)
name
description
price
final_price
height
length
width
weight
size
uom
brand
compile_products
Input:
product_metadata (~7000 unique products)
Output
Pandas dataframe containing all product metadata
Columns
latitude
longitude
product_url
name
description
price
final_price
height
length
width
weight
size
uom
brand
Unit tests
The scraper must include unit tests that cover main and error test cases, including optional parameters
Submission
The following must be submitted:
- Scraper code and unit tests
- README file for running and deploying the scraper
- Output of all SKUs from the online grocery specified above, with latitude and longitude = -6.292679, 106.784794 (South Jakarta location)
Project Acceptance Testing
The scraper output will be manually verified within 24 hours of submission and SKU list must be 95% accurate. The SKU list is the most important aspect of the scraper output.
Additional remarks:
Previously, we successfully scraped the website using the attached script (note: this script is a simplified version and lacks all the required functionalities). However, recent Cloudflare security implementations have unfortunately blocked its access.
Ideal Skills:
- Proficient in web scraping tools and techniques
- Experience in data extraction and processing
Please provide examples of previous similar projects you've completed
Key Requirements:
Create an executable scraper in Python that can extract the SKU (product) list & their metadata from the following Indonesian online grocery website protected by cloudflare
The website will be provided via direct message
The scraper must be IP-block tolerant, meaning if the scraper gets blocked, execution can be continued from the point where the last row of data was successfully extracted.
Scraper functions:
get_product_urls
Input:
latitude
longitude
Output:
List of URLs of ALL products available (being sold) in the whole website when users access the website from location = (latitude, longitude)
Benchmark: there should be around ~7000 products
get_product_metadata
Input:
latitude
longitude
product_url
Output:
list of metadata of product from product_url, accessed from (latitude, longitude)
Mandatory metadata to store (please see inspect element details of the sample product URL provided)
name
description
price
final_price
height
length
width
weight
size
uom
brand
compile_products
Input:
product_metadata (~7000 unique products)
Output
Pandas dataframe containing all product metadata
Columns
latitude
longitude
product_url
name
description
price
final_price
height
length
width
weight
size
uom
brand
Unit tests
The scraper must include unit tests that cover main and error test cases, including optional parameters
Submission
The following must be submitted:
- Scraper code and unit tests
- README file for running and deploying the scraper
- Output of all SKUs from the online grocery specified above, with latitude and longitude = -6.292679, 106.784794 (South Jakarta location)
Project Acceptance Testing
The scraper output will be manually verified within 24 hours of submission and SKU list must be 95% accurate. The SKU list is the most important aspect of the scraper output.
Additional remarks:
Previously, we successfully scraped the website using the attached script (note: this script is a simplified version and lacks all the required functionalities). However, recent Cloudflare security implementations have unfortunately blocked its access.
Ideal Skills:
- Proficient in web scraping tools and techniques
- Experience in data extraction and processing
Please provide examples of previous similar projects you've completed
Related categories:
Python
Data Processing
Web Scraping
Software Architecture
Data Mining
Data Extraction
Cloudflare
Pandas