Chegg Data Scraper & JSON Formatter
Budget: $1,500 – $3,000 USD
1. Project Overview
We need to develop a Python-based web scraper that extracts specific data from Chegg (https://www.chegg.com) and outputs it in a structured JSON format. The JSON file should mirror the provided sample (e.g., “chegg_demo_api_graphql_response (2).json”) and include the following fields for each scraped entry:
序号 (Index/Number): A sequential identifier.
连接 (URL): The corresponding link for each data entry.
问题 (Question): The question associated with the answer.
内容 (Content): The detailed content or answer text.
2. Key Requirements
2.1 Data Extraction
Target Website:
Scrape data from https://www.chegg.com.
Data Elements:
For each Chegg answer, extract:
序号: A sequential number.
连接: The URL corresponding to the answer or each step.
问题: The question text.
内容: The answer content (including all individual steps).
Duplicate Prevention:
Implement a mechanism to ensure that duplicate content is not fetched or stored.
2.2 Output Format
File Format:
The final deliverable should be a JSON file.
Structure:
The JSON should follow a structure similar to the attached sample, containing keys for "序号", "连接", "问题", and "内容".
2.3 Technical Approach
Programming Language:
Use Python.
Scraping Method:
Since Chegg does not provide an API interface, web scraping is required. Consider using libraries such as:
Requests & BeautifulSoup: For simple HTTP requests and HTML parsing.
Selenium: If browser automation is needed (for dynamic content).
Concurrency & Performance:
Consider implementing concurrent or asynchronous requests (using modules like asyncio, threading, or multiprocessing) to improve scraping speed.
Include configurable parameters for concurrency settings so that the performance can be tuned as needed.
Workflow Steps:
Identify & Define Targets: Determine which pages and sections on Chegg need to be scraped.
Data Extraction Logic: Write routines to extract the required fields (序号, 连接, 问题, 内容) from each page.
Data Structuring: Organize the scraped data into the predefined JSON format.
Error Handling & Deduplication: Implement robust error handling and duplicate detection to ensure clean data.
Testing & Validation: Validate the JSON output against the provided sample to ensure compliance.
3. Delivery Timeline
Deadline:
The complete solution (including code and data output) must be delivered within 1 day.
Note: If you are not confident in meeting this deadline, please do not bid.
4. Communication & Next Steps
Clarification of Requirements:
Please confirm that you understand all the requirements and outline your approach in detail, including your plan for handling concurrency and duplicate data.
Relevant Experience:
Provide examples or links to similar projects you have completed to help us evaluate your expertise in this area.
Discussion:
If needed, we are available for a call to discuss any specifics or clarify further details regarding the task.
Budget & Timeline Confirmation:
Ensure that your bid reflects the agreed budget and that you can adhere to the 1-day turnaround.
5. Final Notes
The solution should be a well-documented Python script with clear comments and modular code.
It is essential that every Chegg answer is completely scraped (with all individual steps), each accompanied by its corresponding URL, and structured exactly as per the required JSON format.
Please share your complete workflow and any configuration options (such as concurrent settings) that will be implemented.
We need to develop a Python-based web scraper that extracts specific data from Chegg (https://www.chegg.com) and outputs it in a structured JSON format. The JSON file should mirror the provided sample (e.g., “chegg_demo_api_graphql_response (2).json”) and include the following fields for each scraped entry:
序号 (Index/Number): A sequential identifier.
连接 (URL): The corresponding link for each data entry.
问题 (Question): The question associated with the answer.
内容 (Content): The detailed content or answer text.
2. Key Requirements
2.1 Data Extraction
Target Website:
Scrape data from https://www.chegg.com.
Data Elements:
For each Chegg answer, extract:
序号: A sequential number.
连接: The URL corresponding to the answer or each step.
问题: The question text.
内容: The answer content (including all individual steps).
Duplicate Prevention:
Implement a mechanism to ensure that duplicate content is not fetched or stored.
2.2 Output Format
File Format:
The final deliverable should be a JSON file.
Structure:
The JSON should follow a structure similar to the attached sample, containing keys for "序号", "连接", "问题", and "内容".
2.3 Technical Approach
Programming Language:
Use Python.
Scraping Method:
Since Chegg does not provide an API interface, web scraping is required. Consider using libraries such as:
Requests & BeautifulSoup: For simple HTTP requests and HTML parsing.
Selenium: If browser automation is needed (for dynamic content).
Concurrency & Performance:
Consider implementing concurrent or asynchronous requests (using modules like asyncio, threading, or multiprocessing) to improve scraping speed.
Include configurable parameters for concurrency settings so that the performance can be tuned as needed.
Workflow Steps:
Identify & Define Targets: Determine which pages and sections on Chegg need to be scraped.
Data Extraction Logic: Write routines to extract the required fields (序号, 连接, 问题, 内容) from each page.
Data Structuring: Organize the scraped data into the predefined JSON format.
Error Handling & Deduplication: Implement robust error handling and duplicate detection to ensure clean data.
Testing & Validation: Validate the JSON output against the provided sample to ensure compliance.
3. Delivery Timeline
Deadline:
The complete solution (including code and data output) must be delivered within 1 day.
Note: If you are not confident in meeting this deadline, please do not bid.
4. Communication & Next Steps
Clarification of Requirements:
Please confirm that you understand all the requirements and outline your approach in detail, including your plan for handling concurrency and duplicate data.
Relevant Experience:
Provide examples or links to similar projects you have completed to help us evaluate your expertise in this area.
Discussion:
If needed, we are available for a call to discuss any specifics or clarify further details regarding the task.
Budget & Timeline Confirmation:
Ensure that your bid reflects the agreed budget and that you can adhere to the 1-day turnaround.
5. Final Notes
The solution should be a well-documented Python script with clear comments and modular code.
It is essential that every Chegg answer is completely scraped (with all individual steps), each accompanied by its corresponding URL, and structured exactly as per the required JSON format.
Please share your complete workflow and any configuration options (such as concurrent settings) that will be implemented.