Web Scraping (WM)
Budget: $250 – $750 USD
I'm seeking coders experienced in Python for a project that involves scraping data from a particular car listing web site on a daily basis. The scraped data should be stored in a Postgre database.
Key Requirements:
- Web scraping data from a particular brazilian website
- Collecting car details, pricing information and seller contact details
- Proficiency in Python
- Experience working with Postgre databases
- Implementing a daily data scraping schedule
The project requires someone who can ensure the scraped data is accurate, timely, and is able to handle the data efficiently. A background in web scraping, database management and Python programming is an absolute must.
---
Create a program that daily scrapes car listings from a particular brazilian car listing website, for a specified region,
**so that** I can collect and store detailed car information in a PostgreSQL database.
**Acceptance Criteria:**
1. **Program Initialization:**
- The program should log the date/time and the input parameter at startup.
- If no input parameter is provided, the program should terminate and log that there was an attempt to execute without the input parameter.
2. **Web Scraping:**
- Navigate to (website) > "comprar" > "carros usados" > "localização".
- For each car listed, access the detail link and collect the following data:
- **Images:** Collection of car images (*1)
- **Brand:** (*2)
- **Model:** (*3)
- **Specification:** (*4)
- **Price:** (*21)
- **Location:** (*5)
- **Year / Model Year:** (*6)
- **Mileage:** (*7)
- **Transmission:** (*8)
- **Body Type:** (*9)
- **Fuel Type:** (*10)
- **License Plate / Plate End:** (*11)
- **Color:** (*12)
- **Accepts Trade:** (yes/no) (*13)
- **IPVA Paid:** (*14)
- **Licensed:** (*15)
- **Features:** [airbag, alarm, air conditioning, ABS brakes] (*16)
- **Inspected:** (yes/no) (*17)
- **Advertised Price:** (*18)
- **Promotional Price:** (*19)
- **FIPE Value:** (*20)
3. **Database Storage:**
- Log the timestamp of when the detail page was accessed.
- Generate or locate a unique identifier for each record to be used as the primary key.
- Store the data in a PostgreSQL table "tb_scrap_webm":
- URI: "sURI" (varchar 255) (UK)
- Brand: "sMarca" (varchar 30)
- Model: "sModelo" (varchar 30)
- Specification: "sEspecificacao" (varchar 60)
- Price: "dValor" (double)
- Location: "sLocalizacao" (varchar 60)
- Year: "iAno" (integer)
- Model Year: "iAnoModelo" (integer)
- Mileage: "dKM" (double)
- Transmission: "sCambio" (varchar 15)
- Body Type: "sCarroceria" (varchar 15)
- Fuel Type: "sCombustivel" (varchar 30)
- License Plate / Plate End: "sFinalPlaca" (char 1)
- IPVA Paid: "bIPVAPago" (boolean)
- Licensed: "bLicenciado" (boolean)
- Features: "sItens" (varchar 255)
- Inspected: "bVistoriado" (boolean)
- Advertised Price: "dValor" (double)
- Promotional Price: "dValorPromocional" (double)
- FIPE Value: "dValorFipe" (double)
- Last Accessed: "dtUltimoAcesso" (datetime)
- Store each car image in a filesystem repository using the car identifier followed by a sequential number for each image.
4. **Logging and Error Handling:**
- Log the successful processing of each car record.
- Log the successful completion of the entire process.
- Implement resilience for the following failures:
- Connection failure: wait 2 minutes and retry.
- Handle throttling from the source site by reducing the number of calls or waiting the necessary time to avoid throttling.
- Upon resuming, continue from the last successful query.
- Log all failures with time, page URL, and technical error description.
5. **Code Quality and Documentation:**
- The program should be written in Python with an object-oriented approach in at least two layers (data and application).
- Ensure the code is readable with comments on at least 70% of IF clauses and 100% of variable and function declarations, explaining their purpose and decisions.
- Deliver all source code.
**Acceptance Test:**
- Execute the program with the search parameter "Paulínia" and expect at least 20 cars in the listing.
- Ensure all listed cars are recorded in the database and log any failures.
- Success is defined as at least 80% of the listed cars being recorded/updated in the database.
Key Requirements:
- Web scraping data from a particular brazilian website
- Collecting car details, pricing information and seller contact details
- Proficiency in Python
- Experience working with Postgre databases
- Implementing a daily data scraping schedule
The project requires someone who can ensure the scraped data is accurate, timely, and is able to handle the data efficiently. A background in web scraping, database management and Python programming is an absolute must.
---
Create a program that daily scrapes car listings from a particular brazilian car listing website, for a specified region,
**so that** I can collect and store detailed car information in a PostgreSQL database.
**Acceptance Criteria:**
1. **Program Initialization:**
- The program should log the date/time and the input parameter at startup.
- If no input parameter is provided, the program should terminate and log that there was an attempt to execute without the input parameter.
2. **Web Scraping:**
- Navigate to (website) > "comprar" > "carros usados" > "localização".
- For each car listed, access the detail link and collect the following data:
- **Images:** Collection of car images (*1)
- **Brand:** (*2)
- **Model:** (*3)
- **Specification:** (*4)
- **Price:** (*21)
- **Location:** (*5)
- **Year / Model Year:** (*6)
- **Mileage:** (*7)
- **Transmission:** (*8)
- **Body Type:** (*9)
- **Fuel Type:** (*10)
- **License Plate / Plate End:** (*11)
- **Color:** (*12)
- **Accepts Trade:** (yes/no) (*13)
- **IPVA Paid:** (*14)
- **Licensed:** (*15)
- **Features:** [airbag, alarm, air conditioning, ABS brakes] (*16)
- **Inspected:** (yes/no) (*17)
- **Advertised Price:** (*18)
- **Promotional Price:** (*19)
- **FIPE Value:** (*20)
3. **Database Storage:**
- Log the timestamp of when the detail page was accessed.
- Generate or locate a unique identifier for each record to be used as the primary key.
- Store the data in a PostgreSQL table "tb_scrap_webm":
- URI: "sURI" (varchar 255) (UK)
- Brand: "sMarca" (varchar 30)
- Model: "sModelo" (varchar 30)
- Specification: "sEspecificacao" (varchar 60)
- Price: "dValor" (double)
- Location: "sLocalizacao" (varchar 60)
- Year: "iAno" (integer)
- Model Year: "iAnoModelo" (integer)
- Mileage: "dKM" (double)
- Transmission: "sCambio" (varchar 15)
- Body Type: "sCarroceria" (varchar 15)
- Fuel Type: "sCombustivel" (varchar 30)
- License Plate / Plate End: "sFinalPlaca" (char 1)
- IPVA Paid: "bIPVAPago" (boolean)
- Licensed: "bLicenciado" (boolean)
- Features: "sItens" (varchar 255)
- Inspected: "bVistoriado" (boolean)
- Advertised Price: "dValor" (double)
- Promotional Price: "dValorPromocional" (double)
- FIPE Value: "dValorFipe" (double)
- Last Accessed: "dtUltimoAcesso" (datetime)
- Store each car image in a filesystem repository using the car identifier followed by a sequential number for each image.
4. **Logging and Error Handling:**
- Log the successful processing of each car record.
- Log the successful completion of the entire process.
- Implement resilience for the following failures:
- Connection failure: wait 2 minutes and retry.
- Handle throttling from the source site by reducing the number of calls or waiting the necessary time to avoid throttling.
- Upon resuming, continue from the last successful query.
- Log all failures with time, page URL, and technical error description.
5. **Code Quality and Documentation:**
- The program should be written in Python with an object-oriented approach in at least two layers (data and application).
- Ensure the code is readable with comments on at least 70% of IF clauses and 100% of variable and function declarations, explaining their purpose and decisions.
- Deliver all source code.
**Acceptance Test:**
- Execute the program with the search parameter "Paulínia" and expect at least 20 cars in the listing.
- Ensure all listed cars are recorded in the database and log any failures.
- Success is defined as at least 80% of the listed cars being recorded/updated in the database.