Web Scraping Expert for Automotive Marketplace
Budget: €30 – €250 EUR
Project Overview: We are seeking experienced web scraping experts to develop robust and reliable scraping scripts for second-hand automotive marketplace: https://auto.amag.ch/.
The goal is to build scripts that we can integrate into our existing systems to run daily, ensuring high-quality data extraction with minimal downtime.
Key Requirements:
1. Script Development:
* Create efficient web scraping scripts for various automotive marketplaces.
* Ensure the code is clean, well-documented, and easy to maintain.
* Deliver the source code, as we will be running and integrating the scripts within our existing infrastructure.
* Structure the scraped data by car brand to improve tracking and control: ensure the output format groups listings under their respective brands, making it easier to identify missing data or sources that require adjustments.
* Properly structuring extracted data fields: Some attributes need to be correctly broken down to maintain data consistency. For example, horsepower values should be distinguished between CV (metric horsepower) and kW (kilowatts) instead of mixing them. Similarly, vehicle details should be structured properly, ensuring that Make, Model, and Version are extracted as separate fields rather than a single concatenated string, which can lead to inconsistencies in analysis and filtering.
2. Anti-Bot Detection and CAPTCHAs:
* Implement strategies to avoid bot detection and handle CAPTCHAs effectively.
* Ensure scripts are resilient to changes in the websites and can handle long periods of continuous requests without issues.
3. Maintenance Support:
* Provide at least 1 month of maintenance to address potential issues such as CAPTCHA updates, changes in website structure, or detection mechanisms.
* Offer quick fixes and updates during this period to guarantee script stability.
4. Scalability and Performance:
* Design scripts to run efficiently and collect data without overloading the source website.
* Ensure that each script can run on a daily schedule (every 24 hours) without errors.
5. Output Format:
* The extracted data must be provided in JSON format.
* Ensure the JSON structure is consistent, well-organized, and easy to parse for integration into our existing systems.
6. Performance Indicators:
* Provide performance metrics for the scraping scripts, including execution time per source and data extraction success rate.
* These indicators will help us evaluate the efficiency and reliability of each script.
7. Future Opportunities:
* We have over 40 additional sources to scrape. Successful completion of this project with high-quality work will lead to further job opportunities.
---
Tech stack
* Google Cloud on Ubuntu powers our infrastructure.
* Docker handles containerization for consistent environments.
* Django (Python) is our backend framework, paired with Celery for background tasks and asynchronous processing.
---
To Apply
When submitting your proposal, please include:
1. Examples of similar web scraping projects you have worked on.
2. Your approach to handling anti-bot detection and CAPTCHAs.
3. Estimated time and cost for developing a single scraping script.
4. Confirmation of your ability to provide 1 month of maintenance support.
5. Your hourly or fixed pricing model.
Note: We are looking for professionals who can offer realistic pricing upfront. Please avoid providing low initial quotes only to increase them after discussing the project. Clear and transparent communication is key.
Thank you! We look forward to your proposals.
The goal is to build scripts that we can integrate into our existing systems to run daily, ensuring high-quality data extraction with minimal downtime.
Key Requirements:
1. Script Development:
* Create efficient web scraping scripts for various automotive marketplaces.
* Ensure the code is clean, well-documented, and easy to maintain.
* Deliver the source code, as we will be running and integrating the scripts within our existing infrastructure.
* Structure the scraped data by car brand to improve tracking and control: ensure the output format groups listings under their respective brands, making it easier to identify missing data or sources that require adjustments.
* Properly structuring extracted data fields: Some attributes need to be correctly broken down to maintain data consistency. For example, horsepower values should be distinguished between CV (metric horsepower) and kW (kilowatts) instead of mixing them. Similarly, vehicle details should be structured properly, ensuring that Make, Model, and Version are extracted as separate fields rather than a single concatenated string, which can lead to inconsistencies in analysis and filtering.
2. Anti-Bot Detection and CAPTCHAs:
* Implement strategies to avoid bot detection and handle CAPTCHAs effectively.
* Ensure scripts are resilient to changes in the websites and can handle long periods of continuous requests without issues.
3. Maintenance Support:
* Provide at least 1 month of maintenance to address potential issues such as CAPTCHA updates, changes in website structure, or detection mechanisms.
* Offer quick fixes and updates during this period to guarantee script stability.
4. Scalability and Performance:
* Design scripts to run efficiently and collect data without overloading the source website.
* Ensure that each script can run on a daily schedule (every 24 hours) without errors.
5. Output Format:
* The extracted data must be provided in JSON format.
* Ensure the JSON structure is consistent, well-organized, and easy to parse for integration into our existing systems.
6. Performance Indicators:
* Provide performance metrics for the scraping scripts, including execution time per source and data extraction success rate.
* These indicators will help us evaluate the efficiency and reliability of each script.
7. Future Opportunities:
* We have over 40 additional sources to scrape. Successful completion of this project with high-quality work will lead to further job opportunities.
---
Tech stack
* Google Cloud on Ubuntu powers our infrastructure.
* Docker handles containerization for consistent environments.
* Django (Python) is our backend framework, paired with Celery for background tasks and asynchronous processing.
---
To Apply
When submitting your proposal, please include:
1. Examples of similar web scraping projects you have worked on.
2. Your approach to handling anti-bot detection and CAPTCHAs.
3. Estimated time and cost for developing a single scraping script.
4. Confirmation of your ability to provide 1 month of maintenance support.
5. Your hourly or fixed pricing model.
Note: We are looking for professionals who can offer realistic pricing upfront. Please avoid providing low initial quotes only to increase them after discussing the project. Clear and transparent communication is key.
Thank you! We look forward to your proposals.