Optimize Python-Based Real-Estate Web Scraper
Budget: $30 – $250 USD
We have a working but partially complete Python platform that scrapes, consolidates and
normalises real-estate listings from several Angolan sources into a single database. We need an
experienced developer to improve and optimise it — not build from scratch. The full technical
specification is attached.
What already works
6 sources: AngoCasa (5,000+ listings), AngoImóveis, MIA (Supabase API), Zenki, REE Gera, and RE/MAX (Azure Cognitive Search API — partial; basis to be confirmed).
SQLite database, deduplication, resumable checkpoints, price-change history, Excel export.
What we need — the focus of this job (improvements)
RE/MAX parser: fix extraction of headline, typology, bedrooms, bathrooms (WC), area and description (the API often returns internal references that must be reconstructed).
AngoCasa: reliably extract phone, WhatsApp, advertiser/agency and main image URL.
Performance: replace sequential detail-fetching with parallel execution (ThreadPoolExecutor or async) + per-domain rate limiting. Target: 60–80% faster. The goal is to extract all listings from each source (the --max-anuncios flag is only for testing).
Excel: monetary format 8 500 000,00, Yes/No for booleans, and interactive Excel dashboards/filters (analytics are delivered inside Excel).
Normalisation: currencies AKZ / USD / EUR (BNA reference rate, stored per conversion), typologies (T0/T1/V4) and a neighbourhood (bairro) dictionary.
Later milestones (optional)
New public sources: Facebook Marketplace (search-term based), then Facebook groups and Instagram. Propose a stable, lawful collection approach.
SQLite → PostgreSQL migration; opportunity score; monthly executive report.
Hard requirements
Source isolation — if one source fails, the others must keep running.
Maintainable, modular, documented code (one module per source) + README, requirements.txt, config file, logs, basic tests and selector-maintenance notes.
How to bid — please include
Examples/links of resilient scrapers and API integrations you've built (Azure Cognitive Search / Supabase a plus).
Your approach to parallelisation and to handling sites that change frequently.
A realistic timeline and milestone breakdown.
normalises real-estate listings from several Angolan sources into a single database. We need an
experienced developer to improve and optimise it — not build from scratch. The full technical
specification is attached.
What already works
6 sources: AngoCasa (5,000+ listings), AngoImóveis, MIA (Supabase API), Zenki, REE Gera, and RE/MAX (Azure Cognitive Search API — partial; basis to be confirmed).
SQLite database, deduplication, resumable checkpoints, price-change history, Excel export.
What we need — the focus of this job (improvements)
RE/MAX parser: fix extraction of headline, typology, bedrooms, bathrooms (WC), area and description (the API often returns internal references that must be reconstructed).
AngoCasa: reliably extract phone, WhatsApp, advertiser/agency and main image URL.
Performance: replace sequential detail-fetching with parallel execution (ThreadPoolExecutor or async) + per-domain rate limiting. Target: 60–80% faster. The goal is to extract all listings from each source (the --max-anuncios flag is only for testing).
Excel: monetary format 8 500 000,00, Yes/No for booleans, and interactive Excel dashboards/filters (analytics are delivered inside Excel).
Normalisation: currencies AKZ / USD / EUR (BNA reference rate, stored per conversion), typologies (T0/T1/V4) and a neighbourhood (bairro) dictionary.
Later milestones (optional)
New public sources: Facebook Marketplace (search-term based), then Facebook groups and Instagram. Propose a stable, lawful collection approach.
SQLite → PostgreSQL migration; opportunity score; monthly executive report.
Hard requirements
Source isolation — if one source fails, the others must keep running.
Maintainable, modular, documented code (one module per source) + README, requirements.txt, config file, logs, basic tests and selector-maintenance notes.
How to bid — please include
Examples/links of resilient scrapers and API integrations you've built (Azure Cognitive Search / Supabase a plus).
Your approach to parallelisation and to handling sites that change frequently.
A realistic timeline and milestone breakdown.
Related categories:
Python
Excel
Web Scraping
PostgreSQL
SQLite
Elasticsearch
Data Analytics
Data Extraction
API Integration