Real Estate Data Aggregator
Budget: $30 – $250 USD
I want to build a production-ready platform that automatically gathers real-estate listings from several online sources, starting with the main Luanda property websites and expanding later to social media or classifieds when useful. Each run must be incremental, capturing only new or changed records while preserving full change history.
Key extraction targets are the advertised price and precise location for every property; additional attributes such as size or room count can follow once the core pipeline is stable. After collection, the data should be normalised by:
• Standardising all units of measurement (m², Km, etc.)
• Formatting addresses consistently so neighbourhood and street fields align
• Applying a single, uniform set of property-type categories
The workflow must include robust deduplication, resumable checkpoints (in case of network or site interruptions), and a clear audit log of every step. Clean, consolidated data should land in a relational database that is optimised for later market analysis, opportunity detection and future analytical products, with a one-click Excel export for quick ad-hoc reporting.
Deliverables:
1. Source-specific scrapers/crawlers with respectful, throttled requests
2. Normalisation and deduplication pipeline with unit tests
3. Relational database schema (PostgreSQL preferred) storing full change history
4. Command-line or lightweight web interface to run, resume, and monitor jobs
5. Excel/CSV export module
6. Deployment instructions and brief technical documentation
Python, Scrapy, BeautifulSoup, Selenium or comparable tooling are all acceptable—use what you’re most efficient with, provided the system is modular enough to add new sources later.
Key extraction targets are the advertised price and precise location for every property; additional attributes such as size or room count can follow once the core pipeline is stable. After collection, the data should be normalised by:
• Standardising all units of measurement (m², Km, etc.)
• Formatting addresses consistently so neighbourhood and street fields align
• Applying a single, uniform set of property-type categories
The workflow must include robust deduplication, resumable checkpoints (in case of network or site interruptions), and a clear audit log of every step. Clean, consolidated data should land in a relational database that is optimised for later market analysis, opportunity detection and future analytical products, with a one-click Excel export for quick ad-hoc reporting.
Deliverables:
1. Source-specific scrapers/crawlers with respectful, throttled requests
2. Normalisation and deduplication pipeline with unit tests
3. Relational database schema (PostgreSQL preferred) storing full change history
4. Command-line or lightweight web interface to run, resume, and monitor jobs
5. Excel/CSV export module
6. Deployment instructions and brief technical documentation
Python, Scrapy, BeautifulSoup, Selenium or comparable tooling are all acceptable—use what you’re most efficient with, provided the system is modular enough to add new sources later.