Build Structured Web Crawling System

# Python Developer (Scrapy / FastAPI / Docker) for Crawling Project

We are looking for an experienced Python developer to build a structured web crawling system.

This is not a large corporate project. It is a technically clean, well-defined system with a long-term perspective. We value clean architecture, stability, and understandable code.

The system will process approximately 30,000 profiles per week and will be connected to an existing Laravel-based admin dashboard.

---

## Responsibilities

* Develop a crawler using **Scrapy**
* Build a small **FastAPI interface** (start / stop / status / statistics)
* Integrate with **MariaDB**
* Implement a media pipeline: download → local cache → upload to Wasabi (S3)
* Implement hash-based change detection
* Implement soft-delete logic with a 3-month lifecycle
* Dockerize the system
* Provide short technical documentation

---

## Environment

* Python 3.11+
* Scrapy
* FastAPI
* MariaDB
* Redis
* Docker (running on an Unraid server)
* Wasabi (S3-compatible storage)
* Existing Laravel admin panel (used to control the crawler)

---

## What We’re Looking For

* Experience with web crawling projects
* Clean and structured coding style
* Understanding of API architecture
* Docker experience
* Ability to work independently

---

## Project Scope

* Initial implementation: approx. 4–6 weeks
* Remote work
* Potential long-term collaboration (maintenance and future extensions)

---

## Application

Please include:

* Short introduction
* Relevant references (crawling / API projects)
* Availability
* Hourly rate or project estimate

We are looking for someone pragmatic, technically solid, and interested in building a maintainable system for long-term use.