Projeto RPA + Web Scraping + Banco de Dados + Telas Web
Budget: $15 – $25 USD
I am putting together a full stack that automates the collection of public-website content and turns it into a reliable, queryable data service. The core flow looks like this:
• RPA bots kick things off by logging in where needed, navigating multi-step flows, and handling any upload/download or reconciliation steps that keep the pipeline healthy.
• A headless Playwright crawler then gathers text and image assets from those public sites. I already lean on Cheerio for DOM parsing, so the extraction layer should normalise results into a consistent shape before they ever reach storage.
• All harvested records need to land in PostgreSQL first (MySQL as a fallback). Please wire in flyway-style schema versioning, row-level auditing, and soft-delete tracking so nothing is lost.
• On top of the database, expose REST and GraphQL endpoints for ingestion and querying, plus outbound webhooks that notify external systems whenever new data drops or jobs fail.
• Finally, deliver a small web dashboard that surfaces queue depth, job status, success rates, and SLA metrics; lets me run ad-hoc queries; and offers CSV/JSON exports.
Non-negotiables:
– LGPD compliance: encryption in transit and at rest, data minimisation/masking, and full consent/audit trails.
– Observability: structured logs, metrics, and basic alert hooks.
– Stability and scale: the system should support horizontal task workers without manual tweaks.
I’ll consider the project complete when I can spin up the stack with a single command, point it at a list of public URLs, watch jobs flow through the dashboard, query the normalised text and image metadata over the API, and see every security control in place with passing tests and documentation.
• RPA bots kick things off by logging in where needed, navigating multi-step flows, and handling any upload/download or reconciliation steps that keep the pipeline healthy.
• A headless Playwright crawler then gathers text and image assets from those public sites. I already lean on Cheerio for DOM parsing, so the extraction layer should normalise results into a consistent shape before they ever reach storage.
• All harvested records need to land in PostgreSQL first (MySQL as a fallback). Please wire in flyway-style schema versioning, row-level auditing, and soft-delete tracking so nothing is lost.
• On top of the database, expose REST and GraphQL endpoints for ingestion and querying, plus outbound webhooks that notify external systems whenever new data drops or jobs fail.
• Finally, deliver a small web dashboard that surfaces queue depth, job status, success rates, and SLA metrics; lets me run ad-hoc queries; and offers CSV/JSON exports.
Non-negotiables:
– LGPD compliance: encryption in transit and at rest, data minimisation/masking, and full consent/audit trails.
– Observability: structured logs, metrics, and basic alert hooks.
– Stability and scale: the system should support horizontal task workers without manual tweaks.
I’ll consider the project complete when I can spin up the stack with a single command, point it at a list of public URLs, watch jobs flow through the dashboard, query the normalised text and image metadata over the API, and see every security control in place with passing tests and documentation.