Custom Data Collection Platform
Budget: $5,000 – $10,000 USD
This Statement of Work (SOW) defines the scope of work for the Data Collection Platform (DCP) project based on the attached documents. All functional requirements, non-functional requirements, architecture, system boundaries, technical standards, APIs, internal gRPC contracts, data models, Control Plane modules, collector worker requirements, quota and tenant isolation, deduplication, retention, security, observability, deployment, .NET SDK, testing strategy, and acceptance criteria shall be governed by DCP_SRS.md. The Contractor shall design, develop, test, deploy, and hand over the DCP as an on-premise, async job-based data collection platform that accepts collection tasks from downstream applications, dispatches tasks to source-specific collectors, temporarily stores collected metadata and raw content, exposes task lifecycle and item retrieval APIs, and supports the required source types including websites, forums, libraries, Facebook, TikTok, YouTube, X/Twitter, and Truth Social.
The delivery roadmap, implementation phases, phase dependencies, deliverables, testing gates, acceptance gates, and handover milestones shall be governed by DCP_Delivery_Plan.md. Phase 1 shall be treated as a capability-proving phase for the Contractor, covering the complete platform infrastructure, full Control Plane capabilities, deployment foundation, observability, .NET SDK baseline, and the production-ready Websites collector. If the Contractor fails to deliver Phase 1 in accordance with the agreed acceptance criteria and handover conditions, the Contractor shall refund to the Customer all amounts paid for Phase 1, unless otherwise agreed in writing by both parties. Subject to successful acceptance of Phase 1, the Contractor shall proceed with subsequent phases for the YouTube, Facebook, X/Twitter, Truth Social, TikTok, Forums, and Libraries collectors.
The delivery roadmap, implementation phases, phase dependencies, deliverables, testing gates, acceptance gates, and handover milestones shall be governed by DCP_Delivery_Plan.md. Phase 1 shall be treated as a capability-proving phase for the Contractor, covering the complete platform infrastructure, full Control Plane capabilities, deployment foundation, observability, .NET SDK baseline, and the production-ready Websites collector. If the Contractor fails to deliver Phase 1 in accordance with the agreed acceptance criteria and handover conditions, the Contractor shall refund to the Customer all amounts paid for Phase 1, unless otherwise agreed in writing by both parties. Subject to successful acceptance of Phase 1, the Contractor shall proceed with subsequent phases for the YouTube, Facebook, X/Twitter, Truth Social, TikTok, Forums, and Libraries collectors.
Related categories:
Python
Web Scraping
Docker
Web Development
Full Stack Development
NoSQL
Relational Databases
REST API