Scalable Real-Time Data Backend
Budget: €12 – €18 EUR
I need a robust backend architecture that can ingest and process streamed sensor data in real time. The system should be highly scalable, resilient to spikes, and designed for efficient data processing rather than heavy graphics or UI work.
Core expectations
• Real-time pipeline that consumes live sensor streams, cleans and enriches them, then exposes the results through well-documented APIs.
• Horizontal scalability—spinning up additional instances automatically as throughput grows.
• Low-latency processing so downstream consumers receive near-instant updates.
• Fault tolerance and graceful recovery when individual nodes or message brokers fail.
• Clear, human-readable documentation covering setup, deployment, and code structure.
Preferred stack (flexible if you can justify alternatives)
• Cloud services such as AWS (Kinesis, Lambda, DynamoDB) or GCP equivalents.
• Message streaming—Kafka, Kinesis, or Pulsar.
• Containerization with Docker and orchestration via Kubernetes or managed serverless.
• Written in a performant language you’re comfortable maintaining (Go, Node.js, Python, or similar).
Deliverables
1. Source code with clean, commented modules.
2. Infrastructure-as-Code templates (Terraform, CloudFormation, or Pulumi).
3. Read-me and architectural diagram detailing data flow and scaling strategy.
4. A short video or live walkthrough demonstrating the system handling sample sensor streams in real time.
If you have proven experience building real-time data backends and can think several steps ahead in terms of scaling and resilience, I’d love to see your proposal and a couple of relevant project links or repos.
Core expectations
• Real-time pipeline that consumes live sensor streams, cleans and enriches them, then exposes the results through well-documented APIs.
• Horizontal scalability—spinning up additional instances automatically as throughput grows.
• Low-latency processing so downstream consumers receive near-instant updates.
• Fault tolerance and graceful recovery when individual nodes or message brokers fail.
• Clear, human-readable documentation covering setup, deployment, and code structure.
Preferred stack (flexible if you can justify alternatives)
• Cloud services such as AWS (Kinesis, Lambda, DynamoDB) or GCP equivalents.
• Message streaming—Kafka, Kinesis, or Pulsar.
• Containerization with Docker and orchestration via Kubernetes or managed serverless.
• Written in a performant language you’re comfortable maintaining (Go, Node.js, Python, or similar).
Deliverables
1. Source code with clean, commented modules.
2. Infrastructure-as-Code templates (Terraform, CloudFormation, or Pulumi).
3. Read-me and architectural diagram detailing data flow and scaling strategy.
4. A short video or live walkthrough demonstrating the system handling sample sensor streams in real time.
If you have proven experience building real-time data backends and can think several steps ahead in terms of scaling and resilience, I’d love to see your proposal and a couple of relevant project links or repos.