Azure - Automating Parallel Processing -- 2
Budget: $2 – $8 USD
Project: Cloud Architecture for Parallel File Processing with Auto-Scaling
Overview
I’m designing a cloud-native architecture to process files uploaded to object storage by automatically launching virtual machines or compute nodes on demand.
The goal is to build a fully automated, parallel, and scalable pipeline, where each uploaded file triggers its own processing workflow — and once the job is completed, all compute resources automatically shut down to minimize costs.
-------------------------------------------------------------------
Concept Flow
File Upload
A user or external system uploads a file to a designated input container in object storage.
Trigger Function (Stage 1)
A serverless function listens for the upload event and sends a message to a message queue, notifying that a new file is ready for processing.
The message includes basic metadata such as filename, location, and job type.
Orchestrator Function (Stage 2)
Another function is triggered by the queue message and:
Reads the file information.
Creates a processing job in a batch or compute service.
Launches a dynamic pool of compute nodes to handle the workload.
Defines the command or script to be executed on each node.
Parallel Processing
The batch service coordinates the execution of multiple files in parallel.
Each node processes one or more files, produces an output dataset, and writes results to an output container in object storage.
Completion and Auto-Shutdown
When all jobs finish, the compute pool automatically scales down to zero active nodes.
Logs and execution results are stored in a separate container for monitoring and auditing.
-------------------------------------------------------------------
Objectives
Build a cloud-native orchestration that runs automatically without manual intervention.
Enable parallel processing with fault tolerance and auto-scaling.
Reduce costs by keeping compute resources off when idle.
Provide transparent logs and performance metrics for traceability.
Current Progress
The core components are conceptually defined:
Object storage with containers for input, output, and logs.
Event-driven function that reacts to file uploads and writes to a message queue.
Concept design for a second function that orchestrates job creation in a batch or scalable compute environment.
-------------------------------------------------------------------
What I’m Looking For
I’m seeking a Cloud / Serverless / Batch Computing specialist who can:
Implement the complete end-to-end pipeline.
Configure the compute pool, job definitions, and scaling logic.
Implement both serverless functions (trigger and orchestrator).
Standardize the infrastructure using Infrastructure as Code (Terraform or Bicep).
Deliver full documentation and ensure the entire system is visible and manageable through the cloud console.
-------------------------------------------------------------------
Preferred Stack
Batch or compute orchestration service, serverless functions, object storage, message queue, and secret manager.
Python or .NET for function logic.
Terraform or Bicep for infrastructure as code.
Monitoring and logging tools for observability.
-------------------------------------------------------------------
Deliverables
A complete and well-documented solution that:
Processes files concurrently and efficiently.
Automatically deallocates resources after execution.
Is reproducible and observable through the cloud portal.
Includes full logs, metrics, and error handling.
---------------------------------------------------------------------------------------------------------------------
I’m looking for someone to work on an hourly basis (around 4 hours per day). The goal is to collaborate closely throughout the process — I want to understand the architecture and decisions being made, not just receive a final deliverable. Daily updates and short explanations of progress, reasoning, and implementation choices are required. This is a hands-on, transparent collaboration, not a “build and disappear” type of project.
Overview
I’m designing a cloud-native architecture to process files uploaded to object storage by automatically launching virtual machines or compute nodes on demand.
The goal is to build a fully automated, parallel, and scalable pipeline, where each uploaded file triggers its own processing workflow — and once the job is completed, all compute resources automatically shut down to minimize costs.
-------------------------------------------------------------------
Concept Flow
File Upload
A user or external system uploads a file to a designated input container in object storage.
Trigger Function (Stage 1)
A serverless function listens for the upload event and sends a message to a message queue, notifying that a new file is ready for processing.
The message includes basic metadata such as filename, location, and job type.
Orchestrator Function (Stage 2)
Another function is triggered by the queue message and:
Reads the file information.
Creates a processing job in a batch or compute service.
Launches a dynamic pool of compute nodes to handle the workload.
Defines the command or script to be executed on each node.
Parallel Processing
The batch service coordinates the execution of multiple files in parallel.
Each node processes one or more files, produces an output dataset, and writes results to an output container in object storage.
Completion and Auto-Shutdown
When all jobs finish, the compute pool automatically scales down to zero active nodes.
Logs and execution results are stored in a separate container for monitoring and auditing.
-------------------------------------------------------------------
Objectives
Build a cloud-native orchestration that runs automatically without manual intervention.
Enable parallel processing with fault tolerance and auto-scaling.
Reduce costs by keeping compute resources off when idle.
Provide transparent logs and performance metrics for traceability.
Current Progress
The core components are conceptually defined:
Object storage with containers for input, output, and logs.
Event-driven function that reacts to file uploads and writes to a message queue.
Concept design for a second function that orchestrates job creation in a batch or scalable compute environment.
-------------------------------------------------------------------
What I’m Looking For
I’m seeking a Cloud / Serverless / Batch Computing specialist who can:
Implement the complete end-to-end pipeline.
Configure the compute pool, job definitions, and scaling logic.
Implement both serverless functions (trigger and orchestrator).
Standardize the infrastructure using Infrastructure as Code (Terraform or Bicep).
Deliver full documentation and ensure the entire system is visible and manageable through the cloud console.
-------------------------------------------------------------------
Preferred Stack
Batch or compute orchestration service, serverless functions, object storage, message queue, and secret manager.
Python or .NET for function logic.
Terraform or Bicep for infrastructure as code.
Monitoring and logging tools for observability.
-------------------------------------------------------------------
Deliverables
A complete and well-documented solution that:
Processes files concurrently and efficiently.
Automatically deallocates resources after execution.
Is reproducible and observable through the cloud portal.
Includes full logs, metrics, and error handling.
---------------------------------------------------------------------------------------------------------------------
I’m looking for someone to work on an hourly basis (around 4 hours per day). The goal is to collaborate closely throughout the process — I want to understand the architecture and decisions being made, not just receive a final deliverable. Daily updates and short explanations of progress, reasoning, and implementation choices are required. This is a hands-on, transparent collaboration, not a “build and disappear” type of project.
Related categories:
Python
.NET
Data Processing
Cloud Computing
Azure
VMware
Parallel Processing
Documentation
Microsoft Azure
Terraform