Middleware Job Aggregation Platform
Budget: ₹12,500 – ₹37,500 INR
Job Posting: Middleware Job Aggregation Platform (Using Apache NiFi)
Title: Middleware Job Aggregation & Normalization Layer using Apache NiFi
Overview:
We are building a data ingestion and normalization platform using Apache NiFi to consolidate multiple third-party job data sources (XML feeds, REST APIs, FTP-based exports). The goal is to standardize all incoming job listings into a uniform format that can be inserted into our backend system (Laravel + MySQL).
Technical Scope:
Input sources:
XML job listing feeds (5 different ATS systems)
REST API job feeds (some with OAuth tokens or keys)
FTP drops with recurring XML/CSV files
Normalization layer:
Apache NiFi will be used to parse, map, and transform various data structures
Each source will require a unique sub-flow, followed by output to a single schema
Output format:
Unified JSON or DB-ready schema (we will provide final spec)
Output must either:
(A) Write to our MySQL DB directly using PutSQL, or
(B) POST normalized JSON to our Laravel backend endpoint (e.g., /api/normalized-jobs)
Volume: 5+ XML/API sources initially
Estimated 1,000–5,000 jobs per day per feed
Error handling & logging:
Failed records must be logged with error reasons
Retry logic for API or FTP failures
Authentication support:
OAuth2 tokens for some REST APIs
Basic Auth / key-based FTP and HTTP access
Environment:
Middleware: Apache NiFi (must be containerized via Docker or orchestrated on server)
Backend: Laravel (PHP)
Database: MySQL
Hosting: Cloud VPS (SSH access can be granted for setup)
Deliverables:
NiFi flow templates (one per feed/source)
Standardized mapping logic (e.g., Workday → Internal Schema)
Output module to Laravel-compatible endpoint or MySQL
Robust logging / error capture
Deployment instructions (Docker Compose preferred)
(Optional) Admin flow visualization panel (if NiFi UI access isn’t public)
Timeline:
Must include weekly update/demo
Requirements for Proposal:
Please include the following in your quote:
Examples of prior Apache NiFi work (especially involving XML/API normalization)
Expected project cost (fixed or hourly with estimated hours)
Timeline to complete MVP version
Any recommended enhancements (security, scale, etc.)
Bonus:
Experience with ATS XML feeds (e.g., iCIMS, Taleo, Workday)
Knowledge of Laravel APIs and MySQL normalization logic
Experience deploying NiFi in production
Title: Middleware Job Aggregation & Normalization Layer using Apache NiFi
Overview:
We are building a data ingestion and normalization platform using Apache NiFi to consolidate multiple third-party job data sources (XML feeds, REST APIs, FTP-based exports). The goal is to standardize all incoming job listings into a uniform format that can be inserted into our backend system (Laravel + MySQL).
Technical Scope:
Input sources:
XML job listing feeds (5 different ATS systems)
REST API job feeds (some with OAuth tokens or keys)
FTP drops with recurring XML/CSV files
Normalization layer:
Apache NiFi will be used to parse, map, and transform various data structures
Each source will require a unique sub-flow, followed by output to a single schema
Output format:
Unified JSON or DB-ready schema (we will provide final spec)
Output must either:
(A) Write to our MySQL DB directly using PutSQL, or
(B) POST normalized JSON to our Laravel backend endpoint (e.g., /api/normalized-jobs)
Volume: 5+ XML/API sources initially
Estimated 1,000–5,000 jobs per day per feed
Error handling & logging:
Failed records must be logged with error reasons
Retry logic for API or FTP failures
Authentication support:
OAuth2 tokens for some REST APIs
Basic Auth / key-based FTP and HTTP access
Environment:
Middleware: Apache NiFi (must be containerized via Docker or orchestrated on server)
Backend: Laravel (PHP)
Database: MySQL
Hosting: Cloud VPS (SSH access can be granted for setup)
Deliverables:
NiFi flow templates (one per feed/source)
Standardized mapping logic (e.g., Workday → Internal Schema)
Output module to Laravel-compatible endpoint or MySQL
Robust logging / error capture
Deployment instructions (Docker Compose preferred)
(Optional) Admin flow visualization panel (if NiFi UI access isn’t public)
Timeline:
Must include weekly update/demo
Requirements for Proposal:
Please include the following in your quote:
Examples of prior Apache NiFi work (especially involving XML/API normalization)
Expected project cost (fixed or hourly with estimated hours)
Timeline to complete MVP version
Any recommended enhancements (security, scale, etc.)
Bonus:
Experience with ATS XML feeds (e.g., iCIMS, Taleo, Workday)
Knowledge of Laravel APIs and MySQL normalization logic
Experience deploying NiFi in production