Build Data Ingestion Pipeline
Budget: $250 – $750 USD
Senior Data Pipeline / ETL Engineer (FastAPI, PostgreSQL, OpenSearch) – Build MVP Data Ingestion Pipeline for Financial Intelligence Platform
Overview
We are building a financial intelligence infrastructure platform that aggregates global corporate registries, sanctions lists, and ownership data to produce compliance-grade investigative reports.
The GitHub repository, architecture documentation, and core backend services already exist. What we need now is a senior data pipeline engineer who can complete the data ingestion and normalization pipeline so that our MVP feature works perfectly.
The primary MVP workflow is:
Search a person or company → resolve the entity → check sanctions exposure → reconstruct ownership relationships → produce an evidence-backed report.
Your role will be to build the ingestion pipeline that powers this workflow.
This is not a greenfield project.
You will be working inside an existing architecture and repository.
⸻
What You Will Build
You will implement the ETL / data pipeline layer that ingests and prepares structured data for the RealScore screening workflow.
The pipeline must support ingestion of:
• Global sanctions lists
• OFAC SDN
• EU Consolidated
• UN Sanctions
• Corporate registry data
• Beneficial ownership data
• Structured entity datasets
The system must perform:
1. Data Ingestion
Automated ingestion jobs that pull structured datasets and load them into PostgreSQL.
Requirements:
• Idempotent ingestion
• SHA-256 checksum tracking
• Version tracking for data updates
• Retry mechanisms for failed jobs
⸻
2. Data Normalization
Convert raw records into a standardized entity schema.
Examples:
• normalize company names
• remove legal suffixes
• standardize jurisdictions
• normalize identifiers
The output should populate:
• entities
• identifiers
• relationships
• evidence_records
⸻
3. Entity Resolution
Implement a multi-pass entity resolution system:
Pass 1 — Deterministic
• exact identifier matches (LEI, registry IDs)
Pass 2 — Semi-deterministic
• normalized name match
Pass 3 — Probabilistic
• fuzzy matching via OpenSearch
• Jaro / similarity scoring
Goal:
resolve duplicate records across datasets.
⸻
4. Relationship Construction
Build ownership and control relationships:
Examples:
• company → director
• company → shareholder
• entity → sanctioned entity
• entity → related entities
Relationships must be stored so they can be used by the risk engine.
⸻
5. Pipeline Orchestration
Pipeline must support:
• scheduled ingestion jobs
• dependency ordering
• failure recovery
• logging
Suggested tools (already in repo):
• Python
• FastAPI
• PostgreSQL
• Celery / Redis
• OpenSearch
⸻
Expected Output
When a user searches for a person or company:
The system must be able to:
1. Resolve the entity
2. Check sanctions exposure
3. Trace ownership relationships
4. Generate a structured evidence report
This pipeline is the core engine powering that workflow.
⸻
Existing Stack
You will work inside an existing repository that includes:
• FastAPI backend
• PostgreSQL database
• ingestion service skeleton
• normalization schemas
• entity service
• GitHub repo with architecture documentation
You will extend the current pipeline, not rebuild from scratch.
⸻
Required Experience
Minimum requirements:
• 5+ years building production ETL pipelines
• Python data engineering experience
• PostgreSQL data modeling
• Search engines (OpenSearch or Elasticsearch)
• Experience with large structured datasets
Preferred experience:
• sanctions / compliance data
• entity resolution systems
• corporate registry datasets
• financial intelligence or AML systems
⸻
Deliverables
You will deliver:
• fully functional ingestion pipelines
• normalized entity datasets
• entity resolution implementation
• pipeline orchestration
• documentation inside the repo
The pipeline must run via:
make seed
make pipeline
and populate the database correctly.
⸻
Important
This is a serious engineering role, not a simple script task.
We are looking for someone who can think like a data systems architect, not just write quick ETL scripts.
Please include in your proposal:
1. Examples of ETL pipelines you have built
2. Your experience with entity resolution
3. Experience with large data ingestion systems
4. Your GitHub profile
⸻
Budget
Open to fixed price or milestone structure depending on experience.
We prioritize quality and reliability over the lowest bid.
⸻
If you have experience building serious data pipelines and investigative data systems, we would like to hear from you.
All proposals must include GitHub links showing your past work specifically with pipelines.
Overview
We are building a financial intelligence infrastructure platform that aggregates global corporate registries, sanctions lists, and ownership data to produce compliance-grade investigative reports.
The GitHub repository, architecture documentation, and core backend services already exist. What we need now is a senior data pipeline engineer who can complete the data ingestion and normalization pipeline so that our MVP feature works perfectly.
The primary MVP workflow is:
Search a person or company → resolve the entity → check sanctions exposure → reconstruct ownership relationships → produce an evidence-backed report.
Your role will be to build the ingestion pipeline that powers this workflow.
This is not a greenfield project.
You will be working inside an existing architecture and repository.
⸻
What You Will Build
You will implement the ETL / data pipeline layer that ingests and prepares structured data for the RealScore screening workflow.
The pipeline must support ingestion of:
• Global sanctions lists
• OFAC SDN
• EU Consolidated
• UN Sanctions
• Corporate registry data
• Beneficial ownership data
• Structured entity datasets
The system must perform:
1. Data Ingestion
Automated ingestion jobs that pull structured datasets and load them into PostgreSQL.
Requirements:
• Idempotent ingestion
• SHA-256 checksum tracking
• Version tracking for data updates
• Retry mechanisms for failed jobs
⸻
2. Data Normalization
Convert raw records into a standardized entity schema.
Examples:
• normalize company names
• remove legal suffixes
• standardize jurisdictions
• normalize identifiers
The output should populate:
• entities
• identifiers
• relationships
• evidence_records
⸻
3. Entity Resolution
Implement a multi-pass entity resolution system:
Pass 1 — Deterministic
• exact identifier matches (LEI, registry IDs)
Pass 2 — Semi-deterministic
• normalized name match
Pass 3 — Probabilistic
• fuzzy matching via OpenSearch
• Jaro / similarity scoring
Goal:
resolve duplicate records across datasets.
⸻
4. Relationship Construction
Build ownership and control relationships:
Examples:
• company → director
• company → shareholder
• entity → sanctioned entity
• entity → related entities
Relationships must be stored so they can be used by the risk engine.
⸻
5. Pipeline Orchestration
Pipeline must support:
• scheduled ingestion jobs
• dependency ordering
• failure recovery
• logging
Suggested tools (already in repo):
• Python
• FastAPI
• PostgreSQL
• Celery / Redis
• OpenSearch
⸻
Expected Output
When a user searches for a person or company:
The system must be able to:
1. Resolve the entity
2. Check sanctions exposure
3. Trace ownership relationships
4. Generate a structured evidence report
This pipeline is the core engine powering that workflow.
⸻
Existing Stack
You will work inside an existing repository that includes:
• FastAPI backend
• PostgreSQL database
• ingestion service skeleton
• normalization schemas
• entity service
• GitHub repo with architecture documentation
You will extend the current pipeline, not rebuild from scratch.
⸻
Required Experience
Minimum requirements:
• 5+ years building production ETL pipelines
• Python data engineering experience
• PostgreSQL data modeling
• Search engines (OpenSearch or Elasticsearch)
• Experience with large structured datasets
Preferred experience:
• sanctions / compliance data
• entity resolution systems
• corporate registry datasets
• financial intelligence or AML systems
⸻
Deliverables
You will deliver:
• fully functional ingestion pipelines
• normalized entity datasets
• entity resolution implementation
• pipeline orchestration
• documentation inside the repo
The pipeline must run via:
make seed
make pipeline
and populate the database correctly.
⸻
Important
This is a serious engineering role, not a simple script task.
We are looking for someone who can think like a data systems architect, not just write quick ETL scripts.
Please include in your proposal:
1. Examples of ETL pipelines you have built
2. Your experience with entity resolution
3. Experience with large data ingestion systems
4. Your GitHub profile
⸻
Budget
Open to fixed price or milestone structure depending on experience.
We prioritize quality and reliability over the lowest bid.
⸻
If you have experience building serious data pipelines and investigative data systems, we would like to hear from you.
All proposals must include GitHub links showing your past work specifically with pipelines.