Python Data Pipeline to Solr
Budget: ₹600 – ₹1,500 INR
I need a clean, well-tested Python pipeline that listens to our MongoDB (DocumentDB) product collection, transforms each incoming SKU document, and pushes the result into Apache Solr so the search index is always in lock-step with the source data.
What the transform must do
• Enrich taxonomy on two fronts: category as well as attribute level.
• Keep only English content for now, but structure the code so Spanish or French could be switched on with minimal effort.
• Filter out the fields we don’t want surfaced—specifically the SKU number, product description and price—while leaving all other data intact.
• Merge duplicates that share the same SKU logic we will provide.
Runtime behaviour
The service should behave like a sink connector: near-real-time change-stream consumption from MongoDB, conversion, enrichment, de-duplication, then a commit into Solr using the bulk API. Failures must retry gracefully and never poison the queue.
Tech stack expectations
Python 3.10+, pymongo (or Amazon DocumentDB compatible driver), requests or Solr-py for indexing, and preferably asyncio so we do not block on network calls.
Deliverables
1. Production-ready Python source code with clear module separation.
2. Unit tests covering the enrichment, filtering and deduplication logic.
3. A README that explains local setup (Docker Compose for MongoDB + Solr is fine), environment variables, and deployment steps.
4. Sample configuration file showing where taxonomy mappings, language choices and field filters are declared.
Acceptance criteria
• End-to-end run inserts, updates and deletes in MongoDB and reflects them in Solr within the agreed latency.
• All English SKUs appear; non-English documents are ignored.
• Filtered fields never reach Solr.
• Duplicate SKUs index only once.
If you have prior experience wiring MongoDB change streams to Solr or have built similar ETL connectors, I’d love to see a short code snippet or link in your proposal.
What the transform must do
• Enrich taxonomy on two fronts: category as well as attribute level.
• Keep only English content for now, but structure the code so Spanish or French could be switched on with minimal effort.
• Filter out the fields we don’t want surfaced—specifically the SKU number, product description and price—while leaving all other data intact.
• Merge duplicates that share the same SKU logic we will provide.
Runtime behaviour
The service should behave like a sink connector: near-real-time change-stream consumption from MongoDB, conversion, enrichment, de-duplication, then a commit into Solr using the bulk API. Failures must retry gracefully and never poison the queue.
Tech stack expectations
Python 3.10+, pymongo (or Amazon DocumentDB compatible driver), requests or Solr-py for indexing, and preferably asyncio so we do not block on network calls.
Deliverables
1. Production-ready Python source code with clear module separation.
2. Unit tests covering the enrichment, filtering and deduplication logic.
3. A README that explains local setup (Docker Compose for MongoDB + Solr is fine), environment variables, and deployment steps.
4. Sample configuration file showing where taxonomy mappings, language choices and field filters are declared.
Acceptance criteria
• End-to-end run inserts, updates and deletes in MongoDB and reflects them in Solr within the agreed latency.
• All English SKUs appear; non-English documents are ignored.
• Filtered fields never reach Solr.
• Duplicate SKUs index only once.
If you have prior experience wiring MongoDB change streams to Solr or have built similar ETL connectors, I’d love to see a short code snippet or link in your proposal.