Data Engineer Needed for UGC Indexation with Next.js and AI Integration
Budget: ₹1,500 – ₹12,500 INR
Project Overview
We are developing an AI-powered search engine focused on User-Generated Content (UGC) related to events, businesses, and articles. The goal is to create a robust backend infrastructure for efficient UGC indexation, leveraging the Morphic repository (https://github.com/miurla/morphic) as a template. The focus will be on data engineering, AI/ML integration, and indexing.
Scope of Work
1. Database Setup:
- Configure and set up databases for storing UGC and metadata.
2. Data Ingestion Pipelines:
- Develop pipelines for collecting data from various sources, including social media and external APIs.
3. Data Processing and Indexing:
- Implement efficient data processing and indexing mechanisms.
4. AI Integration:
- Integrate AI open-source models for generating articles and enhancing content.
5. Performance Optimization:
- Ensure scalability and performance optimization of the backend system.
Deliverables
1. Configured databases for UGC and metadata.
2. Functional data ingestion pipelines.
3. Implemented data processing and indexing mechanisms.
4. Integrated AI models for content generation.
5. Performance-optimized backend system ready for deployment.
Requirements
- Proven experience in data engineering and backend development.
- Proficiency in Python or Node.js.
- Strong knowledge and experience with Next.js.
- Experience with databases like MongoDB, PostgreSQL, or Elasticsearch.
- Expertise in building and optimizing data pipelines.
- Knowledge of AI models and their integration, particularly for NLP and embedding-based databases.
- Familiarity with the Morphic repository or similar projects is a plus.
- Ability to complete the project within the specified 1-month timeframe.
- Excellent communication skills and ability to work collaboratively in a remote team environment
We are developing an AI-powered search engine focused on User-Generated Content (UGC) related to events, businesses, and articles. The goal is to create a robust backend infrastructure for efficient UGC indexation, leveraging the Morphic repository (https://github.com/miurla/morphic) as a template. The focus will be on data engineering, AI/ML integration, and indexing.
Scope of Work
1. Database Setup:
- Configure and set up databases for storing UGC and metadata.
2. Data Ingestion Pipelines:
- Develop pipelines for collecting data from various sources, including social media and external APIs.
3. Data Processing and Indexing:
- Implement efficient data processing and indexing mechanisms.
4. AI Integration:
- Integrate AI open-source models for generating articles and enhancing content.
5. Performance Optimization:
- Ensure scalability and performance optimization of the backend system.
Deliverables
1. Configured databases for UGC and metadata.
2. Functional data ingestion pipelines.
3. Implemented data processing and indexing mechanisms.
4. Integrated AI models for content generation.
5. Performance-optimized backend system ready for deployment.
Requirements
- Proven experience in data engineering and backend development.
- Proficiency in Python or Node.js.
- Strong knowledge and experience with Next.js.
- Experience with databases like MongoDB, PostgreSQL, or Elasticsearch.
- Expertise in building and optimizing data pipelines.
- Knowledge of AI models and their integration, particularly for NLP and embedding-based databases.
- Familiarity with the Morphic repository or similar projects is a plus.
- Ability to complete the project within the specified 1-month timeframe.
- Excellent communication skills and ability to work collaboratively in a remote team environment