Python Server for ClickHouse Data Pipeline
Budget: $30 – $250 USD
### **1. Introduction**
I am seeking proposals from qualified freelancers or agencies to develop a Python-based server/service for a data pipeline project. This project involves extracting data from a ClickHouse database, cleaning and processing the data, generating vectors, using these vectors with a pre-trained PyTorch model to generate a score, and updating the database with the results.
I am looking for a partner who can deliver a robust, scalable, and efficient solution. The service should run continuously and process data in configurable batch size and be designed for both reliability and maintainability.
---
### **2. Project Overview**
The goal of the project is to create a data pipeline that runs continuously to and processes batch to:
1. **Extract** data from ClickHouseDB.
2. **Process** and clean the extracted data. (Code/steps available in Jupyter Notebook)
3. **Generate** vectors from the processed data. (Code/steps available in Jupyter Notebook)
4. **Use** a pre-trained PyTorch model to compute scores based on the vectors. (Code/steps available in Jupyter Notebook)
5. **Update** ClickHouseDB with the computed scores.
---
### **3. Additional Considerations**
#### **Architecture**
- Create multiple Python classes that way all the steps are generic and can be changed in the future. i.e., the database can be changed from ClickhouseDB to some other DB. Data processing and vector generation can be changed if the model is updated.
#### **Logging & Monitoring**
- Implement logging to track the status of the service, errors, and performance metrics.
- Provide a basic monitoring system to alert on any potential issues, such as connection failures, data inconsistencies, or model performance degradation.
---
### **4. Deliverables**
The key deliverables of this project include:
1. A Python-based service that continuously reads data from ClickHouseDB.
2. Integration with a pre-trained PyTorch model for scoring.
3. Process for updating the ClickHouseDB with the computed scores.
4. Complete documentation of the solution, including installation instructions, architecture overview, and code comments.
5. Logs and monitoring setup for service reliability and error detection.
#### Given:
1. Data cleaning and processing routines.
2. PyTorch vector generation logic.
---
### **5. Milestones**
We encourage freelancers to propose milestones for the project. However, we suggest the following tentative milestones to guide your proposal:
#### **Milestone 1: Project Setup & Initial Data Extraction**
- Weight: 10 %
#### **Milestone 2: Data Cleaning & Processing and Vector Generation**
- Weight: 30 %
#### **Milestone 3: Model Integration & Scoring**
- Weight: 30 %
#### **Milestone 4: Update Database & Final Testing**
- Weight: 20 %
#### **Milestone 5: Logging, Monitoring, and Final Documentation**
- Weight: 10 %
---
### **6. Evaluation Criteria**
Proposals will be evaluated based on:
- **Technical Expertise:** Proven experience in developing data pipelines, working with Python, ClickHouse, and PyTorch.
- **Proposed Solution:** The completeness and thoughtfulness of the proposed solution.
- **Milestones & Timeline:** Clear and realistic milestones and timelines for project completion.
- **Cost:** Competitive pricing aligned with project scope.
- **Past Experience:** Relevant projects and client feedback.
I am seeking proposals from qualified freelancers or agencies to develop a Python-based server/service for a data pipeline project. This project involves extracting data from a ClickHouse database, cleaning and processing the data, generating vectors, using these vectors with a pre-trained PyTorch model to generate a score, and updating the database with the results.
I am looking for a partner who can deliver a robust, scalable, and efficient solution. The service should run continuously and process data in configurable batch size and be designed for both reliability and maintainability.
---
### **2. Project Overview**
The goal of the project is to create a data pipeline that runs continuously to and processes batch to:
1. **Extract** data from ClickHouseDB.
2. **Process** and clean the extracted data. (Code/steps available in Jupyter Notebook)
3. **Generate** vectors from the processed data. (Code/steps available in Jupyter Notebook)
4. **Use** a pre-trained PyTorch model to compute scores based on the vectors. (Code/steps available in Jupyter Notebook)
5. **Update** ClickHouseDB with the computed scores.
---
### **3. Additional Considerations**
#### **Architecture**
- Create multiple Python classes that way all the steps are generic and can be changed in the future. i.e., the database can be changed from ClickhouseDB to some other DB. Data processing and vector generation can be changed if the model is updated.
#### **Logging & Monitoring**
- Implement logging to track the status of the service, errors, and performance metrics.
- Provide a basic monitoring system to alert on any potential issues, such as connection failures, data inconsistencies, or model performance degradation.
---
### **4. Deliverables**
The key deliverables of this project include:
1. A Python-based service that continuously reads data from ClickHouseDB.
2. Integration with a pre-trained PyTorch model for scoring.
3. Process for updating the ClickHouseDB with the computed scores.
4. Complete documentation of the solution, including installation instructions, architecture overview, and code comments.
5. Logs and monitoring setup for service reliability and error detection.
#### Given:
1. Data cleaning and processing routines.
2. PyTorch vector generation logic.
---
### **5. Milestones**
We encourage freelancers to propose milestones for the project. However, we suggest the following tentative milestones to guide your proposal:
#### **Milestone 1: Project Setup & Initial Data Extraction**
- Weight: 10 %
#### **Milestone 2: Data Cleaning & Processing and Vector Generation**
- Weight: 30 %
#### **Milestone 3: Model Integration & Scoring**
- Weight: 30 %
#### **Milestone 4: Update Database & Final Testing**
- Weight: 20 %
#### **Milestone 5: Logging, Monitoring, and Final Documentation**
- Weight: 10 %
---
### **6. Evaluation Criteria**
Proposals will be evaluated based on:
- **Technical Expertise:** Proven experience in developing data pipelines, working with Python, ClickHouse, and PyTorch.
- **Proposed Solution:** The completeness and thoughtfulness of the proposed solution.
- **Milestones & Timeline:** Clear and realistic milestones and timelines for project completion.
- **Cost:** Competitive pricing aligned with project scope.
- **Past Experience:** Relevant projects and client feedback.