SIEM Orchestrator Layer Development
Budget: $250 – $750 USD
Project Overview
The goal is to develop a proprietary Orchestrator Layer for a custom SIEM system. The system will use Vector.dev for data collection and ClickHouse for storage. The developer needs to build the "brain" that connects these components, manages detection rules, and provides an API for the UI.
Module 1: Data Ingestion & Schema (The Gateway)
Task: Create a FastAPI (Python) or Gin (Golang) service that acts as an intermediary between Vector and ClickHouse.
Requirements:
Implement an endpoint to receive JSON logs from Vector.
Data Normalization: Map incoming logs to a unified schema (ECS-based: timestamp, event_id, source_ip, user, action).
Buffering: Implement async writing to ClickHouse to handle high EPS (Events Per Second).
Module 2: Detection Engine (The Brain)
Task: Build a rule-based engine that scans ClickHouse data for security threats.
Requirements:
Rule Support: The engine must read rules from YAML files (Sigma-inspired).
Query Generator: Convert YAML logic into optimized SQL queries for ClickHouse.
Scheduler: Run specific checks every X minutes (e.g., "Check for 5 failed logins in 1 min").
Module 3: Alert & Incident Management
Task: Create a system to store and notify about triggered rules.
Requirements:
Store "Incidents" in a separate PostgreSQL or SQLite database.
Integrate Webhooks (for Telegram/Slack notifications).
Status management: (New, In Progress, Closed, False Positive).
Technical Stack (Required)
Backend: Python (FastAPI/Pydantic) OR Golang.
Database: ClickHouse (Logs), PostgreSQL (Metadata/Rules).
Containerization: Docker & Docker Compose for the entire stack.
Deliverables & Intellectual Property (IP)
Full Source Code: All custom-written scripts and configurations.
Documentation: API documentation (Swagger/ReDoc) and setup guide.
Ownership: Crucial Clause: "All code developed under this project is a 'Work Made for Hire'. The Buyer (Me) retains 100% of the Intellectual Property rights and Copyright."
The goal is to develop a proprietary Orchestrator Layer for a custom SIEM system. The system will use Vector.dev for data collection and ClickHouse for storage. The developer needs to build the "brain" that connects these components, manages detection rules, and provides an API for the UI.
Module 1: Data Ingestion & Schema (The Gateway)
Task: Create a FastAPI (Python) or Gin (Golang) service that acts as an intermediary between Vector and ClickHouse.
Requirements:
Implement an endpoint to receive JSON logs from Vector.
Data Normalization: Map incoming logs to a unified schema (ECS-based: timestamp, event_id, source_ip, user, action).
Buffering: Implement async writing to ClickHouse to handle high EPS (Events Per Second).
Module 2: Detection Engine (The Brain)
Task: Build a rule-based engine that scans ClickHouse data for security threats.
Requirements:
Rule Support: The engine must read rules from YAML files (Sigma-inspired).
Query Generator: Convert YAML logic into optimized SQL queries for ClickHouse.
Scheduler: Run specific checks every X minutes (e.g., "Check for 5 failed logins in 1 min").
Module 3: Alert & Incident Management
Task: Create a system to store and notify about triggered rules.
Requirements:
Store "Incidents" in a separate PostgreSQL or SQLite database.
Integrate Webhooks (for Telegram/Slack notifications).
Status management: (New, In Progress, Closed, False Positive).
Technical Stack (Required)
Backend: Python (FastAPI/Pydantic) OR Golang.
Database: ClickHouse (Logs), PostgreSQL (Metadata/Rules).
Containerization: Docker & Docker Compose for the entire stack.
Deliverables & Intellectual Property (IP)
Full Source Code: All custom-written scripts and configurations.
Documentation: API documentation (Swagger/ReDoc) and setup guide.
Ownership: Crucial Clause: "All code developed under this project is a 'Work Made for Hire'. The Buyer (Me) retains 100% of the Intellectual Property rights and Copyright."