Kafka Data Pipeline Integration

Requirement :
Job Title: Kafka Developer – Data Streaming & CDC Specialist
Role Overview
Seeking a highly skilled Kafka Developer with deep expertise in Kafka KSQL Streams-based transformations and hands-on experience working with Oracle, MS SQL, and PostgreSQL databases—especially in environments with non-standard schemas (e.g., tables without primary keys, with/without constraints, and foreign keys). The ideal candidate will be responsible for designing and implementing scalable, real-time data pipelines and ensuring robust CDC (Change Data Capture) mechanisms for incremental data replication.
Key Responsibilities
Design and implement Kafka Streams and KSQL-based transformations for real-time data processing.
Develop and maintain CDC pipelines using Kafka Connect, Debezium, or custom connectors.
Handle initial data loads from large relational datasets via manual exports and ensure seamless transition to incremental CDC.
Work with complex relational schemas, including:
Tables without primary keys
Tables with/without constraints
Foreign key relationships
Optimize data ingestion and transformation pipelines for performance, reliability, and scalability.
Collaborate with data architects, DBAs, and application teams to ensure data integrity and consistency.
Document technical designs, data flow diagrams, and operational procedures.
Communicate effectively with cross-functional teams and stakeholders.
Required Skills and Experience
Strong hands-on experience with Apache Kafka, Kafka Streams, and KSQL.
Proficiency in Kafka Connect and CDC tools (Debezium, Confluent).
Deep understanding of Oracle, MS SQL Server, and PostgreSQL internals and schema design.
Experience handling non-standard table structures and resolving challenges in CDC replication.
Familiarity with manual data export/import strategies and their integration into streaming pipelines.
Strong knowledge of data serialization formats (Avro, JSON, Protobuf).
Proficient in Java or Scala for custom Kafka development.
Excellent communication skills—both written and verbal.
Preferred Qualifications
Experience with schema registry, data governance, and data quality frameworks.
Familiarity with CI/CD pipelines, GitOps, and containerized deployments (Docker, Kubernetes).
Prior experience in data architecture or data platform engineering roles.