Spark Data Engineer Needed ASAP

I’m looking for a seasoned data engineer (ideally 4–13 years in the field) who can jump straight into an active project and get production-ready code running fast. Our stack centres on Apache Spark for both batch and streaming workloads, with Hive as the primary metastore. You’ll be writing and optimising pipelines in either Python, Scala or Java—whichever you are most effective with—while orchestrating everything in Apache Airflow, which is already in place.

Day-to-day work will involve:
• Designing, coding and unit-testing Spark jobs that read from Kafka/Solace, land in S3 or MinIO, and publish to Lakehouse tables in Iceberg, Delta Lake or Hudi.
• Building SQL-based transformations in DBT and wiring them into our Airflow DAGs.
• Containerising your work with Docker and deploying it to Kubernetes alongside the rest of the platform.

You’ll have direct access to the existing clusters, CI/CD pipeline and monitoring dashboards; I expect you to commit high-quality, documented code, follow our Git flow, and deliver an initial working pipeline within the first week so we can start validating data downstream.

Because the need is urgent, please respond only if you can begin immediately and dedicate enough time to hit the ground running. Let me know your relevant Spark projects, the language you prefer, and when you can start.