Senior AWS Databricks Administration

I run an AWS-hosted data platform and need an experienced Databricks administrator to take full ownership of the workspace. The immediate mandate is to keep our clusters healthy, our jobs running on time, and our costs under control while squeezing every bit of performance out of Spark.

Your day-to-day work will revolve around three core areas:

• Cluster setup and management – size, autoscale, libraries, networking, security, and integration with our existing Terraform modules.
• Job scheduling and monitoring – creating dependable workflows, wiring alerts into CloudWatch/Datadog, and documenting run-books so the team can diagnose issues quickly.
• Performance tuning and optimization – reviewing Spark configurations, rewriting inefficient code paths when needed, and leaving each pipeline faster and cheaper than you found it.

Our stack is mostly AWS (VPC-peered to Databricks), Terraform, Python, Bash and a sprinkling of Ansible. You are free to introduce additional tooling as long as it is automated, version-controlled, and auditable.

I am looking for someone who can work independently, communicate progress proactively, and back up recommendations with data. If that sounds like you, let’s talk through the first milestone: standing up a production-grade cluster configuration that passes load testing and security review.