Databricks Data Engineer for Delta Modelling
Budget: $15 – $35 USD
We are seeking highly skilled databricks data engineer who have experience in building data pipelines and data modelling
in delta tables.Candidate should have a deep understanding of Databricks advanced features, including Unity Catalog, Delta Lake optimizations,
storage efficiency techniques, and security best practices.
Key Responsibilities:
Design and develop scalable, high-performance ETL/ELT pipelines using Databricks and Apache Spark.
Implement Delta Lake best practices, including schema evolution, vacuum, and optimization for efficient data storage and query performance.
Manage security, access control, and data governance using Unity Catalog and other Databricks-native security features.
Optimize query performance by leveraging Z-Order indexing, data skipping, caching, and clustering.
Ensure data quality through automated testing, data validation, and monitoring.
Collaborate with data scientists, analysts, and business teams to deliver reliable and well-governed datasets.
Nice to Have:
Experience with Databricks AI/LLM capabilities, including Databricks Model Serving and Lakehouse AI.
Familiarity with Databricks Agentic Framework for building AI-powered data retrieval and analysis applications.
Experience in integrating LLMs with Databricks, such as vector search, embedding models, and fine-tuning models for enterprise use cases.
Understanding of MLflow for model tracking, deployment, and versioning.
in delta tables.Candidate should have a deep understanding of Databricks advanced features, including Unity Catalog, Delta Lake optimizations,
storage efficiency techniques, and security best practices.
Key Responsibilities:
Design and develop scalable, high-performance ETL/ELT pipelines using Databricks and Apache Spark.
Implement Delta Lake best practices, including schema evolution, vacuum, and optimization for efficient data storage and query performance.
Manage security, access control, and data governance using Unity Catalog and other Databricks-native security features.
Optimize query performance by leveraging Z-Order indexing, data skipping, caching, and clustering.
Ensure data quality through automated testing, data validation, and monitoring.
Collaborate with data scientists, analysts, and business teams to deliver reliable and well-governed datasets.
Nice to Have:
Experience with Databricks AI/LLM capabilities, including Databricks Model Serving and Lakehouse AI.
Familiarity with Databricks Agentic Framework for building AI-powered data retrieval and analysis applications.
Experience in integrating LLMs with Databricks, such as vector search, embedding models, and fine-tuning models for enterprise use cases.
Understanding of MLflow for model tracking, deployment, and versioning.