Azure DataBricks Engineer

Quick Facts

• Azure Databricks DataBricks Engineer
• Focus: migrations, data processing, and optimization on Databricks
• Work context: projects mostly for US (typically limited overlap until ~18:00), often short-term with high likelihood of extensions
• Industries: medical and financial, but open to other sectors

Description

The role involves designing, implementing, and maintaining scalable Data Lakehouse solutions in Databricks (Delta Lake) and building/optimizing ETL/ELT pipelines for both batch and streaming processing. You will participate in migrations (e.g., YellowBrick → Databricks), refactor legacy logic (SSIS/Boomi) into Spark/Python solutions, and implement governance and security using Unity Catalog, RBAC, and Terraform. You will also monitor and tune performance, automate processes to meet SLA (including a four-hour data refresh window), and collaborate with data scientists, data engineers, and architects.

Responsibilities

• Design, deploy, and maintain scalable Data Lakehouse in Databricks and Delta Lake using Bronze/Silver/Gold layers
• Create and optimize ETL/ELT pipelines in PySpark and DLT for batch and streaming
• Participate in data migrations (e.g., YellowBrick → Databricks)
• Refactor SSIS and Boomi logic into Spark/Python solutions
• Implement governance and security: Unity Catalog, RBAC, and Terraform
• Monitor, tune performance, and automate to meet SLA requirements (including a 4-hour refresh window)
• Collaborate with Data Scientists, Data Engineers, and Architects
• Help select tools and integrate with Smartsheet and SailPoint

Requirements

• 8+ years of experience as a Data Engineer or in related roles, including transformation and migration projects
• 3–4+ years of experience with Databricks and Apache Spark: clusters, workspaces, security, migrations, ETL, integrations, PySpark, and query optimization
• Very good Python knowledge
• Delta Lake and Unity Catalog experience (or similar governance solutions)
• Delta Live Tables experience (or similar)
• Azure experience (e.g., Data Factory, Synapse, Logic Apps, Data Lake) and/or AWS (e.g., Redshift, Athena, Glue)
• SQL: schema design, query optimization, and implementing business logic
• Independence and initiative
• English enabling smooth communication within the team

Benefits

• Stable ongoing demand for specialists
• Mostly short-term projects with a high probability of extensions and the chance to propose new topics after project completion