Senior Databricks Data Engineer

Quick Facts

Role: Senior Databricks Data Engineer (2 positions)

Engagement: Initial 6-month engagement

Rate: £600/day (Inside IR35)

Location: 2–3 days on site in London

Description

Build and scale a modern Lakehouse platform in Databricks using Spec-Driven Development (SDD), Apache Airflow, and dbt. Translate architectural requirements into production solutions with clear data contracts, transformations, and fully automated orchestration and testing.

Responsibilities

• Design and develop scalable Databricks data pipelines using Medallion Architecture (Bronze/Silver/Gold) with Delta Lake.
• Lead adoption of SDD by creating machine-readable specifications and data contracts before implementation.
• Build, monitor, and optimize Airflow workflow orchestration pipelines.
• Develop, test, and maintain modular dbt transformation models integrated with Databricks SQL and Spark.
• Optimize PySpark workloads, manage cluster configurations, and control compute and storage costs.
• Implement data governance, granular access controls, and data lineage via Unity Catalog.

Requirements

• 5+ years of hands-on data engineering experience, including extensive production Databricks experience.
• Strong expertise in Apache Airflow (scheduling, dependencies, error handling).
• Advanced dbt knowledge (modeling, testing practices, and macro development).
• Practical experience with SDD and metadata-driven data platform design.
• Expert-level Python (PySpark) and SQL skills.
• Familiarity with Git, CI/CD, and infrastructure-as-code practices.
• Multi-cloud experience: AWS, Azure, and/or GCP.

Benefits

• Supportive culture based on values, including inclusion, collaboration, well-being, and learning & development.
• Hybrid working model with increased agility and flexibility.
• Ongoing training and development investment.
• Autonomy to take ownership and demonstrate individual expertise.
• Transparent performance and career management.