Data Engineer (Azure Databricks)

Quick Facts

• Role: Data Engineer (Azure Databricks)

• Focus: Enhancing a cloud-based data platform for drug development data products

Description

You will enhance a cloud-based data platform and deliver scalable data solutions for scientific and business stakeholders in the drug development domain. The work centers on building and operating multi-layer Azure Databricks pipelines (Landing → Raw → Enriched → Curated) with reliable ingestion, schema governance, orchestration, and monitoring for SLA-based delivery. You will also support CI/CD and MLOps/AI pipelines on Databricks when the project requires it.

Responsibilities

• Build and maintain multi-layer pipelines (Landing → Raw → Enriched → Curated) on Azure Databricks using Spark and Delta Lake

• Orchestrate pipelines using Databricks Jobs / Workflows (dependencies, retries, alerting)

• Implement batch and API-based ingestion from multiple source systems

• Enforce schemas and handle schema drift; run data quality checks at ingestion

• Use dbt for pipeline observability and data governance

• Monitor job failures, data freshness, and overall pipeline health against SLA-based targets

• Set up alerts, logging, and retry strategies for production pipelines

• Work in controlled Dev/Test/Prod with access control, audit trails, pipeline documentation, and requirement traceability

• Contribute to CI/CD and MLOps/AI pipelines on Databricks as needed

Requirements

• 4+ years of commercial experience in Data Engineering

• Strong hands-on experience with Azure Databricks (Spark, Delta Lake)

• Experience building and maintaining multi-layer data pipelines (Landing → Raw → Enriched → Curated)

• Experience with schema enforcement and schema drift handling

• Ability to implement data quality checks at ingestion

• Experience orchestrating pipelines with Databricks Jobs / Workflows (dependencies, retries, alerts, failure handling)

• Experience with batch and API-based data ingestion

• Strong hands-on experience with dbt

• Ability to monitor job failures, data freshness, and pipeline health

• Experience delivering data in line with defined SLAs

• Experience implementing alerting, logging, and retry strategies

• Experience designing and maintaining access control models

• Understanding of audit trails, data lineage, and end-to-end traceability

• Experience with logging and evidence generation for audit and compliance

• Understanding of segregation of duties

• Experience defining and maintaining data contracts

• Ability to create and maintain clear pipeline documentation and traceability to requirements

• Experience working across controlled Dev, Test, and Prod environments

• Level of English: Upper-Intermediate or above

Benefits

• Fully remote or hybrid work options

• Professional, financial, and career growth with mentoring and onboarding support

• Up to an additional 1,000 EUR/month bonus depending on expertise level

• Access to a corporate training portal (continuously updated)

• Corporate events and perks

• Certification compensation (e.g., AWS, PMP)

• Referral program

• Private health insurance and sports compensation (based on employment type)