Senior Data Engineer (Databricks)

Quick Facts

• Role: Senior Data Engineer (Databricks)

Description

Design and optimize scalable data processing pipelines for both streaming and batch workloads. Architect end-to-end data platforms with high availability, performance, and reliability, using Databricks and modern orchestration/transformation tooling. Collaborate across Data Science and engineering to deliver governed, secure data solutions and support ML-related workflows.

Responsibilities

• Design and optimize scalable data processing pipelines for streaming and batch workloads using Databricks, Apache Airflow, and Dagster

• Architect and implement end-to-end data platforms with high availability, performance, and reliability

• Lead CI/CD and MLOps automation for deployments, monitoring, and model lifecycle management

• Build and maintain applications for aggregating, processing, and analyzing data from diverse sources

• Collaborate with Data Science on ML projects (text/image analysis, feature engineering, predictive model deployment)

• Design and manage complex data transformations using Databricks, DBT, and Apache Airflow while ensuring data integrity and consistency

• Translate business requirements into scalable technical solutions with optimal performance and data quality

• Ensure data security, compliance, and governance best practices across all data pipelines

Requirements

• At least 5 years of commercial experience implementing, developing, or maintaining Big Data systems

• Strong Python (clean code, OOP)

• Strong SQL (performance tuning, query optimization) and data warehousing experience

• Experience designing and implementing data governance and data management processes

• Deep expertise in Big Data technologies: Apache Airflow, Dagster, Databricks, and modern orchestration/transformation tools

• Cloud deployment experience (preferably Azure)

• Knowledge of building and deploying Power BI reports and dashboards

• Strong understanding of dimensional data and data modeling techniques

• Consulting experience to guide clients through architectural decisions and technology selection

• Ability to work independently and take ownership of deliverables

• Familiarity with Spark and Azure Event Hub or Kafka

• Master’s or Ph.D. in Computer Science, Big Data, Mathematics, Physics, or related field

Benefits

• Flexible work arrangements (remote or modern offices/coworking spaces)

• Career paths, knowledge-sharing initiatives, language classes, and sponsored training/conferences (including Databricks and Anthropic training/certifications)

• Team-building events and integration budget

• Health and well-being support (medical and sports packages, eye care, psychotherapy and coaching)

• Full work equipment (e.g., laptop and necessary devices)

• Smooth onboarding with a dedicated buddy and an autonomous, supportive culture