Senior Data Engineer (Databricks)

Quick Facts

• Role: Senior Data Engineer

• Focus: Scalable batch + streaming data platforms using Databricks/Azure

Description

Design and optimize scalable data processing pipelines for both streaming and batch workloads using Big Data technologies such as Databricks, Apache Airflow, and Dagster. Architect and implement end-to-end data platforms with high availability, performance, and reliability, while leading CI/CD and MLOps processes for automated deployments and model lifecycle management. Collaborate with Data Science teams to support ML use cases and build secure, governed data transformations.

Responsibilities

• Design and optimize scalable streaming and batch data processing pipelines

• Architect and implement end-to-end data platforms ensuring availability, performance, and reliability

• Lead CI/CD and MLOps processes for automated deployments, monitoring, and model lifecycle management

• Develop and maintain data aggregation, processing, and analytics applications

• Collaborate with Data Science on ML projects (e.g., text/image analysis, feature engineering, predictive model deployment)

• Design and manage complex data transformations using Databricks, dbt, and orchestration workflows

• Translate business requirements into scalable, efficient technical solutions while ensuring data quality and performance

• Ensure data security, compliance, and governance best practices across all data pipelines

Requirements

• At least 5 years of commercial experience implementing, developing, or maintaining Big Data systems

• Strong Python skills (clean code, OOP design)

• Strong SQL skills (performance tuning and query optimization) and data warehousing experience

• Experience designing and implementing data governance and data management processes

• Deep expertise in Big Data technologies including Databricks, Spark, Kafka, EventHub, Airflow, and Dagster

• Experience deploying solutions in cloud environments (preferably Azure)

• Knowledge of building Power BI reports and dashboards

• Excellent understanding of dimensional data and data modeling techniques

• Consulting experience and ability to guide clients through architectural decisions and technology selection

• Ability to work independently and take ownership of deliverables

• Master’s or Ph.D. in a relevant field (Computer Science, Big Data, Mathematics, Physics, or related)

Benefits

• Supportive team and opportunities to work on international projects with global enterprises and startups

• Flexible work arrangements (remote or modern offices/coworking spaces)

• Career growth through training, knowledge-sharing, language classes, and sponsored conferences (including Databricks/Anthropic training & certifications)

• Team-building events and an integration budget

• Medical and sports packages, eye care, and well-being support (including psychotherapy and coaching)

• Full work equipment for optimal productivity

• Opportunities to build your personal brand via conferences, blog writing, and meetups