Senior Data Engineer (Databricks)

Quick Facts

• Senior Data Engineer focused on Databricks
• Builds scalable data platforms and transformation pipelines for streaming and batch workloads

Description

Design and optimize end-to-end data processing pipelines for both streaming and batch systems using Big Data technologies such as Databricks and modern orchestration tools. Lead CI/CD and MLOps automation for deployments, monitoring, and model lifecycle management, and collaborate with Data Science teams to deliver ML-ready datasets.

Responsibilities

• Design and optimize scalable data processing pipelines for streaming and batch workloads
• Architect and implement data platforms with high availability, performance, and reliability
• Develop CI/CD and MLOps processes to automate deployments and model lifecycle management
• Build and maintain applications to aggregate, process, and analyze data from diverse sources
• Collaborate with Data Science on ML initiatives (text/image analysis, feature engineering, predictive deployment)
• Manage complex data transformations using Databricks, DBT, and Apache Airflow
• Translate business requirements into scalable solutions with strong data quality and performance
• Ensure security, compliance, and governance best practices across pipelines

Requirements

• 5+ years of commercial experience implementing, developing, or maintaining Big Data systems
• Strong Python skills (clean code, OOP design)
• Strong SQL skills (performance tuning, query optimization, data warehousing)
• Experience with data governance and data management processes
• Deep expertise in Big Data technologies (Databricks, Spark, Kafka, EventHub, Airflow, Dagster, modern orchestration/transformation tools)
• Cloud deployment experience (preferred: Azure)
• Knowledge of Power BI report and dashboard development
• Strong understanding of dimensional data and data modeling
• Consulting experience to guide client architectural decisions and technology selection
• Ability to work independently and take ownership of deliverables
• Master’s or Ph.D. in a relevant field (Computer Science, Big Data, Mathematics, Physics, or related)

Benefits

• Remote or office/hybrid flexibility
• Career growth, knowledge sharing, language classes, and sponsored training/conferences (including Databricks and Anthropic training/certifications)
• Medical and sports packages, eye care, and well-being support (including psychotherapy/coaching)
• Team events and an integration budget
• Full work equipment (laptop and necessary devices)
• Opportunities to build your personal brand via conferences, blog writing, and meetups