Lead Data Engineer (Spark)

Quick Facts

• Role: Lead Data Engineer (Spark)

Description

Lead the design and development of scalable data management architectures and platform solutions for both streaming and batch processing. Build data governance and data management processes, contribute to CI/CD and MLOps, and develop data transformations and pipelines using modern big data technologies. Translate business requirements into technical solutions and collaborate with Data Science on analytics and machine learning use cases.

Responsibilities

• Design and develop scalable data management architectures, infrastructure, and platform solutions for streaming and batch processing (Apache Spark, Hadoop, Iceberg)

• Design and implement data management and data governance processes and best practices

• Contribute to CI/CD and MLOps processes

• Develop applications to aggregate, process, and analyze data from diverse sources

• Collaborate with the Data Science team on text/image analysis and predictive model building

• Develop and organize data transformations using DBT and Apache Airflow

• Translate business requirements into technical solutions, ensuring optimal performance and quality

Requirements

• 5+ years of commercial experience implementing, developing, or maintaining Big Data systems

• Strong programming skills in Python or Java/Scala (clean code, OOP design)

• Experience designing and implementing data governance and data management processes

• Familiarity with Big Data technologies such as Spark, Cloudera, Airflow, NiFi, Docker, Kubernetes, Iceberg, Trino, or Hudi

• Proven expertise deploying solutions in cloud environments (preferred: AWS)

• Excellent understanding of dimensional data and data modeling techniques

• Excellent communication skills and consulting experience with direct interaction with clients

• Ability to work independently and take ownership of deliverables

• Master’s or Ph.D. in Computer Science, Data Science, Mathematics, Physics, or related field

• Fluent English (C1 level)

Benefits

• Supportive AI & Big Data team; work with global enterprises and cutting-edge startups

• Flexible work arrangements (remote or modern offices/coworking spaces)

• Career growth via knowledge sharing, language classes, and sponsored training or conferences (including Databricks training and certifications)

• Choose B2B or contract of mandate; 20 fully paid days off

• Medical and sports packages, eye care, and well-being support (including psychotherapy and coaching)

• Full work equipment for productivity, including a laptop and other necessary devices

• Smooth onboarding with a dedicated buddy