Senior Data Engineer (Spark)

Quick Facts

• Role: Senior Data Engineer (Spark)

Description

You will develop and maintain high-performance Big Data processing platforms, including streaming and batch pipelines. The role includes building scalable ingestion, processing, and storage workflows and working with data lake technologies such as Iceberg. You will collaborate with cross-functional teams and help monitor and troubleshoot platforms to ensure availability, performance, and accuracy.

Responsibilities

• Develop and maintain a scalable, reliable automotive data processing platform
• Design and implement data pipelines for large volumes of streaming and batch data
• Optimize data workflows for ingestion, processing, and storage using Spark, Cloudera, and Airflow
• Work with data lake technologies (e.g., Iceberg) for structured and unstructured data
• Integrate diverse data sources with cross-functional teams
• Monitor, troubleshoot, and improve platform availability, performance, and data accuracy
• Use cloud services (AWS) to manage infrastructure and scaling
• Write and maintain high-quality Python (or Java/Scala) code for processing and automation

Benefits

• Flexible work options (remote or modern offices/coworking spaces)
• Career growth with knowledge sharing, language classes, and sponsored training/conferences, including Databricks and Anthropic certification/training materials
• Medical and sports packages, eye care, and wellbeing support (including psychotherapy and coaching)
• Full work equipment provided
• Team-building events and integration budget
• Celebrations for work anniversaries, birthdays, and milestones
• Opportunity to strengthen your personal brand via conferences, blog, and meetups
• Smooth onboarding with a dedicated buddy

Requirements

• 5+ years of commercial experience implementing/developing/maintaining Big Data systems and related governance/management processes
• Strong Python (or Java/Scala) skills, including clean code and OOP design
• Hands-on experience with Big Data technologies including Spark, Cloudera, Airflow, Kafka, NiFi, Docker, and Iceberg
• Dimensional data and data modeling expertise
• Experience deploying solutions in cloud environments
• Consulting experience with strong communication and client management; experience interacting with clients
• Ability to work independently and take ownership
• Fluent English (C1 or higher)
• Bachelor’s degree in technical or mathematical studies

Nice to Have

• Experience with MLOps frameworks such as Kubeflow or MLFlow
• Familiarity with Databricks and/or dbt