Senior Data Engineer (Spark)

Quick Facts

• Role: Senior Data Engineer (Spark)

Description

Work on high-performance big data platforms that process large volumes of data in both streaming and batch modes. Design and optimize scalable data pipelines and data lake workflows, monitor and troubleshoot for availability and accuracy, and implement cloud-based infrastructure scaling. Write production-ready code in Python (or Java/Scala) for processing and automation.

Responsibilities

• Develop and maintain a high-performance data processing platform for automotive data, ensuring scalability and reliability.

• Design and implement data pipelines that process streaming and batch workloads.

• Optimize data workflows for ingestion, processing, and storage using Spark, Cloudera, and Airflow.

• Use data lake technologies (e.g., Iceberg) to manage structured and unstructured data.

• Collaborate with cross-functional teams to understand data requirements and integrate data sources.

• Monitor and troubleshoot the platform to ensure high availability, performance, and accuracy.

• Use AWS to manage infrastructure and scale processing workloads.

• Write and maintain high-quality Python (or Java/Scala) code for data processing tasks and automation.

Requirements

• At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems, including data governance and data management processes.

• Strong programming skills in Python (or Java/Scala), including clean code and OOP design.

• Hands-on experience with Big Data technologies such as Spark, Cloudera, Airflow, Kafka, NiFi, Docker, and Iceberg.

• Excellent understanding of dimensional data and data modeling techniques.

• Experience implementing and deploying cloud solutions.

• Consulting experience with excellent communication and client management skills, including direct client interaction.

• Ability to work independently and take ownership of deliverables.

• Fluent English (C1 or higher).

• Bachelor’s degree in technical or mathematical studies.

Benefits

• Supportive team focused on AI & Big Data and international projects.

• Flexible work arrangements (remote or from modern offices/coworking spaces).

• Career paths, knowledge sharing, and sponsored training or conferences; training partnerships with Databricks and Anthropic.

• Team-building events and use of an integration budget.

• Medical and sports packages, eye care, and well-being support (including psychotherapy and coaching).

• Full work equipment for productivity (laptop and necessary devices).

• Opportunities to build your personal brand through conferences, blog writing, and meetups.

• Dedicated-buddy onboarding and a friendly, autonomous culture.