Data Engineer (Spark)

Quick Facts

• Role: Data Engineer (Spark)

Description

Develop and maintain high-performance data processing platforms for large-scale streaming and batch workloads. Build and optimize data pipelines, manage data lake storage (e.g., Iceberg), and ensure scalable, reliable, and accurate processing using a Spark-based stack and cloud infrastructure. Write production-quality Python (or Java/Scala) and collaborate with cross-functional teams and clients to deliver data solutions.

Responsibilities

• Develop and maintain scalable, reliable data processing for automotive data
• Design and implement streaming and batch data pipelines
• Optimize ingestion, processing, and storage workflows using Spark, Cloudera, and Airflow
• Work with data lake technologies such as Iceberg
• Collaborate with cross-functional teams to understand data requirements and integrate sources
• Monitor, troubleshoot, and ensure high availability, performance, and accuracy
• Use AWS to manage infrastructure and scale processing workloads
• Write and maintain high-quality Python (or Java/Scala) code for processing and automation

Requirements

• At least 3 years of commercial experience implementing/developing/maintaining Big Data systems, including data governance and data management processes
• Strong Python (or Java/Scala) programming skills with clean code and OOP design
• Hands-on with Big Data technologies including Spark, Cloudera, Airflow, Kafka, NiFi, Docker, and Iceberg
• Strong understanding of dimensional data and data modeling
• Experience deploying solutions in cloud environments
• Consulting experience with strong communication and client management, including direct client interaction
• Ability to work independently and own deliverables
• Fluent English (C1 or higher)
• Bachelor’s degree in technical or mathematical studies

Benefits

• Supportive AI & Big Data team and international projects
• Flexible work (remote or modern offices/coworking)
• Career growth, knowledge-sharing, language classes, and sponsored training/conferences (including Databricks and Anthropic training/certifications)
• Team-building events and integration budget
• Medical and sports packages, eye care, and well-being support (including psychotherapy/coaching)
• Full work equipment provided
• Opportunities to build your personal brand via conferences, blog, and meetups
• Smooth onboarding with a dedicated buddy