Data Engineer (Big Data & Spark)

Quick Facts

• Role: Data Engineer (Big Data & Spark)

• Focus: Building scalable data processing platforms for streaming and batch workloads

Description

You will develop and maintain high-performance data processing platforms that process large volumes of automotive data in streaming and batch modes, and support additional enterprise initiatives across aerospace and telecommunications. Responsibilities include designing data pipelines, optimizing ingestion/processing/storage workflows, and monitoring performance and reliability. You will use technologies such as Spark, Cloudera, Airflow, Iceberg, Python, and cloud services including AWS (and Azure/Databricks for selected projects).

Responsibilities

• Develop and maintain a scalable, reliable automotive data processing platform

• Design and implement streaming and batch data pipelines

• Optimize data workflows for efficient ingestion, processing, and storage

• Use data lake technologies such as Iceberg for structured and unstructured data

• Collaborate with cross-functional teams to align data sources with requirements

• Monitor and troubleshoot the platform to ensure availability, performance, and accuracy

• Leverage AWS for infrastructure management and workload scaling

• Write and maintain high-quality Python code (or Java/Scala) for data processing and automation

Requirements

• 3+ years of commercial experience with Big Data systems and data governance/management processes

• Strong Python (or Java/Scala) skills with clean code and OOP design

• Hands-on experience with Big Data technologies: Spark, Cloudera, Airflow, and related tools (e.g., Kafka, NiFi, Docker, Iceberg)

• Knowledge of dimensional data and data modeling techniques

• Experience deploying data solutions in cloud environments

• Consulting experience with strong communication and client management, including direct client interaction

• Ability to work independently and take ownership of deliverables

• English fluency at least C1

• Bachelor’s degree in technical or mathematical studies

Benefits

• Supportive team of AI & Big Data enthusiasts

• Opportunity to work with global enterprises and cutting-edge startups on international projects

• Flexible work: remote or modern offices/coworking spaces

• Career growth via learning paths, knowledge sharing, language classes, and sponsored training/conferences (including Databricks and Anthropic programs)

• Team-building events and an integration budget

• Medical and sports packages, eye care, and well-being support (including psychotherapy and coaching)

• Full work equipment (laptop and needed devices)

• Support to build your personal brand (speaking, blog writing, meetups)

• Smooth onboarding with a dedicated buddy