Senior Data Engineer (Spark)

Quick Facts

• Role: Senior Data Engineer (Spark)

• Focus: Building and maintaining high-performance big data platforms for streaming and batch workloads

Description

You will develop and maintain scalable data processing platforms that handle large volumes of automotive, aerospace, and telecom data. The work includes designing data pipelines, optimizing ingestion/processing/storage, using data lake technologies, and ensuring reliability through monitoring and troubleshooting. You will also write production-quality code (Python or Java/Scala) and leverage cloud infrastructure (AWS/Azure) for scalable workloads.

Responsibilities

• Develop and maintain high-performance data processing for automotive data with scalability and reliability

• Design and implement data pipelines for streaming and batch processing

• Optimize data workflows for efficient ingestion, processing, and storage using Spark, Cloudera, and Airflow

• Use data lake technologies (e.g., Iceberg) to manage data efficiently

• Collaborate cross-functionally to understand data requirements and integrate data sources

• Monitor and troubleshoot the platform to ensure high availability, performance, and accuracy

• Use cloud services (AWS) to manage infrastructure and scale workloads

• Write and maintain high-quality Python (or Java/Scala) code for processing and automation

Requirements

• At least 5 years of commercial experience implementing, developing, or maintaining Big Data systems, including data governance and data management processes

• Strong Python (or Java/Scala) programming skills, including clean code and OOP design

• Hands-on experience with Big Data technologies such as Spark, Cloudera, Airflow, Kafka, NiFi, Docker, and Iceberg

• Excellent understanding of dimensional data and data modeling techniques

• Experience implementing and deploying solutions in cloud environments

• Consulting experience with strong communication and client management; direct client interaction experience

• Ability to work independently and take ownership of deliverables

• Fluent English (C1 or above)

• Bachelor’s degree in technical or mathematical studies

Benefits

• Flexible work: remote or modern offices and coworking spaces

• Career growth via career paths, knowledge sharing, language classes, and sponsored training/conferences (including Databricks and Anthropic partnership)

• Supportive team working on international projects with global enterprises and startups

• Team-building events and integration budget

• Medical and sports packages, eye care, and well-being support (including psychotherapy/coaching)

• Full work equipment provided (laptop and required devices)

• Opportunities to build a personal brand (conferences, blog, meetups)

• Smooth onboarding with a dedicated buddy; autonomous, friendly culture