Spark / Data Lakehouse Developer

Quick Facts

• Role: Spark / Data Lakehouse Developer

• Focus: Data reconciliation and processing in a Data Lakehouse architecture

• Stack highlights: Apache Spark (Spark SQL/PySpark), Apache Iceberg, MongoDB, Prometheus/Grafana, Jenkins

Description

You will design and implement Spark SQL / PySpark processes for data reconciliation and data quality checks in a Data Lakehouse setting. Work includes reconciling data between MongoDB and source systems, integrating Spark with MongoDB, building Iceberg-based lakehouse layers, and optimizing Spark performance. The role also involves developing CI/CD for data applications, monitoring production processes, and supporting troubleshooting, documentation, and code reviews.

Responsibilities

• Design and implement data processing processes in Spark SQL / PySpark

• Reconcile data between MongoDB and source systems

• Build data quality rules and detect duplicates, missing data, and anomalies

• Integrate Apache Spark with MongoDB

• Build and evolve Data Lakehouse layers

• Optimize performance of Spark jobs

• Develop CI/CD for data applications and processes

• Monitor processes and diagnose production issues

• Create technical documentation and participate in code reviews

Requirements

• Minimum 4 years of practical experience with Apache Spark in production environments

• Very good knowledge of PySpark, Spark SQL, and DataFrames

• Practical experience with Apache Iceberg (mandatory), including versioned tables and upsert/merge operations

• Experience with MongoDB (MongoDB Spark Connector is not required)

• Knowledge of Python / PySpark

• Experience building data reconciliation and data processing workflows

• Experience with Jenkins

• Knowledge of Prometheus and Grafana

• Experience working with Scrum or Kanban and with Jira/Confluence

Benefits

Opportunity to work on data reconciliation and processing solutions in a Data Lakehouse architecture, with a roadmap that includes future streaming capabilities.