Data Engineer (Spark)

Quick Facts

• Role: Data Engineer (Spark)

• Focus: Data reconciliation, ODS + Data Lakehouse (raw/bronze/silver/gold)

• Remote work: Yes

Description

Design, develop, and optimize data reconciliation solutions in an Operational Data Store (ODS) based on MongoDB, and synchronize them with source systems in the banking industry. You will also build and maintain a Data Lakehouse architecture across raw/bronze/silver/gold layers using Apache Spark and Apache Iceberg, including table versioning and CI/CD.

Responsibilities

• Design and implement data reconciliation processes using Apache Spark, PySpark, and Spark SQL, including comparing MongoDB data with source system data

• Create data quality rules; detect duplicates, missing records, and anomalies; generate reports and alerts

• Integrate Apache Spark with MongoDB using MongoDB Spark Connector, optimizing reads and writes

• Design, develop, and maintain Data Lakehouse layers (raw/bronze/silver/gold) with Apache Iceberg

• Implement CI/CD for table schemas, migrations, and data versioning

• Optimize Spark performance (partitioning, caching, broadcast joins, Adaptive Query Execution)

• Monitor data processing and implement observability with Prometheus and Grafana

• Automate build, testing, and deployment of Spark apps using Jenkins; use Docker and Kubernetes depending on the environment

• Diagnose issues related to data quality, consistency, and timeliness, including processing delays

• Maintain technical documentation, runbooks, and data flow diagrams

• Collaborate with Data Engineers, business analysts, testers, and DevOps in Agile; participate in code reviews and knowledge sharing

Requirements

• Minimum 5 years of practical experience as a Data Engineer or Big Data Engineer

• Very strong knowledge of PySpark, Spark SQL, DataFrames, and Structured Streaming

• Experience reading and writing MongoDB data with MongoDB Spark Connector

• Practical experience with Apache Iceberg, including table management, data versioning, and upsert/merge operations

• Very good knowledge of Python and SQL

• Experience building and maintaining CI/CD pipelines using Jenkins

• Experience testing Spark applications, including unit and integration tests

• Experience monitoring data processing with Prometheus and Grafana

• Knowledge of Scrum or Kanban and tools Jira and Confluence

Additional advantages

• Experience with Delta Lake or Apache Hudi

• Knowledge of Kubernetes, Docker, and Spark Operator

• Experience implementing observability with OpenTelemetry or Dynatrace

• Knowledge of data quality, data lineage, and defining SLA/SLO for data processes

• Experience optimizing cost and performance for large-volume data processing