Data Engineer (Apache Spark)

Upvanta sp. z o.o.

Wrocław, Poland • 35m ago • Remote

Senior

Remote

1,200 - 1,300 PLN/yr

  1. Select offer
  2. Add resume
  3. Analyze
  4. Improve
  5. Apply
  6. Auto-send · soon

Match your resume with similar open roles

Upload your resume to automatically match with active openings similar to this role.

PDF up to 10 MB

Your resume is private & secure

Potential salary for Data Engineer (Apache Spark) role

Improve your resume now

PLN 1,200 – PLN 1,300/year

Quick Facts

  • Role: Data Engineer (Apache Spark)

Description

You will design and implement data reconciliation mechanisms in the ODS layer (based on MongoDB) and synchronize data with source systems. You will build and evolve a Data Lakehouse architecture, covering Raw, Bronze, Silver, and Gold layers. Work includes data validation/quality reporting, performance tuning of Spark jobs, and production monitoring and CI/CD for table schemas.

Responsibilities

  • Design and implement data reconciliation processes using Apache Spark, PySpark, and Spark SQL

  • Compare data stored in MongoDB with data in source systems

  • Create data validation and quality rules; detect duplicates, missing records, and anomalies

  • Generate reports and alerts for data inconsistencies

  • Integrate Apache Spark with MongoDB using MongoDB Spark Connector; implement read/write processes

  • Optimize read operations using partitioning, predicate push-down, and schema-aware reads

  • Design and develop Raw/Bronze/Silver/Gold lakehouse layers

  • Implement solutions based on Apache Iceberg, Delta Lake, or Apache Hudi

  • Manage versioned tables and optimize write/update operations (upsert/merge)

  • Implement CI/CD for table schema deployments, migrations, and versioning

  • Tune Spark performance (partitioning, caching, broadcast joins, Adaptive Query Execution)

  • Optimize query and processing costs in the Data Lakehouse environment

  • Configure monitoring and metrics with Prometheus and Grafana

  • Integrate logging and distributed tracing using OpenTelemetry or Dynatrace

  • Define SLA/SLO and alerts for data freshness and task correctness

  • Automate Spark application deployments with Docker, Kubernetes, and Spark Operator

  • Diagnose data quality, delays, and inconsistency issues

  • Maintain technical documentation, runbooks, and data flow diagrams

  • Collaborate with Data Engineers, Business Analysts, testers, and DevOps; work in Scrum/Kanban

  • Conduct code reviews and technical workshops

Requirements

Must-have

  • Minimum 6 years of commercial experience working with Apache Spark and production environments (banking experience welcome)

  • Excellent knowledge of PySpark, Spark SQL, DataFrames, and Structured Streaming

  • Practical experience integrating Apache Spark with MongoDB using MongoDB Spark Connector

  • Knowledge of document modeling and indexing in MongoDB

  • Experience with Apache Iceberg, especially table/version management and upsert/merge

  • Practical Python knowledge

  • Experience building and maintaining CI/CD processes using Jenkins

  • Knowledge of Prometheus and Grafana

  • Experience working in Agile (Scrum/Kanban)

  • Knowledge of Jira and Confluence

Nice to have

  • Experience with Delta Lake or Apache Hudi

  • Knowledge of Docker and Kubernetes and Spark Operator

  • Practical knowledge of OpenTelemetry or Dynatrace

  • Experience designing batch/streaming data monitoring solutions

  • Knowledge of Data Quality, Data Observability, and data lifecycle management

See similar jobs

Explore similar jobs and use your resume score to choose where to apply next

See more

Unlock Full Resume Report