Data Engineer (Apache Spark)
Upvanta sp. z o.o.
Wrocław, Poland • 35m ago • Remote
Senior
Remote
1,200 - 1,300 PLN/yr
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Potential salary for Data Engineer (Apache Spark) role
Improve your resume now
PLN 1,200 – PLN 1,300/year
Quick Facts
Role: Data Engineer (Apache Spark)
Description
You will design and implement data reconciliation mechanisms in the ODS layer (based on MongoDB) and synchronize data with source systems. You will build and evolve a Data Lakehouse architecture, covering Raw, Bronze, Silver, and Gold layers. Work includes data validation/quality reporting, performance tuning of Spark jobs, and production monitoring and CI/CD for table schemas.
Responsibilities
Design and implement data reconciliation processes using Apache Spark, PySpark, and Spark SQL
Compare data stored in MongoDB with data in source systems
Create data validation and quality rules; detect duplicates, missing records, and anomalies
Generate reports and alerts for data inconsistencies
Integrate Apache Spark with MongoDB using MongoDB Spark Connector; implement read/write processes
Optimize read operations using partitioning, predicate push-down, and schema-aware reads
Design and develop Raw/Bronze/Silver/Gold lakehouse layers
Implement solutions based on Apache Iceberg, Delta Lake, or Apache Hudi
Manage versioned tables and optimize write/update operations (upsert/merge)
Implement CI/CD for table schema deployments, migrations, and versioning
Tune Spark performance (partitioning, caching, broadcast joins, Adaptive Query Execution)
Optimize query and processing costs in the Data Lakehouse environment
Configure monitoring and metrics with Prometheus and Grafana
Integrate logging and distributed tracing using OpenTelemetry or Dynatrace
Define SLA/SLO and alerts for data freshness and task correctness
Automate Spark application deployments with Docker, Kubernetes, and Spark Operator
Diagnose data quality, delays, and inconsistency issues
Maintain technical documentation, runbooks, and data flow diagrams
Collaborate with Data Engineers, Business Analysts, testers, and DevOps; work in Scrum/Kanban
Conduct code reviews and technical workshops
Requirements
Must-have
Minimum 6 years of commercial experience working with Apache Spark and production environments (banking experience welcome)
Excellent knowledge of PySpark, Spark SQL, DataFrames, and Structured Streaming
Practical experience integrating Apache Spark with MongoDB using MongoDB Spark Connector
Knowledge of document modeling and indexing in MongoDB
Experience with Apache Iceberg, especially table/version management and upsert/merge
Practical Python knowledge
Experience building and maintaining CI/CD processes using Jenkins
Knowledge of Prometheus and Grafana
Experience working in Agile (Scrum/Kanban)
Knowledge of Jira and Confluence
Nice to have
Experience with Delta Lake or Apache Hudi
Knowledge of Docker and Kubernetes and Spark Operator
Practical knowledge of OpenTelemetry or Dynatrace
Experience designing batch/streaming data monitoring solutions
Knowledge of Data Quality, Data Observability, and data lifecycle management
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See moreData Engineer (Spark)
Upvanta sp. z o.o.
Senior
Technology
Wrocław, DL, Poland · Remote
N/A
10 days ago

Senior Data Engineer – Spark / Data Lakehouse
Clurgo
Senior
Technology
Wrocław, DS, Poland · On-site
N/A
14 days ago
Spark / Data Lakehouse Developer
Ideamotive
Senior
Technology
Wrocław, DS, Poland · On-site
N/A
1 days ago

Spark / Data Lakehouse Developer (m/k/n)
B2Bnetwork
Senior
Technology
Wrocław, DS, Poland · On-site
N/A
11 days ago
Spark / Data Lakehouse Developer
Upvanta sp. z o.o.
Senior
Technology
Wrocław, DL, Poland · On-site
1k zł - 1k zł/yr
6 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Spark / Data Lakehouse Developer (k/m/x)
B2Bnetwork
Mid
Technology
Wrocław, DS, Poland · Remote
N/A
14 days ago
Databricks Data Engineer
RemoDevs
Mid
Technology
Warsaw, Poland · Remote
N/A
12 days ago

Senior Data Engineer / Tech Lead (Spark)
Addepto
Senior
Technology
Warsaw, Poland · Remote
21k zł - 29k zł/yr
7 days ago
Senior Databricks Engineer
Greyson Consulting
Senior
Technology
Prague, Czechia · Hybrid
N/A
6 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights