Senior/Lead Data Software Engineer

Quick Facts

Senior/Lead Data Software Engineer (Python, Spark, Azure)

Description

Lead the migration and optimization of large-scale data workflows to an Azure-native Databricks environment for an ML-ready, governance-focused platform. Build and maintain PySpark/Delta-based ETL/ELT pipelines, improve data quality, and automate deployment and operations. Support GenAI integration, troubleshoot performance issues, and contribute to regulatory compliance.

Responsibilities

• Migrate and optimize 500+ data jobs using Azure Databricks optimization techniques
• Manage and process ~12 TB of data across platforms
• Tune machine learning models for Azure environments using Java Spark and Delta tables
• Update and maintain libraries to address security vulnerabilities
• Develop and maintain ETL/ELT pipelines using PySpark
• Integrate and monitor GenAI capabilities within data workflows
• Monitor data quality and implement improvements
• Automate deployment and operational tasks using Terraform and GitLab CI/CD
• Support data governance initiatives for regulatory compliance
• Troubleshoot and resolve performance issues in data processing systems
• Document system processes and provide technical guidance to junior engineers
• Follow best practices for code quality and data security
• Participate in code reviews and knowledge-sharing
• Optimize costs for data storage and processing

Requirements

• 3+ years in data engineering roles; proficiency in Python and Spark
• Strong experience with Azure Databricks and PySpark
• Proven expertise designing and implementing ETL/ELT solutions
• Experience migrating big data platforms to Azure-native services
• Proficiency with Delta tables for model tuning
• Knowledge of data governance and regulatory compliance frameworks
• Familiarity with Docker, Kubernetes (AKS), and Terraform
• Ability to manage large data volumes efficiently
• Strong problem-solving and analytical skills
• Excellent communication and collaboration abilities
• English proficiency at B2 level or higher

Benefits

• Flexible schedule and opportunity to work remotely within Poland
• Opportunity to work abroad up to 60 days annually and business-driven relocation support
• Career growth: leadership development, career advising, soft skills, and well-being programs
• Certification support (GCP, Azure, AWS)
• Unlimited access to learning platforms (LinkedIn Learning, Get Abstract, Cloud Guru) plus English classes
• Stable income via employment contract or B2B, plus Employee Stock Purchase Plan
• Benefits package: health insurance, multisport, shopping vouchers
• Office perks and community events, including referral bonuses and corporate social well-being events