Senior/Lead Data Software Engineer (Python, Spark, Azure)

Quick Facts

• Role: Senior/Lead Data Software Engineer (Python, Spark, Azure)

• Focus: Azure Databricks migration, ML-ready data workflows, governance & regulatory compliance

Description

You will join a team building a scalable, ML-ready platform to enhance portfolio model development and deployment with advanced data governance and AI capabilities. The role is centered on migrating from an IaaS Big Data platform to Azure-native Databricks, optimizing data workflows and improving data quality. You’ll also tune ML models in Azure environments and help integrate GenAI capabilities into data workflows.

Responsibilities

• Migrate and optimize 500+ data jobs using Azure Databricks optimization techniques

• Manage and process ~12 TB of data across platforms

• Tune machine learning models for Azure environments using Java Spark and Delta tables

• Update and maintain libraries to address security vulnerabilities

• Build and maintain ETL/ELT pipelines using PySpark and related technologies

• Integrate GenAI capabilities into data workflows with cross-functional teams

• Monitor data quality and implement improvements to ensure accuracy and reliability

• Automate deployment and operational tasks using Terraform and GitLab CI/CD

• Support data governance initiatives for regulatory compliance

• Troubleshoot and resolve performance issues in data processing systems

• Document system processes and provide technical guidance to junior engineers

• Implement best practices for code quality and data security

• Participate in code reviews and knowledge sharing

• Optimize costs for data storage and processing

Requirements

• 3+ years proficiency in Python and Spark in data engineering roles

• Strong experience with Azure Databricks and PySpark

• Proven ETL/ELT solution design and implementation experience

• Experience migrating big data platforms to Azure-native services

• Proficiency with Delta tables for model tuning

• Knowledge of data governance and regulatory compliance frameworks

• Familiarity with Docker, Kubernetes (AKS), and Terraform

• Ability to manage large data volumes efficiently

• Strong problem-solving and analytical skills

• Strong communication and collaboration abilities

• English proficiency at B2 level or higher

Benefits

• Stable pay

• Employee Stock Purchase Plan with a 15% discount

• Benefits package (health insurance, multisport, shopping vouchers)

• Referral bonuses up to $2,000

• Hybrid work (remote within Poland)

• Opportunity to work abroad up to 60 days annually

• Business-driven relocation opportunities

• Career development programs, mentoring, and soft skills & well-being programs

• Certifications (Anthropic, Gemini, GCP, Azure, AWS)

• English classes

• Offices with entertainment and relaxation zones; table tennis and football; free snacks, coffee, and more

• Corporate, social, and well-being events