Software Engineer (Spark/Databricks)

Quick Facts

• Role: Software Engineer (Spark/Databricks)

Description

Design, develop, and optimize large-scale data pipelines and distributed streaming data processing solutions using Spark and Databricks. Work on Azure-based, low-latency architectures integrated with Kafka, build DevOps/CI-CD and Kubernetes deployments, and support production stability. Collaborate with Agile teams to analyze complex technical flows, improve designs, and communicate effectively with stakeholders.

Responsibilities

• Design, develop, and enhance data pipelines that feed and reconcile inbound data into regulatory and settlement systems
• Build and optimize high-performance, high-availability, low-latency distributed data processing solutions
• Develop and maintain streaming architectures leveraging Kafka and cloud-native services within Azure
• Collaborate with cross-functional Agile teams to analyze technical flows and propose improvements
• Contribute to DevOps practices, CI/CD pipelines, containerized deployments, and Kubernetes-based environments
• Investigate and resolve complex production issues to support stability
• Communicate technical concepts to business and technology stakeholders
• Drive continuous improvement by independently solving challenging technical problems and enhancing existing platforms

Requirements

• 7+ years of software engineering or data engineering experience
• Extensive hands-on experience with Spark, Databricks, and PySpark
• Strong development skills in Python and Java
• Deep experience with Azure Cloud technologies
• Experience working with Kubernetes, Kafka architectures, and streaming data
• Experience implementing and maintaining DevOps practices and CI/CD pipelines
• Strong analytical and problem-solving skills with the ability to work independently on complex technical challenges
• Excellent communication skills and experience collaborating with business and technology stakeholders in Agile environments

Benefits

• Work on a strategic, large-scale data platform supporting critical business and operational processes
• Collaborate with international teams and stakeholders and gain exposure to enterprise architectures and modern data technologies
• Continuous learning and professional growth opportunities in data and cloud environments
• Flexible work location with a mix of remote and onsite time depending on project circumstances
• Permanent employment contract or B2B
• Individual support from a People Lead plus a development path and optional coaching session
• Training package (soft, technical, and language training)
• Employee Assistance Program (legal, financial, and psychological consultations)
• Employee Share Purchase Plan eligibility with potential quarterly dividends
• Paid employee referral program
• Private medical care and life insurance
• Access to the MyBenefit platform (e.g., Multisport card)