Senior Data Engineer

Quick Facts

Senior Data Engineer for Data Management & Data Foundation (Azure + Databricks)

Description

Design, develop, and optimize complex data pipelines and data solutions on Azure and Databricks, taking end-to-end technical ownership from challenge understanding through design, implementation, testing, and production deployment. Build production-grade code using Python, PySpark, and SQL; work extensively with Azure Databricks and Databricks ecosystem capabilities, including AI-enabled features such as RAG-based approaches and conversational access to enterprise data. Contribute to migration and modernization of existing data platforms into Databricks-based architectures and work with Unity Catalog for scalable data discovery, governance, and access.

Responsibilities

• Design, develop, and optimize complex data solutions and data pipelines on Azure and Databricks
• Take end-to-end technical ownership of assigned engineering problems
• Write high-quality production-grade code using Python, PySpark, and SQL
• Work extensively with Azure Databricks and the Databricks ecosystem
• Explore and implement AI-enabled capabilities within the data platform (Databricks AI features, RAG, data discovery, conversational access)
• Contribute to migration and modernization into Databricks-based architectures
• Work with Unity Catalog for data discovery, governance, and access
• Solve technically difficult or non-standard problems
• Improve architecture, performance, scalability, reliability, and maintainability
• Challenge existing approaches with better technical solutions
• Collaborate with Data Engineers, Architects, Product Owners, DataOps, QA, DevOps, and other teams
• Support architectural discussions and contribute to technical design decisions
• Move between strategic initiatives as priorities evolve

Requirements

• 8+ years of professional experience in Data Engineering, Software Engineering, or a closely related field
• Strong hands-on commercial experience with Azure Databricks
• Very good knowledge of the Azure cloud ecosystem and cloud-based data architectures
• Excellent programming skills in Python (strong coder)
• Advanced PySpark skills and experience working with large-scale data processing workloads
• Advanced SQL, including complex transformations, optimization, and analytical/window functions
• Strong understanding of data engineering principles, ETL/ELT patterns, data pipelines, distributed processing, and production data platforms
• Experience designing and delivering solutions independently, with limited supervision
• Ability to understand an unfamiliar technical problem, propose an approach, and drive it through to implementation
• Strong troubleshooting and problem-solving skills
• Experience with production-grade engineering practices, including version control, testing, CI/CD, monitoring, and deployment processes
• Strong understanding of software development principles and ability to produce clean, maintainable, well-structured code
• High level of ownership, independence, and proactivity
• Excellent English communication skills, both written and spoken

Benefits

• Opportunity to work on high-impact, strategic data initiatives rather than repetitive business-as-usual development
• Significant technical ownership and freedom in how problems are solved
• Hands-on work with Azure, Databricks, Python, PySpark, SQL, and emerging AI capabilities
• Exposure to modern applications of AI and RAG in enterprise data management and discovery
• Opportunity to influence technical architecture and engineering practices
• Collaboration with experienced engineers and architects across an international technology organization

Additional “Particularly valuable” skills

• Hands-on experience with AI capabilities within Databricks and modern AI-enabled data solutions
• Experience with RAG architectures, LLM-based applications, or conversational interfaces built on enterprise data
• Experience with Databricks Genie or similar conversational data discovery capabilities
• Strong knowledge of Unity Catalog
• Experience migrating enterprise data platforms or data management solutions to Databricks
• Understanding of data cataloguing, metadata management, lineage, governance, and data discovery
• Strong awareness of solution and data architecture and ability to contribute to architectural decisions
• Experience designing reusable frameworks/components/engineering standards
• Performance optimization experience in Databricks/Spark environments
• Experience with Azure DevOps and mature CI/CD practices for data solutions
• Architectural experience is not required, but strong coding plus architectural awareness is a plus