Senior Data Engineer (Azure)

Quick Facts

• Remote collaboration: 100% remote
• Availability: start immediately or within up to 1 month

Description

Build and optimize scalable data processing pipelines on Azure, with a focus on performance, reliability, and scalability for large data volumes. Develop transformations with Azure Databricks, store data in Azure Data Lake, and orchestrate integrations using Azure Data Factory. You will also monitor data quality, troubleshoot data-flow issues, and help guide technical architecture decisions.

Responsibilities

• Design and develop scalable data pipelines and data processing workflows in Azure
• Develop Azure Databricks solutions for transforming and processing large datasets
• Build and evolve a data storage layer based on Azure Data Lake
• Create and maintain data integration and orchestration workflows with Azure Data Factory
• Implement data transformation processes in Python and SQL
• Integrate data coming from multiple systems and sources
• Optimize pipelines, queries, and processing steps for performance and scalability
• Monitor the quality and correctness of processed data and diagnose issues in data flows
• Develop mechanisms that support the reliability and maintainability of the data platform
• Participate in technical decision-making for data platform architecture and further development

Requirements

• Minimum 6 years of commercial Data Engineering experience
• Practical, production-grade knowledge of Azure Databricks
• Experience designing and developing solutions using Azure Data Lake
• Strong knowledge of Azure Data Factory and experience creating/orchestrating data pipelines
• Strong SQL knowledge, including complex queries and optimization
• Strong Python knowledge for data processing and transformation
• Experience designing ETL/ELT processes and integrating data from multiple sources
• Ability to optimize performance for large-scale data processing
• Experience analyzing data quality, consistency, and data flow problems
• Ability to independently design solutions and make technical decisions in Data Engineering
• English proficiency at C1 or higher for effective communication in an international environment

Benefits

• Contribute to a scalable data processing platform built on Azure Databricks, Azure Data Lake, and Azure Data Factory
• Work on complex topics spanning integration, transformation, data quality, and processing optimization
• Participate in architectural and technical decision-making for the platform’s further evolution