Senior Big Data Engineer

Quick Facts

• Role: Senior Big Data Engineer

• Work mode: 100% remote

• Availability: starting immediately or within 14 days

Description

You will design, develop, and optimize scalable data processing solutions in Microsoft Azure. The project includes ETL/ELT pipeline development, large-scale data processing with Apache Spark, and Data Lake solutions using Parquet and Delta Lake to support advanced analytics and Machine Learning.

Responsibilities

• Design, build, and develop scalable ETL/ELT pipelines in Microsoft Azure.

• Optimize data processing in Apache Spark (DataFrame API, Spark SQL) for performance, stability, and reliability.

• Design and develop Data Lake solutions using Parquet, Delta Lake, and data partitioning mechanisms.

• Implement data quality controls and optimize data processing workflows.

• Develop and maintain code with Git and automate deployments using CI/CD.

• Prepare and process data for analytics and Machine Learning initiatives.

• Analyze and resolve technical issues related to performance and large-scale data processing.

• Consult technical and business requirements with project stakeholders.

Requirements

• Senior-level experience designing and developing Big Data / Data Engineering solutions.

• Very good SQL knowledge and data modeling skills.

• Advanced Apache Spark knowledge, including DataFrame API, Spark SQL, and performance optimization.

• Practical Python experience in Data Engineering.

• Experience designing, building, and optimizing ETL/ELT pipelines.

• Practical experience processing large structured and semi-structured datasets.

• Hands-on Data Lake architecture knowledge, including Parquet, Delta Lake, and data partitioning.

• Experience with cloud platforms, especially Microsoft Azure.

• Knowledge of Git and CI/CD processes.

• Basic knowledge of statistics, data analysis, and data quality control mechanisms.

• Ability to troubleshoot technical problems and optimize solution performance.

• Ability to collaborate with technical and business stakeholders.

Benefits

• 100% remote work.

• Opportunity to work on scalable Azure-based data engineering and Machine Learning enablement.