Data Engineer

Quick Facts

• Role: Data Engineer

• Focus: Execute large-scale data platform modernization and dataset migrations to Snowflake

• Orchestration: Apache Airflow 3

Description

You will work embedded with the data engineering function and partner with the data platform team to migrate approximately 1,000 datasets to Snowflake. Responsibilities include time-sensitive migrations such as Apache Airflow 2→3, Confluent region migration, and moving pipelines from Hive to Iceberg. The architecture and technical approach are pre-defined, so the role emphasizes execution, pipeline reliability, and monitoring.

Responsibilities

• Migrate approximately 1,000 datasets to Snowflake using Apache Airflow 3 for orchestration

• Migrate pipelines from Apache Airflow 2 to Apache Airflow 3

• Migrate Confluent connections to a new region

• Migrate pipelines from Hive to Iceberg using Snowpark Connect

• Adapt existing pipelines to Snowflake while ensuring data quality and reliability

• Support workflow scheduling and monitoring in Apache Airflow

Requirements

• 3+ years of industry experience working with large-scale datasets, including Airflow and Spark

• Strong software engineering fundamentals

• Hands-on Python development experience

• Advanced SQL (complex joins, CTEs, window functions, performance tuning)

• Snowflake data modeling experience

• Experience developing and testing pipelines in Apache Airflow and Python

• Experience with Apache Kafka

• Experience with Apache Spark, including hands-on PySpark

• Experience with Iceberg and Parquet data formats

• AWS experience, particularly S3 (data lakes, partitioning, lifecycle policies)

• Testing strategy experience with the ability to debug and test code hands-on

• Strong communication skills

• Ability to work independently

Benefits

• US and EU projects using advanced technologies

• Competitive compensation based on skills and experience

• Regular performance appraisals

• Flexibility to work remote or from the office

• Bonuses for article writing, public talks, and other activities

• Generous time off (vacation, national holidays, sick leave, family days)

• Personalized learning programs

• Free tech webinars and meetups

• Regular corporate online activities