Data Engineer

Quick Facts

• Role: Data Engineer

• Focus: Big Data and data engineering solutions

• Platform: Google Cloud Platform (GCP)

Description

Design, develop, and maintain data pipelines for a large financial organization, building solutions in PySpark/Apache Spark and automating their execution with Airflow. Work with deployment and scheduling using Git/GitHub plus Ansible and Jenkins, and collaborate with both technical and business stakeholders.

Responsibilities

• Design, develop, and maintain data pipelines on GCP

• Develop solutions based on PySpark/Apache Spark

• Automate scheduling and processing with Airflow

• Use Git/GitHub and automation tools such as Ansible and Jenkins

• Collaborate with technical and business teams

• Present solutions and work progress to technical and non-technical audiences

• Support other team members with delivery of their tasks

• Identify opportunities to use AI to accelerate and automate processes

Requirements

• Practical experience with PySpark or Scala

• Experience with Google Cloud Platform (GCP)

• Experience building and maintaining data pipelines

• Experience with Airflow

• Strong knowledge of Python, Apache Hadoop, YARN, Hive, Spark SQL, MapReduce, and SQL

• Comfortable working in Unix/Linux

• Experience with Git and GitHub

• Knowledge of deployment automation tools such as Ansible and Jenkins

• Ability to present solutions to both business and technical stakeholders

• Comfortable communication in English

Benefits

• B2B contract

• Hybrid work model: 8 days per month in the Kraków office