Data Engineer (Google Cloud Platform)

Quick Facts

• Role: Data Engineer (Google Cloud Platform)

• Key focus: data pipeline design, ETL/ELT (batch/streaming/CDC), orchestration, security & governance, monitoring, performance/cost optimization.

Description

You will design, build, and maintain data processing pipelines on Google Cloud Platform (GCP). The role covers implementing ETL/ELT and data integration across multiple source types (including CDC), as well as orchestration and CI/CD for data workflows. You will also ensure secure access control, data quality validation, monitoring, and performance/cost optimization, including integrations with SAP and other cloud platforms.

Responsibilities

• Analyze business and technical requirements and design GCP-based data pipeline architecture

• Select data processing patterns aligned with SLA requirements

• Design, implement, and maintain ETL/ELT pipelines using Dataproc, Dataflow, BigQuery, and Dataform

• Implement transformation logic with SQL, Python, and dbt

• Integrate data from on-premise systems, cloud platforms, APIs, files, and streaming sources

• Implement CDC (Change Data Capture)

• Design and maintain orchestration DAGs with Cloud Composer / Apache Airflow

• Handle scheduling, dependencies, error handling, and retry mechanisms

• Develop and maintain CI/CD pipelines for data workflows using GitLab

• Automate testing, deployments, and versioning of data processes

• Define and manage IaC resources for data processing (e.g., GCS, BigQuery, Cloud Composer, Dataproc)

• Monitor data processing runs and respond to incidents in the data logic layer

• Optimize performance and processing costs (BigQuery query tuning, partitioning, clustering)

• Refactor and modernize existing processes and support data migration

• Implement data validation and quality controls

• Design and implement fine-grained data access permissions (dataset/table/column/row)

• Configure IAM, row-level and column-level security, and policy tags

• Manage access using Workforce Identity Federation and Microsoft Entra ID

• Support data classification and cataloging with Dataplex Universal Catalog

• Collaborate with infrastructure, security, and Data Governance teams

• Prepare and keep technical documentation up to date

• Perform code reviews and provide SME support for data teams

Requirements

• At least 3 years of experience as a Data Engineer or a GCP Consultant

• Practical knowledge of GCP data processing and integration services

• BigQuery expertise: data modeling, SQL, query optimization, partitioning, clustering

• Practical knowledge of Cloud Composer / Apache Airflow

• Knowledge of Dataproc, Dataflow, and Datastream

• Knowledge of Cloud Run / Cloud Run Functions

• Knowledge of Cloud Storage for data processing

• Strong command of Python and SQL

• Knowledge of GitLab including CI/CD and versioning

• Knowledge of Terraform for defining data-processing resources

• Integration patterns knowledge: batch, streaming, CDC

• Experience translating business requirements into technical solutions

• Knowledge of data access control in GCP: IAM, authorized views, row-level/column-level security, policy tags

• Knowledge of Workforce Identity Federation (permissions for federated groups from Microsoft Entra ID)

• Dataplex Universal Catalog knowledge for classification, tagging, and cataloging

• Understanding of data classification and compliance (GDPR/RODO) for access management

• Knowledge of GCP integration with Informatica IDMC

• Knowledge of multi-cloud governance architecture including SAP Datasphere and Azure Databricks

• Strong communication, organizational, and analytical skills

Benefits

Not specified in the provided job posting.