Data Engineer

Quick Facts

• Role: Data Engineer

Description

Design, implement, and evolve data engineering solutions on Google Cloud Platform using BigQuery, Cloud Storage, and Pub/Sub. Migrate and redesign data processing and ingestion flows from Oracle/on-premises to GCP, building batch and streaming pipelines with data quality controls, error handling, and retry mechanisms. Optimize BigQuery data structures and SQL queries, support ETL transformation logic with SQL and Python, and participate in testing, deployments, documentation, and incident resolution.

Responsibilities

• Design, implement, and develop Data Engineering solutions on GCP (BigQuery, Cloud Storage, Pub/Sub)

• Migrate and rebuild data processing and ingestion processes from Oracle/on-premises to GCP

• Build batch and streaming processing pipelines with data quality checks, error handling, and retry mechanisms

• Design and develop BigQuery data structures and models with performance, scalability, and cost optimization

• Analyze data and prepare logical and physical data models for Oracle, PostgreSQL, and GCP environments

• Create, analyze, and optimize SQL queries for large data volumes (including execution plan analysis)

• Analyze existing Informatica PowerCenter processes and migrate them to target solutions in GCP

• Implement transformation, validation, and data integration logic using SQL and Python

• Perform technical impact/dependency analyses and support specification alignment

• Prepare unit and integration tests; support system, performance, regression, and UAT testing

• Prepare changes for deployment, create deployment instructions, and support rollout and post-deployment verification

• Diagnose and resolve incidents related to data processing and data quality

• Maintain technical documentation and work with code repositories and CI/CD processes following DevOps practices

Requirements

• Minimum 2 years of commercial experience designing and implementing data integration or data processing processes

• Practical experience with Google Cloud Platform, especially BigQuery, Cloud Storage, and Pub/Sub

• Working knowledge of Oracle, including creating and optimizing SQL queries for large data volumes

• Developer experience with Informatica PowerCenter to analyze and modify ETL processes

• Knowledge of data modeling concepts and data processing pipeline design

• Ability to optimize SQL queries for performance

• Knowledge of data quality, consistency, and validation topics

• Experience preparing and running tests and creating technical documentation