Data Engineer

Quick Facts

• Role: Data Engineer (B2B, hybrid)
• Focus: Migration of Oracle/on-premise data processing to GCP using BigQuery, Pub/Sub, and Cloud Storage

Description

Design, implement, and evolve data engineering solutions on Google Cloud Platform, primarily using BigQuery, Cloud Storage, and Pub/Sub. Migrate and rebuild data processing and loading processes from Oracle/on-premise to GCP, covering both batch and streaming pipelines with data quality controls, retries, and error handling. Support SQL performance tuning for large volumes, implement transformation/integration logic in SQL and Python (as needed), and handle testing and deployment verification.

Responsibilities

• Design and implement batch and streaming data processing pipelines
• Implement data quality mechanisms, retries, and error handling
• Design data structures and models in BigQuery for performance, scalability, and processing/storage cost
• Analyze data and prepare logical/physical models for Oracle, PostgreSQL, and GCP data layers
• Create, analyze, and tune SQL queries; review execution plans and identify bottlenecks
• Analyze existing Informatica PowerCenter ETL processes and migrate them to target GCP solutions
• Implement transformation, validation, and integration logic using SQL and Python when required
• Perform technical analysis, impact/dependency analysis, and participate in solution specification alignment
• Prepare unit/integration tests and support system testing, performance testing, regression, and UAT
• Prepare deployment changes and instructions; assist with rollout and post-deployment verification
• Diagnose and resolve incidents; participate in root-cause analysis for processing and data quality issues
• Maintain technical documentation and work with code repositories and CI/CD following DevOps practices

Requirements

• Minimum 2 years of commercial experience designing and implementing data integration or processing pipelines, with predominantly hands-on GCP experience (BigQuery, Cloud Storage, Pub/Sub)
• Practical experience with Oracle, including creating and optimizing SQL queries for large data volumes
• Developer experience with Informatica PowerCenter enabling independent analysis and modification of ETL processes (PowerCenter is not the main technology of the role)