Lead Data Engineer

Quick Facts

• Role: Lead Data Engineer

Description

Lead data engineering workstreams and provide hands-on implementation for resilient, scalable batch and streaming pipelines across multi-functional squads. Drive migrations from legacy schedulers to modern, cloud-native workflow orchestration while establishing data governance, lineage tracking, metadata management, and role-based security (e.g., Unity Catalog). Optimize performance, query efficiency, and platform cost, and implement data virtualization and unified integration across disparate enterprise systems.

Responsibilities

• Lead data engineering workstreams and hands-on technical implementation across multi-functional squads

• Design and deploy resilient, scalable batch and streaming pipelines and data processing frameworks

• Drive platform migrations from legacy schedulers to modern cloud-native workflow orchestration tools

• Establish data governance policies, lineage tracking, metadata management, and role-based security controls

• Implement data virtualization and unified integration patterns across disparate multi-source enterprise systems

• Lead performance tuning, query optimization, cost management, and operational reliability across the data platform

Requirements

• 6–9 years of professional experience in data engineering and enterprise data platform delivery

• Expert-level hands-on experience with Databricks and Apache Spark

• Deep knowledge of Data Lakehouse architecture (Delta Lake, medallion architecture, schema enforcement)

• Advanced proficiency in Python, SQL, and PySpark for data transformation and distributed computing

• Strong background in workflow orchestration tools (Airflow, Databricks Workflows, Dagster) and scheduler migration projects

• Practical experience implementing data governance, cataloging, metadata, and data access security frameworks

• Proven track record architecting data virtualization and integration solutions across heterogeneous cloud and on-premise sources

Benefits

Not specified in the provided posting.