Data Engineer Lead

Quick Facts

• Role: Data Engineer Lead (hands-on)
• Focus: Production-grade data pipelines and platforms; end-to-end data product delivery

Description

The Data Engineer Lead is a hands-on engineering role responsible for designing, building, and operating production-grade data pipelines and platforms. You will deliver end-to-end data products from requirements through production in an Agile/Scrum framework, translating business needs into technical designs with SLAs and ensuring sound architecture, engineering standards, data quality, and observability. You will also leverage GenAI tooling (including Claude Code and Snowflake Cortex) as a productivity multiplier across the development lifecycle.

Responsibilities

• Design, build, and operate production-grade PySpark and Python pipelines on AWS EMR, Glue, and S3 with integrated data quality checks and observability
• Own end-to-end delivery of data products from requirements through production (sprint planning, CI/CD, release management, production readiness) in Agile/Scrum
• Translate business and product requirements into technical designs: pipeline structure, data models, and SLAs
• Apply architecture and design patterns; evaluate trade-offs and explain rationale
• Set and enforce engineering standards: coding conventions, data quality frameworks, reusable pipeline patterns, observability hooks
• Conduct thorough code reviews and pair with engineers on hard problems
• Leverage GenAI tooling (Claude Code CLI, Snowflake Cortex) to improve code quality and delivery speed
• Collaborate with product owners to refine requirements and push back when scope is technically unworkable
• Communicate design decisions and delivery updates to both technical and non-technical stakeholders
• Mentor junior engineers through review, pairing, and knowledge sharing
• Produce technical documentation as a standard: ADR, runbooks, data dictionaries, pipeline lineage notes
• Evaluate and drive adoption of productivity tooling and best practices

Requirements

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related technical field
• 7+ years of hands-on data engineering experience delivering production data pipelines end-to-end
• Strong command of PySpark, Python, and SQL, including Spark optimization, partition management, and query performance tuning in Snowflake
• Hands-on experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM, CloudWatch
• Working knowledge of architecture and design principles (lakehouse patterns, data modeling, fault-tolerant ingestion)
• Strong analytical and problem-solving skills for decomposing requirements and making well-justified trade-offs
• Clear communication skills to explain technical decisions in plain language to peers and business stakeholders
• Self-motivated learner who stays current with evolving tools and best practices

Benefits

• Competitive compensation
• Comprehensive benefits
• Hybrid flexibility; in-person collaboration expected primarily in the office