Senior Data Engineer

Quick Facts

• Role: Senior Data Engineer

• Focus: Cloud-native, near-real-time data platforms using Databricks and AWS

Description

Design and scale greenfield-to-production data platforms with near-real-time, event-driven ingestion pipelines. Modernize large-scale Spark and PySpark workloads, enforce Lakehouse/Data Lake standards (including medallion architecture and data contracts), and improve reliability, observability, and data quality. Drive engineering excellence through automation, CI/CD best practices, and adoption of AI tooling within the data platform ecosystem.

Responsibilities

• Design and build scalable cloud-native data platforms from greenfield to production

• Implement near-real-time ingestion pipelines using event-driven patterns

• Define and enforce data platform standards (Lakehouse principles, medallion architecture, data contracts)

• Refactor, optimize, and modernize Spark and PySpark scripts

• Introduce code quality, testing, and CI/CD best practices across data pipelines

• Drive adoption of AI tooling and agentic workflows within the data engineering team

• Ensure data quality, observability, and reliability across pipelines and platforms

• Design and deliver self-service tooling and microservices that abstract platform complexity

• Collaborate with scientific and engineering teams

• Monitor platform performance, reliability, and cloud infrastructure costs

Requirements

• 5+ years of data engineering experience

• 3+ years production experience with Databricks and Delta Lake

• Strong expertise in medallion architecture and event-driven asynchronous pipeline design

• Advanced proficiency in Python, PySpark, and SQL

• Deep hands-on Databricks experience, including Unity Catalog and Lakeflow Jobs/Workflows/job optimization

• Strong AWS expertise including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway

• Infrastructure as code and CI/CD with Terraform and Databricks Asset Bundles

• Git workflows with pull request reviews and automated testing

• Secure CI pipeline configuration using OIDC or service principals

• Practical MLflow experience (tracking, model registration, reproducibility)

• Understanding of data quality, lineage/provenance, schema contracts, and versioning

• Data quality frameworks experience: Great Expectations, Soda, pyDeequ, or Databricks Labs DQX

• Monitoring, observability, and cloud cost optimization experience

• Upper-Intermediate English or higher

Benefits

• Opportunity to contribute to breakthrough clean-energy research while building scalable data platforms

• Engineering excellence focus: automation, reliability, and AI adoption in the data platform ecosystem

• Work closely with scientific and engineering teams to deliver platform capabilities