Data Engineer

Department: Digital Factory

Role Overview

The Data Engineer is responsible for designing, building, and operating scalable, reliable, and secure data pipelines and data platforms that power digital products, analytics use cases, reporting, and data-driven services. Acting as a key contributor within cross-functional delivery teams, the role collaborates closely with Product, Backend, Frontend, Architecture, Data Analytics, and platform teams to deliver trusted data assets across ingestion, transformation, storage, and access layers.

At Supervising Associate level, the role leads complex data engineering delivery, sets technical direction, and strengthens team capability across initiatives.

Key Responsibilities

Lead technical delivery across complex data workstreams:

• Make design decisions and resolve cross-platform dependencies, governance risks, and escalations.
• Mentor engineers, enforce engineering and data standards, and drive improvements in architecture, quality, lineage, security, observability, cost, and technical debt management.

Data Pipeline & Platform Delivery

• Design, implement, and maintain batch and streaming data pipelines using Python and modern data engineering frameworks.
• Build reliable ingestion and transformation processes from internal and external data sources, including APIs, databases, files, and events.
• Develop reusable data processing components and frameworks following software engineering best practices.

Data Quality, Reliability & Observability

• Implement data validation, quality checks, and monitoring to ensure accuracy, completeness, timeliness, and trustworthiness of data.
• Write clean, testable code with unit and integration tests for data pipelines and processing components.
• Instrument pipelines with logging, metrics, alerts, and operational monitoring.

Performance, Scalability & Architecture

• Optimize data processing performance, storage layouts, query efficiency, and platform resource usage.
• Design pipelines for scalability and resilience using retries, idempotency, checkpointing, and backpressure patterns.
• Apply partitioning, caching, and parallelization strategies where appropriate.

Security, Governance & Continuous Improvement

• Implement secure data handling practices, including access controls, encryption, secrets management, and compliant data processing.
• Ensure alignment with enterprise, regulatory, privacy, data classification, retention, and lineage requirements.
• Support audits and data governance processes where required.

What Success in This Role Will Look Like

• Successful leadership of complex data delivery, stronger engineering capability, and consistent adoption of platform, quality, and governance standards.
• Reliable delivery and operation of scalable data pipelines and data platform components.
• Trusted data assets with strong accuracy, completeness, timeliness, and observability controls.
• Clean, testable, documented, and maintainable Python- and SQL-based data engineering solutions.
• Effective support for incidents, troubleshooting, root-cause analysis, and durable remediation.
• Mentorship and guidance for less experienced team members.

Qualifications & Experience

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, Data Engineering, Information Technology, or a related field.
• Professional experience building data pipelines and data platforms using Python.
• Strong experience with SQL and data modeling concepts, including dimensional, normalized, analytical, or lakehouse patterns.
• Hands-on experience with relational and analytical data stores, such as PostgreSQL, SQL Server, data warehouses, or data lakes.
• Experience with CI/CD, automated testing, and code reviews for data workloads.
• Expertise with cloud platforms, including Azure/Fabric and relevant data services.
• Ability to work effectively as part of a team, contribute to collective success, and remain open to feedback and coaching from more experienced colleagues.
• Strong understanding of data security, access control, observability, and production support practices.
• 7+ years of experience required.

Preferred Experience

• Experience with modern data frameworks and tools such as Spark, Airflow, dbt, Azure Data Factory, Microsoft Fabric, Databricks, or equivalent platforms.
• Hands-on experience with streaming and event-based data processing using Kafka, Azure Event Hubs, Service Bus, or similar technologies.
• Experience with lakehouse or warehouse platforms.