Remote Senior Big Data Engineer (Python)

We are looking for Data Engineers to work remotely for an Adtech company that leverages machine learning and data science to build an identity graph that can scale to reach millions of users via brands with programmatically selected households. The work includes scaling a big data asset that combines billions of transaction data points, including intent, conversions, and first-party data, into an identity graph.

We value technical excellence and provide the resources and time to deliver world-class code. This is a 100% remote position, working with team members in NYC. If you like solving hard and technically challenging problems, join us to create real-time, concurrent, globally distributed systems applications and services.

Quick Facts

• 100% remote position
• Work with team members in NYC
• Build real-time, concurrent, globally distributed systems applications and services
• Work with data sets exceeding billions of records

Role Details

• Design, develop, and maintain highly scalable data pipelines, ETL processes, and data models using Python, Spark, and other big data technologies in a cloud environment (AWS/GCP).
• Lead the integration of AI and LLMs into data products, working with data scientists to productionize machine learning models and agentic workflows efficiently and ethically.
• Champion and utilize AI-driven software development tools, such as GitHub Copilot, to boost development productivity, improve code quality, and accelerate delivery.
• Create and maintain reliable and scalable distributed data processing systems.
• Become a core maintainer of the data lake.
• Maintain the data lake by building searchable data sets for broader business uses.
• Scale, troubleshoot, and fix existing applications and services.
• Own a complex set of services and applications.
• Ensure data pipelines run 24/7.
• Lead technical discussions that improve tools, processes, or projects.
• Scale the identity graph to deliver impactful advertising campaigns.
• Scale the MLOps platform using both traditional ML and LLM/generative-AI-based applications.

Key Requirements

• 8+ years of professional software engineering experience, with a focus on data engineering in big data environments.
• Expert-level proficiency in Python and at least one other high-level programming language, such as Java, Scala, C++, or C#.
• Proven experience building and optimizing large-scale data pipelines using Apache Spark.
• Hands-on experience developing and deploying data solutions in a major cloud platform: AWS, GCP, or Azure.
• Experience working with AI, LLMs, agents, and/or generative AI technologies in product applications and for development productivity.
• Relevant technology experience in the advertising industry.
• Experience with Scala.
• Experience with big data visualization and analytics using OLAP tools.
• Familiarity with big data tools and frameworks such as MLFlow, dbt, Kafka, Airflow, or Databricks.
• Experience with large-scale data management formats such as Parquet, Delta Lake, or Iceberg.

Recruitment Process

• Short call with Varwise
• Initial call with hiring manager or team member
• Panel interview with tech team, including a live coding/live debugging session
• Panel interview with product team for culture and team fit
• Final decision meeting with the CTO