Senior Software Engineer

Quick Facts

• Role: Senior Software Engineer (Data & Insights)

• Focus: PiB-scale batch and streaming data processing, real-time insights, and data quality

Description

You will lead the design and ownership of scalable data analytics and insight platforms that process petabyte-scale data daily. Responsibilities include building fault-tolerant pipelines and core datasets with Spark and Flink, delivering real-time insights, and implementing data quality and anomaly detection with strict performance and reliability standards. You will collaborate cross-functionally and use AI practices across coding, testing, data processing, and validation.

Responsibilities

• Design, build, and own the optimal data processing architecture for new data and analytics pipelines, dashboards, and metrics using Azure and open-source technologies

• Build core datasets as well as scalable and fault-tolerant pipelines

• Build data anomaly detection and data quality checks; optimize pipelines for ideal compute and storage systems while guaranteeing high quality and SLA datasets

• Understand data compliance and data governance

• Collaborate with data scientists and product managers to design and develop end-to-end data assets

• Hands-on ownership of at least one data process technology (Spark or Flink) and at least one message queue technology (Kafka or EventHub)

• Hands-on work with data storage technologies (HDFS, Azure Blob, etc.)

• Leverage AI technology in coding, testing, data processing, and data validation

Requirements

• Bachelor’s degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in C, C++, C#, Java, JavaScript, or Python (or equivalent experience)

• 6+ years backend service development; 2+ years batch or streaming data processing for PiB-level data

• Domain expert on Spark or Flink to handle PiB-level data every day

• Experience optimizing data pipeline performance

• Experience using AI in data processing and data validation

• Fluence/verbal and written English communication

• Preferred: 6+ years in one or more coding languages (C#, C++, Java); familiarity with Spark/Flink; familiarity with Kafka/EventHub; cloud experience such as Azure; storage experience such as HDFS/Azure Blob; AI in coding/testing; data compliance/governance experience (plus)

Benefits

• Certain roles may be eligible for benefits and other compensation

• Base pay range (U.S.): USD $119,800–$234,700 per year; San Francisco Bay area and New York City metro: USD $160,200–$261,000 per year