Data Platform Architect

Quick Facts

• Role: Data Platform Architect (Data Substrate)

• Work style: Remote (United States); occasional travel for in-person meetings

Description

Twilio is hiring an Architect to set the direction for a major shift in its data platform over the next 3–5 years. You will own technical strategy and reference architectures across lake/lakehouse, storage and compute engines, governance/catalog, and privacy-by-design.

Responsibilities

• Own the multi-year technical strategy for the data platform (lake/lakehouse, ClickHouse, Elasticsearch, governance) and keep components coherent as one platform.

• Define where each engine fits and document migration paths between engines (e.g., ClickHouse vs open table formats on object storage).

• Create and document major architecture decisions (ADRs/RFCs) for storage, open table formats, catalog strategy, compute engines, tenancy, data placement, and build-vs-adopt tradeoffs with cost/performance constraints at petabyte scale.

• Set reference architectures for deployment, tuning, scaling, and operations of shared storage and query engines, including capacity, cost, tenant isolation, and upgrade strategy.

• Define reliability model: availability targets, failure domains, disaster recovery, observability, and incident response patterns.

• Build privacy and compliance into the architecture (deletion/right-to-erasure, retention, data residency, PII classification) across storage layers, backups, derived datasets, and upstream streaming data.

• Drive data governance including classification, catalog/metadata, lineage, access control integrated with authentication and authorization, data contracts, and quality/integrity standards.

• Review designs across the org and act as escalation point on hard distributed-systems and performance problems.

• Mentor principal and senior engineers and grow architectural judgment across the organization.

• Partner with product, security, infrastructure, and consuming teams to translate business needs into platform roadmaps; say no or re-sequence when tradeoffs require it.

• Track emerging data technologies (e.g., streaming-first architectures, federated query, AI-ready data layers) and recommend adoption based on prototypes or benchmarks.

Requirements

• Bachelor’s or Master’s degree in Computer Science/Engineering (or related) or equivalent practical experience.

• 15+ years of software development or related experience; at least 8 years in architecture or senior technical leadership.

• Proven track record architecting and delivering large-scale data platforms and setting cloud strategy on AWS using Kubernetes, Terraform, and Helm.

• Deep expertise in analytical storage and compute: columnar/open table formats, lake/lakehouse architecture, partitioning, indexing, and query planning with petabyte-scale cost/performance impact.

• Breadth across analytical data storage/retrieval technologies with hands-on depth in at least two of: ClickHouse, Elasticsearch/OpenSearch, Spark/Trino/Flink.

• Working knowledge of streaming systems (e.g., Kafka) and how they interact with downstream storage.

• Practical experience designing data governance at scale: classification, catalog/metadata, lineage, and access control integrated with authentication/authorization.

• Practical experience implementing privacy and compliance (deletion, retention, residency) across a large heterogeneous data estate.

• Proficiency in one or more of Python, Java, and Go.

• Track record writing and driving architecture decisions across multiple teams through influence.

• Experience elevating engineering standards and mentoring senior engineers.

• Operational excellence: high availability, observability, and incident response in distributed systems.

• Strong written and verbal communication.

Benefits

• Competitive pay

• Generous time off

• Parental and wellness leave

• Healthcare

• Retirement savings program

• Paid sick time

• Paid personal time off

• Paid parental leave

Equity plan and corporate bonus eligibility may apply (role-dependent).