Software Engineer III (Python, Databricks, AWS)

Quick Facts

• Role: Software Engineer III (Python, Databricks, AWS)
• Focus: Data engineering for ingestion, pipelines, governance, reliability, and performance on the Databricks lakehouse (AWS)

Description

You’ll serve as a seasoned core technical contributor on an agile team, enhancing and delivering secure, stable, scalable data collection, storage, access, and analytics solutions. The role centers on maintaining critical data pipelines and architectures, including resilient batch and streaming ingestion from AWS sources into Databricks with CDC where needed.

Responsibilities

• Implement resilient ingestion at scale from AWS sources into Databricks (batch + streaming), including CDC when required.
• Build maintainable pipelines using Delta Live Tables and/or standard Databricks Jobs with modular structure, testing, and documentation.
• Productionize workloads using Databricks Workflows/Jobs with retries, checkpointing, idempotency, and safe re-runs.
• Enforce governance-by-design: least privilege, PII/data classification, auditing, lineage/metadata, and controlled sharing/consumption.
• Tune Spark/Delta workloads and manage performance and cost (cluster sizing, storage layout, and job/warehouse spend).
• Implement CI/CD and infrastructure-as-code using Terraform (preferred) for Databricks + AWS resources with environment promotion.
• Add unit/integration tests, data quality checks, contract testing, and support replay/backfill procedures.
• Follow version control and code review best practices with clear documentation and runbooks.
• Use enterprise-authorized AI coding assist tools to improve delivery speed and code quality, while validating outputs via peer review, automated testing, and secure coding standards.

Requirements

• 5+ years hands-on Software Development Life Cycle experience; formal training or certification in software engineering concepts.
• Experience building and maintaining data engineering pipelines.
• Hands-on experience operating a Databricks Lakehouse hosted on AWS.
• Delta Lake experience (ACID tables, partitioning, schema evolution).
• Spark on Databricks experience (performance tuning, cluster sizing, skew mitigation, joins, caching, file sizing).
• Experience with batch and streaming pipelines (Structured Streaming, incremental processing, backfills, late-arriving data).
• AWS fundamentals for data platforms (S3, IAM, KMS, VPC/security groups, logging/auditing).
• Databricks governance/security controls experience (e.g., Unity Catalog, table/column permissions).
• Reliability practices (monitoring/alerting, incident response, RCA, SLO/SLA management).
• Knowledge of responsible AI usage in engineering workflows with secure handling of inputs/outputs.
• Hands-on experience with enterprise-authorized AI-assisted development tools and the ability to critically evaluate and refine AI-generated outputs.

Benefits

• Competitive total rewards package; base salary determined by role, experience, skills, and location.
• Possible commission-based pay and/or discretionary incentive compensation, including cash and/or forfeitable equity.
• Comprehensive health care coverage and on-site health & wellness centers.
• Retirement savings plan.
• Backup childcare and tuition reimbursement.
• Mental health support and financial coaching.
• Additional total compensation and benefits details provided during hiring.