Senior Data Engineer (Databricks)

As a Senior Data Engineer, you will work with a team of technology experts on challenging projects across various industries, using cutting-edge technologies.

Projects

• Design and development of a universal data platform for global aerospace companies. This Azure- and Databricks-powered initiative combines diverse enterprise and public data sources. The platform is in the early stages of development, covering architecture and process design with freedom for technology selection.
• Data platform transformation for an energy management association. The project addresses data management challenges and improves user adoption, performance, and data integrity. The team is implementing a comprehensive data catalog using Databricks and Apache Spark/PySpark for simplified data access and governance. Secure integrations and enhanced data-quality monitoring using Delta Live Table tests establish trust in the platform and provide a foundation for future ML components.
• Design of data transformation and DataOps pipelines for a global car manufacturer. The project builds a data-processing system for real-time streaming and batch data to support process monitoring, analysis, and reporting, while exploring LLMs for chatbots and data analysis. Tasks include data cleaning, normalization, and data-model optimization for performance and accuracy.

Responsibilities

• Design and optimize scalable data-processing pipelines for streaming and batch workloads using Big Data technologies such as Databricks, Apache Airflow, and Dagster.
• Architect and implement end-to-end data platforms with high availability, performance, and reliability.
• Lead the development of CI/CD and MLOps processes to automate deployments, monitoring, and model lifecycle management.
• Develop and maintain applications that aggregate, process, and analyze data from diverse sources efficiently and at scale.
• Collaborate with Data Science teams on machine-learning projects, including text and image analysis, feature engineering, and predictive-model deployment.
• Design and manage complex data transformations using Databricks, DBT, and Apache Airflow, ensuring data integrity and consistency.
• Translate business requirements into scalable, efficient technical solutions with optimal performance and data quality.
• Ensure data security, compliance, and governance best practices across all data pipelines.

Requirements

• At least 5 years of commercial experience implementing, developing, or maintaining Big Data systems.
• Strong Python programming skills, including clean code and OOP design.
• Strong SQL skills, including performance tuning, query optimization, and data warehousing solutions.
• Experience designing and implementing data governance and data management processes.
• Deep expertise in Apache Airflow, Dagster, Databricks, and modern data orchestration and transformation tools.
• Experience implementing and deploying solutions in cloud environments; Azure is preferred.
• Knowledge of building and deploying Power BI reports and dashboards for data visualization.
• Excellent understanding of dimensional data and data-modeling techniques.
• Consulting experience and ability to guide clients through architectural decisions, technology selection, and best practices.
• Ability to work independently and take ownership of project deliverables.
• Familiarity with Spark, Azure Event Hub, or Kafka.
• Master’s or Ph.D. in Computer Science, Big Data, Mathematics, Physics, or a related field.

Benefits

• Supportive team of AI and Big Data specialists.
• International projects with global enterprises and startups.
• Flexible remote work, offices, and coworking spaces.
• Career paths, knowledge-sharing initiatives, language classes, and sponsored training or conferences, including Databricks and Anthropic training materials and certifications.
• Team-building events and an integration budget.
• Work anniversary, birthday, and milestone celebrations.
• Medical and sports packages, eye care, and well-being support, including psychotherapy and coaching.
• Full work equipment, including a laptop and necessary devices.
• Opportunities to build a personal brand through conference speaking, blog writing, and meetups.
• Dedicated onboarding buddy and a supportive, autonomous culture.