Data Engineer (Databricks)

As a Data Engineer, you will work with technology experts on challenging projects across various industries, using cutting-edge technologies.

Projects

• Design and development of a universal data platform for global aerospace companies. This Azure- and Databricks-powered initiative combines diverse enterprise and public data sources. The platform is in the early stages of development, covering architecture and process design and allowing freedom in technology selection.
• Data platform transformation for an energy management association body. The project addresses data management challenges to improve user adoption, performance, and data integrity. The team is implementing a comprehensive data catalog using Databricks and Apache Spark/PySpark for simplified data access and governance. Secure integration solutions and enhanced data-quality monitoring using Delta Live Table tests establish trust in the platform and provide a basis for further ML component development.
• Design of data transformation and DataOps pipelines for a global car manufacturer. The project builds a data-processing system for real-time streaming and batch data, supporting process monitoring, analysis, and reporting. It also explores LLMs for chatbots and data analysis. Key tasks include data cleaning, normalization, and data-model optimization for performance and accuracy.

Your main responsibilities

• Design scalable data-processing pipelines for streaming and batch processing using Big Data technologies such as Databricks, Airflow, and/or Dagster.
• Contribute to CI/CD and MLOps process development.
• Develop applications to aggregate, process, and analyze data from diverse sources.
• Collaborate with the Data Science team on machine-learning projects, including text/image analysis and predictive-model development.
• Develop and organize data transformations using Databricks/DBT and Apache Airflow.
• Translate business requirements into technical solutions and ensure optimal performance and quality.

What you’ll need to succeed in this role

• At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems.
• Strong Python programming skills, including clean code and OOP design.
• Experience designing and implementing data governance and data management processes.
• Familiarity with Airflow or Dagster, Databricks, Spark, and DBT.
• Experience implementing and deploying solutions in cloud environments, preferably Azure.
• Knowledge of building and deploying Power BI reports and dashboards for data visualization.
• Excellent understanding of dimensional data and data-modeling techniques.
• Excellent communication skills and consulting experience with direct client interaction.
• Ability to work independently and take ownership of project deliverables.
• Master’s or Ph.D. in Computer Science, Data Science, Mathematics, Physics, or a related field.

Perks & benefits

• Work in a supportive team of AI and Big Data enthusiasts.
• Engage with global enterprises and startups on international projects.
• Flexible remote, office, and coworking work arrangements.
• Career paths, knowledge-sharing initiatives, language classes, and sponsored training or conferences, including access to industry training materials and certifications.
• Team-building events and an integration budget.
• Work-anniversary, birthday, and milestone celebrations.
• Medical and sports packages, eye care, and well-being support, including psychotherapy and coaching.
• Full work equipment, including a laptop and necessary devices.
• Opportunities to build a personal brand through conferences, blog writing, and meetups.
• Smooth onboarding with a dedicated buddy.