Data Lake Developer (Python, PySpark)

Quick Facts

• Role: Data Lake Developer (Python, PySpark)

Description

You will design and develop processes for acquiring, processing, and sharing data for solutions using AI. The role focuses on building Python (PySpark) components that integrate data with LLMs, AI agents, and business systems in Databricks. You will also create ETL/ELT pipelines, REST API integrations, data validation, monitoring, automated tests, and exception handling.

Responsibilities

• Design and build data ingestion, processing, and data sharing processes for AI solutions

• Develop Python (PySpark) components integrating data with LLMs, AI agents, and business systems in Databricks

• Process structured data and content from messages, documents, and attachments

• Build ETL/ELT processes, REST API integrations, and data validation mechanisms

• Ensure data quality, consistency, security, and monitoring

• Create automated tests; analyze errors and improve exception handling

• Collaborate with AI Engineers, Data Scientists, architects, and business experts

• Participate in solution architecture design, perform code reviews, and work in a Scrum team

Requirements

• Minimum 3 years of experience as a Data Engineer, Python Developer, or similar

• Very good Python and SQL knowledge

• Experience with ETL/ELT and integrating data from multiple sources

• Experience designing and using REST APIs

• Ability to work with structured and unstructured data

• Knowledge of Git, CI/CD, automated tests, and software development best practices

• Experience with applications using LLM models

• Communicative English

Benefits

• Interesting challenges and collaboration with experienced IT professionals

• Latest technology stack

• Interesting projects

• Enterprise stability

• Values-driven culture: community, simplicity, customer focus, responsibility, and trust