AI Data Engineer

Quick Facts

• Remote (100%) or Warsaw
• Availability: start immediately or within up to 1 month

Description

You will design and develop data ingestion, processing, and publishing processes for AI solutions. The work includes preparing knowledge sources for RAG systems, building indexing and semantic search mechanisms, generating and updating vector representations (embeddings), and optimizing retrieval to deliver the right context to AI models in an Azure environment.

Responsibilities

• Design and develop data acquisition, processing, and delivery processes for AI solutions
• Build and optimize data pipelines used by AI-based systems
• Prepare and transform data from multiple sources, including unstructured data
• Develop indexing and information retrieval mechanisms used in RAG solutions
• Design processes to generate and update vector representations (embeddings)
• Integrate data sources with AI components and cloud services
• Optimize search processes and context delivery to AI models
• Ensure data quality, consistency, and freshness for the developed solutions
• Automate data processing and dataset update workflows
• Analyze pipeline performance and identify improvement opportunities
• Collaborate with AI Engineering and Data Engineering teams on data architecture
• Create technical documentation and evolve data processing standards

Requirements

• Minimum 4 years of commercial experience in Data Engineering or a related field
• Very strong Python knowledge and experience building data processing workflows
• Practical experience designing and developing data pipelines
• Knowledge of AI solutions and how to prepare data for systems using artificial intelligence
• Experience designing or developing RAG solutions
• Practical knowledge of vector databases and semantic search mechanisms
• Knowledge of embedding creation, storage, and update processes
• Experience with Microsoft Azure and services used for data storage and processing
• Ability to integrate data from different sources and systems
• Experience processing structured and unstructured data
• Strong understanding of data quality, performance optimization, and process scalability
• Ability to independently analyze technical problems and propose appropriate solutions

Benefits

• Opportunity to work on a project combining Data Engineering with modern AI solutions
• Help build the data infrastructure that underpins RAG systems
• Tackle complex topics: semantic search, data processing, and AI integration
• Contribute to data architecture and how data is prepared and delivered