Senior Web Scraping & Data Engineer

Quick Facts

• Role: Senior Web Scraping & Data Engineer

Description

We are looking for an experienced Senior Web Scraping & Data Engineer to build scalable systems for collecting data from diverse online sources. The role focuses on reliable, high-performance web scraping and transforming public, unstructured information into datasets for research and investment teams, using scraping, APIs, data processing, automation, and AI-driven solutions.

Responsibilities

• Design and develop scalable web scraping and data collection systems
• Build scrapers and crawlers for websites with different structures and technologies
• Collect data from websites, APIs, and dynamic online sources
• Clean, transform, and standardize raw data
• Improve accuracy, speed, coverage, and reliability of scraping solutions
• Troubleshoot website changes, data quality, and performance issues
• Build automated pipelines for processing collected data
• Explore AI/ML and agentic technologies for extraction and validation
• Support infrastructure for large-scale data collection
• Implement monitoring and data quality controls
• Work with researchers and analysts to understand data needs
• Explore new technologies and hard-to-source data

Requirements

• 6+ years of experience in software or data engineering
• Strong experience with web scraping, crawling, and automated data collection
• Excellent Python skills, including pandas
• Strong SQL and database knowledge
• Good understanding of HTML, JavaScript, APIs, and web technologies
• Knowledge of networking concepts related to web data collection
• Experience building data pipelines
• Understanding of algorithms, data structures, and data quality
• Experience with large-scale or distributed data collection
• Strong troubleshooting and problem-solving skills
• Interest in AI/LLMs and agentic technologies
• Strong communication and collaboration skills

Benefits

Not specified in the posting.