Overview
At the Web Data Platform Crawler, we build the global web scale Index and the platform that supports it. We are looking for a Principal Software Engineer to help us further scale our platform in Web Crawler team that is fundamental to all Search and AI scenarios.
Today we crawl tens of Billions URLs/day. This is the go-to service for all web crawling needs across Microsoft, both for consumer and enterprise needs. We are on a journey to containerize the service and build the next version of crawler, which can increase the throughput, reach every URL on the web and is deployed in every region of the world, while being intelligent about minimizing the load on websites being crawled.
The system is expected to maintain a fine balance with discovering the latest pages on the web, maintaining freshness of documents in the Index, while avoiding inundating the web servers with crawl requests. It is a distributed platform scalable with Machine Learning (ML) models aiding effective resource usage. With the advent of Large Language Models (LLMs), web scale data has become critical for training needs, in addition to serving use cases.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Qualifications
Required Qualifications:
- Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR equivalent experience.
- 2+ years of hands-on experience working with big data technologies (e.g., Hadoop, Spark, Kafka) or building highly scalable backend systems (e.g., distributed systems, microservices architecture, cloud-native services).
Other Requirements:
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:
- Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Preferred Qualifications:
- Bachelor's Degree in Computer Science
- OR related technical field AND 10+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR Master's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR equivalent experience.
- 4+ years of hands-on experience working with big data technologies (e.g., Hadoop, Spark, Kafka) or building highly scalable backend systems (e.g., distributed systems, microservices architecture, cloud-native services).
Software Engineering IC5 - The typical base pay range for this role across the U.S. is USD $139,900 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
Microsoft will accept applications for the role until June 17, 2025
#MicrosoftAI #BING
Responsibilities
- Collaborates with appropriate stakeholders to determine user requirements for a scenario.
- Drives identification of dependencies and the development of design documents for a product, application, service, or platform.
- Creates, implements, optimizes, debugs, refactors, and reuses code to establish and improve performance and maintainability, effectiveness, and return on investment (ROI).
- Leverages subject-matter expertise of product features and partners with appropriate stakeholders (e.g., project managers) to drive a workgroup's project plans, release plans, and work items.
- Acts as a Designated Responsible Individual (DRI) and guides other engineers by developing and following the playbook, working on call to monitor system/product/service for degradation, downtime, or interruptions, alerting stakeholders about status and initiates actions to restore system/product/service for simple and complex problems when appropriate.
- Proactively seeks new knowledge and adapts to new trends, technical solutions, and patterns that will improve the availability, reliability, efficiency, observability, and performance of products while also driving consistency in monitoring and operations at scale.
