Staff Software Engineer, Web-Scale Crawling & Ingestion

Nebius in London is seeking a Staff Software Engineer to build the content acquisition and crawling infrastructure for a world-scale search platform designed for agentic AI. You will design distributed crawling, scheduling, and ingestion systems capable of handling billions of URLs and terabytes of data.

Collaborate with indexing and ML teams, improve observability, optimize resource use, and drive safe experimentation with crawling strategies.