WIRE, an open-source information retrieval environment (OSWIR 2005 Compiegne)

The document describes WIRE, an open source web crawler project. It was created to allow high-performance crawling of subsets of the web between 1-50 million pages. It uses asynchronous DNS and HTTP requests to require small memory and processing resources. The document outlines the WIRE project, describes the web crawler architecture and scheduling approach, and concludes the presentation.