Blog Spider

find on them RSS channels and store.
The main goal of the project was to learn new technologies and dive into the AKKA.net.

data mining and other Internet applications. It recursively downloads Web pages into local storage, as shown at picture.
The operations can be briefly described as the following four steps:
a. Take a set of seed URLs as initial task URLs.
b. Select a URL from task URLs and download the page from the Web.
c. Extract hyperlinks contained in the downloaded page and if desirable add the new URLs into task URLs in strategic order.
d. Repeat step b and c until either task URLs become empty or the crawl is stopped by the application.
The strategy to determine the order of task URLs to crawl is crawl scheduling. Given a time window T, different scheduling strategies can lead to very different sets of Web pages crawled in T. Scheduling Web pages URL requests Task URLs Seed URLs Initiate Get next URL Get page Extract URLs database Web Figure

must be in
crawler cluster.
Web - web application wich run
some job to crawl.
Tracker - this service which tell what we nee to crawl

cluster. It only has to be operate one occur cluster itself is upgraded and it’s not actually deployed as part of your application, so it should never have to be redeployed when you make no changes but it will need to be upgraded as occur that cluster gets upgraded.