Recently for one of my projects, I needed to write my own custom crawler to crawl some of my favorite blogs. I am a recent Ruby convert and I am totally in love with the language. So i decided to have the crawler in Ruby. After some Googling around, I found out this very neat crawler, Spider written in Ruby by Mike Burns. You can find the source code here. The code is currently maintained by John Nagro.

The crawler is simple and yet has support for Amazon SQS and memcahed, so that you can run multiple crawler processes at the same time.  There are also some drawbacks:

1. The urls ae just crawled in the order they are discovered ( breadth-first search). There is no way to specify the ordering in which the urls should be searched.

2. There is no HTML parsing on top of the crawled data.

The first drawback was not much of a problem for my purposes, coz I needed to crawl all the web pages of a blog. Order of crawling was not a deal for me. The crawler has a way in which you can specify the urls in a regex form which are to be crawled. This served my purposes.

I needed HTML parsing though. But this was easy. I just used the hpricot gem to do HTML parsing on top of the crawled data:

Here is a sample code, on how to crawl a particular blog using Spider and using hpricot to print out the meta content on each page.

 
require 'rubygems'
require 'spider'
require 'hpricot'
 
Spider.start_at('http://simpliflying.com/') do |s|
  # Limit the pages to just this domain.
  s.add_url_check do |a_url|
    a_url =~ %r{^http://simpliflying.com.*}
  end
 
  # Handle 404s.
  s.on 404 do |a_url, err_code|
    puts "URL not found: #{a_url}"
  end
 
  # Handle 2xx.
  s.on :success do |a_url, resp, prior_url|
    puts "#{a_url}: #{resp.code}"
    doc = Hpricot(resp.body)
    (doc/"meta").each do |meta|
      puts meta.attributes['content']
    end
  end
 
  # Handle everything.
  s.on :any do |a_url, resp|
    puts "URL returned anything: #{a_url} with this code #{resp.code}"
  end
end
 

A very good Hpricot documentation is available here.