<!-- <b>Hello</b><%= @article.author %> -->

The above statement will not stop the evaluation of article.author. Instead, @article.author will still be evaluated and placed as a comment in the generated html file. Say the author was Balpreet, then we have a html comment:

<!-- <b>Hello</b>Balpreet -->

So, what if we want to just comment stuff out, so that the embedded ruby statements are not evaluated at all. There are basically two ways that I think we can achieve this:

1. Insert block level comments using embedded ruby.
<%
=begin %>
<b>Hello</b><%= @article.author %><%
=end %>

2. Insert conditional
<% if false %>
<b>Hello</b><%= @article.author %>
<% end %>

One of the dependencies for this was hpricot. Everything was working fine till one day, I updated my hpricot gem and did a cleanup of my rubygems. I started getting an error:

balpreet-pankajs-macbook:digg balpreetpankaj$ ruby bloo.rb
/Library/Ruby/Site/1.8/rubygems.rb:142:in `activate': can't activate hpricot (= 0.6, runtime), already activated hpricot-0.6.164 (Gem::Exception)
from /Library/Ruby/Site/1.8/rubygems.rb:158:in `activate'
from /Library/Ruby/Site/1.8/rubygems.rb:157:in `each'
from /Library/Ruby/Site/1.8/rubygems.rb:157:in `activate'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
from /Library/Ruby/Gems/1.8/gems/activesupport-2.1.2/lib/active_support/dependencies.rb:510:in `require'
from /Library/Ruby/Gems/1.8/gems/activesupport-2.1.2/lib/active_support/dependencies.rb:355:in `new_constants_in'
from /Library/Ruby/Gems/1.8/gems/activesupport-2.1.2/lib/active_support/dependencies.rb:510:in `require'

After a little digging arnd with rubygems, I figured out that the rfeedparser has a requirement of running with a hpricot gem with version = 0.6. I changed that to >= 0.6 and things started working fine. I was using version 0.9.951 of rfeedparser. If you use rfeedparser and run into a similar problem, this is how you fix it:

1) Update the gemspec for rfeedparser. Go to your gem installation dir. You can find this by doing:
gem env gempath
Then edit the $gemPATH/specifications/rfeedparser0.9.951.gemspec and update all references of hpricot from "=0.6" to ">=0.6".
2). Go to $GEMPATH/gems/rfeedparser-0.9.951/lib/rfeedparser.rb and again update the reference of hpricot.

This should get you rolling with all new versions of hpricot.

The crawler is simple and yet has support for Amazon SQS and memcahed, so that you can run multiple crawler processes at the same time.  There are also some drawbacks:

1. The urls ae just crawled in the order they are discovered ( breadth-first search). There is no way to specify the ordering in which the urls should be searched.

2. There is no HTML parsing on top of the crawled data.

The first drawback was not much of a problem for my purposes, coz I needed to crawl all the web pages of a blog. Order of crawling was not a deal for me. The crawler has a way in which you can specify the urls in a regex form which are to be crawled. This served my purposes.

I needed HTML parsing though. But this was easy. I just used the hpricot gem to do HTML parsing on top of the crawled data:

Here is a sample code, on how to crawl a particular blog using Spider and using hpricot to print out the meta content on each page.

 
require 'rubygems'
require 'spider'
require 'hpricot'
 
Spider.start_at('http://simpliflying.com/') do |s|
  # Limit the pages to just this domain.
  s.add_url_check do |a_url|
    a_url =~ %r{^http://simpliflying.com.*}
  end
 
  # Handle 404s.
  s.on 404 do |a_url, err_code|
    puts "URL not found: #{a_url}"
  end
 
  # Handle 2xx.
  s.on :success do |a_url, resp, prior_url|
    puts "#{a_url}: #{resp.code}"
    doc = Hpricot(resp.body)
    (doc/"meta").each do |meta|
      puts meta.attributes['content']
    end
  end
 
  # Handle everything.
  s.on :any do |a_url, resp|
    puts "URL returned anything: #{a_url} with this code #{resp.code}"
  end
end
 

A very good Hpricot documentation is available here.