Ruby HTML parsing has been keeping me quite entertained frustrated lately, so I thought I'd share some thoughts. There are a couple of instance in your rails app when you'll want to parse HTML
- Automated functional/controller testing
- Screen scraping
Functional Testing
The standard method of verifying aspects of resulting HTML in your functional test is HTML::Selector. It's simple, powerful, and baked in. Agile Rails 2nd does a great job of explaining how it's used in functional tests.
RUBY:
-
def test_add_no_name
-
post :add, :color => { :name => '', :hex => '#123456' }
-
assert_template 'add'
-
assert_select "div[id=errorExplanation]" do
-
assert_select "ul" do
-
assert_select "li", 'Name is not present'
-
end
-
end
-
end
Scraping
Several options are available, but oh so popular is why's Hpricot. It's fast and enjoyable (although I experienced no joy while learning how to use it =) It also happens to be used in some of the other scraping/navigating libraries (WWW::Mechanize [rdoc] and scRUBYt!).
Some Thoughts...
So if your just concerned with testing use HTML::Selector and the built in asserts. If you have to do very basic screen scraping I would also suggest going with HTML::Selector (as long as speed is not an issue and the scraping is basic) with open-uri or curb for fetching the pages.
For more serious screen scraping bust out Hpricot and if you need to navigate pages via automation use WWW::Mechanize (Mechanize also uses Hpricot so all of that Hpricot knowledge you've absorbed is directly applicable. Mechanize is Hpricot with the ability to click). Don't worry about scRUBYt!. It's more of a pain to figure out than it's worth (but maybe I'm wrong about it. Any good examples/write-ups?).
Hpricot with CSS selector
RUBY:
-
divs = (doc/"div[@style*='font-weight:'][text()*='$'").inner_html
-
divs.each do |div|
-
if div =~ /\$[0-9]?[0-9]\.[0-9][0-9]/
-
self.price = div.to_s.sub('$', '')
-
end
-
end
Hpricot search with XPath
RUBY:
-
require 'hpricot'
-
require 'open-uri'
-
doc = Hpricot(URI.parse("http://google.com/").read)
-
-
doc.search("/html/body//p")
-
doc.search("//p")
-
doc.search("//p/a")
-
doc.search("//a[@src]")
-
doc.search("//a[@src='google.com']")
Using Mechanize to do a search on google
RUBY:
-
require 'rubygems'
-
require 'mechanize'
-
-
agent = WWW::Mechanize.new
-
agent.user_agent_alias = 'Mac Safari'
-
page = agent.get("http://www.google.com/")
-
search_form = page.forms.with.name("f").first
-
search_form.q = "Hello"
-
search_results = agent.submit(search_form)
-
puts search_results.body
Note that Hpricot lets you use a CSS method of selecting and an XPATH method. Use XPATH if you already have experience otherwise the CSS method is more intuitive.
If you go with XPATH grab the XPather firefox plugin and use it with the DOM Inspector. Also, it works with the firebug firefox plugin. I'm still in awe that it worked when I tried. :) To do this, use firebug to "inspect", choose an element, right click on the page and select "Show in XPather". XPather will open with the selected element locked and loaded.
Finally, if your a Hpricot wiz forget about HTML::Selector and put Hpricot to work for view validation in your functional tests. See this great write up, Testing your Rails views with Hpricot, which demonstrates this elegant solution.
RUBY:
-
assert_equal "My Funky Website", tag('title')
-
assert_equal 20, tags('div.boxout').size
-
assert_equal 'visible', element('div#site_container').attributes['class']
I simply downloaded the 3 Opensolaris files (Nevada release), used cat to concatenate them into a single file, selected Solaris 10 in parallels, selected the image, and it installed without any hang-ups. Ahh yes, and I had to go back and change the memory allocated as it has a 768mb minimum requirement.
ruby-debug = cool + very useful. Many have said it before and I have blissfully ignored until in a sticky situation that helped me realize I need to be able to poke and prod a little more than spitting out to the log in rails. After all poke and prod is my favorite way to learn.