Ruby HTML parsing has been keeping me quite entertained frustrated lately, so I thought I'd share some thoughts. There are a couple of instance in your rails app when you'll want to parse HTML
- Automated functional/controller testing
- Screen scraping
Functional Testing
The standard method of verifying aspects of resulting HTML in your functional test is HTML::Selector. It's simple, powerful, and baked in. Agile Rails 2nd does a great job of explaining how it's used in functional tests.
RUBY:
-
def test_add_no_name
-
post :add, :color => { :name => '', :hex => '#123456' }
-
assert_template 'add'
-
assert_select "div[id=errorExplanation]" do
-
assert_select "ul" do
-
assert_select "li", 'Name is not present'
-
end
-
end
-
end
Scraping
Several options are available, but oh so popular is why's Hpricot. It's fast and enjoyable (although I experienced no joy while learning how to use it =) It also happens to be used in some of the other scraping/navigating libraries (WWW::Mechanize [rdoc] and scRUBYt!).
Some Thoughts...
So if your just concerned with testing use HTML::Selector and the built in asserts. If you have to do very basic screen scraping I would also suggest going with HTML::Selector (as long as speed is not an issue and the scraping is basic) with open-uri or curb for fetching the pages.
For more serious screen scraping bust out Hpricot and if you need to navigate pages via automation use WWW::Mechanize (Mechanize also uses Hpricot so all of that Hpricot knowledge you've absorbed is directly applicable. Mechanize is Hpricot with the ability to click). Don't worry about scRUBYt!. It's more of a pain to figure out than it's worth (but maybe I'm wrong about it. Any good examples/write-ups?).
Hpricot with CSS selector
RUBY:
-
divs = (doc/"div[@style*='font-weight:'][text()*='$'").inner_html
-
divs.each do |div|
-
if div =~ /\$[0-9]?[0-9]\.[0-9][0-9]/
-
self.price = div.to_s.sub('$', '')
-
end
-
end
Hpricot search with XPath
RUBY:
-
require 'hpricot'
-
require 'open-uri'
-
doc = Hpricot(URI.parse("http://google.com/").read)
-
-
doc.search("/html/body//p")
-
doc.search("//p")
-
doc.search("//p/a")
-
doc.search("//a[@src]")
-
doc.search("//a[@src='google.com']")
Using Mechanize to do a search on google
RUBY:
-
require 'rubygems'
-
require 'mechanize'
-
-
agent = WWW::Mechanize.new
-
agent.user_agent_alias = 'Mac Safari'
-
page = agent.get("http://www.google.com/")
-
search_form = page.forms.with.name("f").first
-
search_form.q = "Hello"
-
search_results = agent.submit(search_form)
-
puts search_results.body
Note that Hpricot lets you use a CSS method of selecting and an XPATH method. Use XPATH if you already have experience otherwise the CSS method is more intuitive.
If you go with XPATH grab the XPather firefox plugin and use it with the DOM Inspector. Also, it works with the firebug firefox plugin. I'm still in awe that it worked when I tried. :) To do this, use firebug to "inspect", choose an element, right click on the page and select "Show in XPather". XPather will open with the selected element locked and loaded.
Finally, if your a Hpricot wiz forget about HTML::Selector and put Hpricot to work for view validation in your functional tests. See this great write up, Testing your Rails views with Hpricot, which demonstrates this elegant solution.
RUBY:
-
assert_equal "My Funky Website", tag('title')
-
assert_equal 20, tags('div.boxout').size
-
assert_equal 'visible', element('div#site_container').attributes['class']
I learned an important lesson this weekend while equipping the mobile transportation unit with a hook up for the iPod. Don’t cut a USB cable and plan to reconnect the wires. They are very small and a good portion of life could be spent on this task. Amazingly enough everything worked after the procedure. A wire stripper that works with small wires would have been nice though. Thanks Angad and Maks for the useful car / stereo links.
