scraping code

Never been to DZone Snippets before?

Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

« Newer Snippets

Older Snippets »

Showing 1-2 of 2 total RSS

Mechanize / Hpricot / Scraping setup

require 'rubygems'
require 'cgi'
require 'open-uri'
require 'hpricot'
require 'mechanize'

agent = WWW::Mechanize.new
doc = Hpricot(agent.get(the_url).parser.to_s)

to ruby rails hpricot scraping by sikelianos on Oct 04, 2007

Scraping Google Search Results with Hpricot

// snagged from http://g-module.rubyforge.org/

require 'rubygems'
require 'cgi'
require 'open-uri'
require 'hpricot'

q = %w{meine kleine suchanfrage}.map { |w| CGI.escape(w) }.join("+")
url = "http://www.google.com/search?q=#{q}"
doc = Hpricot(open(url).read)
lucky_url = (doc/"div[@class='g'] a").first["href"]
system 'open #{lucky_url}'

to ruby rails google hpricot scraping by sikelianos on Jun 12, 2007 2 Comments