Ruby GuideBeginner → Production

Web Scraping with Ruby — Complete Guide

From Net::HTTP to production-ready scrapers. Extract web data with Ruby's elegant syntax and powerful libraries.

API Reference

Ruby's expressive syntax and the Nokogiri gem make web scraping concise and readable. Nokogiri is one of the fastest HTML parsers in any language, built on libxml2. Combined with HTTParty or Faraday for HTTP, Ruby provides everything you need for effective data extraction. This guide covers the full workflow from fetching pages to building production scrapers.

Setting Up Your Ruby Scraping Environment

Install Nokogiri for HTML parsing and HTTParty for clean HTTP requests. Nokogiri wraps libxml2 and provides both CSS selector and XPath support.

gem install nokogiri httparty

Or add to your Gemfile:

gem 'nokogiri'
gem 'httparty'
gem 'csv'

For session and form handling, also install Mechanize: gem install mechanize.

gem install nokogiri httparty

Fetching a Web Page with Ruby

HTTParty provides a clean, idiomatic Ruby interface for HTTP requests. Always set a timeout and user-agent header. HTTParty returns a response object with status codes and parsed body.

require 'httparty'

response = HTTParty.get('https://example.com/products',
  headers: {
    'User-Agent' => 'Mozilla/5.0 (compatible; DataBot/1.0)',
    'Accept' => 'text/html'
  },
  timeout: 10
)

puts "Fetched #{response.body.length} bytes (status #{response.code})"

Parsing HTML with Nokogiri

Nokogiri is Ruby's standard HTML parser — fast, reliable, and supporting both CSS selectors and XPath. Use css() or at_css() for CSS selectors, and xpath() or at_xpath() for XPath queries.

require 'nokogiri'

doc = Nokogiri::HTML(response.body)

doc.css('div.product-card').each do |card|
  title = card.at_css('h2.product-title')&.text&.strip
  price = card.at_css('span.price')&.text&.strip
  href = card.at_css('a')&.[]('href')
  puts "#{title}: #{price} (#{href})"
end

Extracting Structured Data

Build an array of hashes during parsing, then output to CSV or JSON. Ruby's safe navigation operator (&.) handles missing elements cleanly without raising exceptions.

require 'json'
require 'csv'

products = doc.css('div.product-card').map do |card|
  {
    title: card.at_css('h2')&.text&.strip || '',
    price: card.at_css('.price')&.text&.strip || '',
    url: card.at_css('a')&.[]('href') || ''
  }
end

# Save to JSON
File.write('products.json', JSON.pretty_generate(products))

# Save to CSV
CSV.open('products.csv', 'w') do |csv|
  csv << %w[title price url]
  products.each { |p| csv << [p[:title], p[:price], p[:url]] }
end

Concurrent Scraping with Threads

Use Ruby's Thread class or the concurrent-ruby gem for parallel scraping. A thread pool limits the number of simultaneous requests, preventing resource exhaustion and rate limiting.

require 'httparty'
require 'nokogiri'

urls = (1..50).map { |p| "https://example.com/page/#{p}" }
results = []
mutex = Mutex.new
concurrency = 10

urls.each_slice(concurrency) do |batch|
  threads = batch.map do |url|
    Thread.new do
      response = HTTParty.get(url, timeout: 10, headers: {
        'User-Agent' => 'Mozilla/5.0 (compatible; DataBot/1.0)'
      })
      doc = Nokogiri::HTML(response.body)
      items = doc.css('div.product-card').map do |card|
        { title: card.at_css('h2')&.text&.strip }
      end
      mutex.synchronize { results.concat(items) }
    rescue StandardError => e
      puts "Error: #{e.message}"
    end
  end
  threads.each(&:join)
  sleep(0.5)
end

puts "Collected #{results.length} items"

Handling JavaScript-Rendered Pages

Ruby's HTTP clients fetch raw HTML — they do not execute JavaScript. For pages that render content dynamically, you need a rendering solution.

Options: use Ferrum or Watir (Ruby bindings for Chrome DevTools Protocol), or use a cloud rendering API. Running browsers from Ruby adds complexity and memory overhead.

AlterLab's API handles rendering server-side. Send a POST request with render_js: true and receive fully rendered HTML — no browser dependencies required.

Common Pitfalls and How to Avoid Them

The most common issues in Ruby web scraping:

**1. Nokogiri encoding issues** — Nokogiri auto-detects encoding, but it can fail. Force encoding with Nokogiri::HTML(html, nil, 'UTF-8') if you see garbled text.

**2. Missing safe navigation** — Using .text on a nil element raises NoMethodError. Always use &.text or &.[]('attr') with the safe navigation operator.

**3. No timeout set** — HTTParty without timeout can hang indefinitely. Always pass timeout: 10.

**4. Thread safety** — Use a Mutex when writing to shared data structures from multiple threads.

**5. Rate limiting** — Add sleep() between request batches. Many sites return 429 Too Many Requests when hit too fast.

Complete Ruby Scraper — Paginated with CSV Output

Complete working scraper with concurrency, error handling, and CSV output.

require 'httparty'
require 'nokogiri'
require 'csv'
require 'logger'

logger = Logger.new($stdout)

HEADERS = {
  'User-Agent' => 'Mozilla/5.0 (compatible; DataBot/1.0)',
  'Accept' => 'text/html,application/xhtml+xml',
  'Accept-Language' => 'en-US,en;q=0.9'
}.freeze

def fetch_page(url)
  response = HTTParty.get(url, headers: HEADERS, timeout: 10)
  raise "HTTP #{response.code}" unless response.success?
  response.body
end

def parse_articles(html)
  doc = Nokogiri::HTML(html)
  doc.css('article.post').map do |article|
    {
      title: article.at_css('h2')&.text&.strip || '',
      url: article.at_css('a')&.[]('href') || '',
      date: article.at_css('time')&.[]('datetime') || ''
    }
  end
end

all_articles = []
(1..10).each do |page|
  url = "https://example.com/articles?page=#{page}"
  logger.info("Scraping page #{page}: #{url}")
  begin
    html = fetch_page(url)
    articles = parse_articles(html)
    break if articles.empty?
    all_articles.concat(articles)
    sleep(1)
  rescue StandardError => e
    logger.error("Error on page #{page}: #{e.message}")
    break
  end
end

CSV.open('output.csv', 'w') do |csv|
  csv << %w[title url date]
  all_articles.each { |a| csv << [a[:title], a[:url], a[:date]] }
end
logger.info("Saved #{all_articles.length} articles")

Or Skip the Complexity — Use AlterLab

AlterLab handles JavaScript rendering, IP rotation, and website compatibility automatically. One POST request returns rendered HTML — no browser management, no proxy configuration. Starts at $0.0002/request with 5,000 free to start.

require 'httparty'
require 'nokogiri'
require 'json'

API_KEY = 'YOUR_API_KEY' # Get free at alterlab.io

def scrape_with_alterlab(url, render_js: false)
  response = HTTParty.post('https://api.alterlab.io/api/v1/scrape',
    headers: {
      'X-API-Key' => API_KEY,
      'Content-Type' => 'application/json'
    },
    body: { url: url, render_js: render_js }.to_json,
    timeout: 30
  )
  JSON.parse(response.body)['html'] || ''
end

html = scrape_with_alterlab('https://example.com/products', render_js: true)

doc = Nokogiri::HTML(html)
products = doc.css('div.product-card').map do |card|
  {
    title: card.at_css('h2')&.text&.strip,
    price: card.at_css('.price')&.text&.strip
  }
end

puts JSON.pretty_generate(products)
API Docs

Choosing Your Approach

HTTParty + Nokogiri

Pros

  • +Clean, readable Ruby syntax
  • +Nokogiri is extremely fast (libxml2-based)
  • +Both CSS selectors and XPath support

Cons

  • −No JavaScript execution
  • −GIL limits true parallelism
  • −Smaller scraping ecosystem than Python

Mechanize

Pros

  • +Built-in form handling and session management
  • +Automatic cookie persistence
  • +Follows redirects and handles relative URLs

Cons

  • −No JavaScript execution
  • −Heavier than HTTParty for simple tasks
  • −Less actively maintained

AlterLab API

Pros

  • +Handles static, JavaScript, and challenge pages
  • +No browser management
  • +Automatic IP rotation
  • +5-tier auto-escalation
  • +From $0.0002/request

Cons

  • −Per-request cost
  • −Requires network access

Frequently Asked Questions

More Ruby & Scraping Resources

Web Scraping with Python

Complete Python scraping guide: requests, BeautifulSoup, async patterns from beginner to production.

Web Scraping with PHP

PHP scraping guide: Guzzle, DOMDocument, and DomCrawler patterns.

JavaScript Rendering API

Render SPAs and dynamic content with headless Chromium — no browser management.

Pricing

From $0.0002/request. No subscriptions. Balance never expires.

Your first scrape.
Sixty seconds.

$1 free credit — up to 5,000 scrapes. No credit card.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires