Web Scraping with Go — Complete Guide
From net/http to production-grade concurrent scrapers. Everything you need to extract web data with Go.
Go is an excellent choice for web scraping when performance and concurrency matter. Its goroutines make it trivial to scrape hundreds of pages simultaneously, and the compiled binary deploys anywhere with zero dependencies. This guide covers fetching pages, parsing HTML with goquery, building concurrent pipelines, and handling common challenges.
Setting Up Your Go Scraping Environment
Go's standard library handles HTTP natively. For HTML parsing, install goquery — a jQuery-like library for Go. For full-featured scraping with built-in rate limiting and caching, consider Colly.
go mod init myscraper
go get github.com/PuerkitoBio/goquery
go get github.com/gocolly/colly/v2Goquery provides CSS selector support on top of Go's html package. Colly adds request queuing, rate limiting, and cookie handling out of the box.
go mod init myscraper
go get github.com/PuerkitoBio/goquery
go get github.com/gocolly/colly/v2Fetching a Web Page with Go
Go's net/http package handles HTTP requests without any external dependency. Always set a timeout on your HTTP client and check the response status code before reading the body.
package main
import (
"fmt"
"io"
"log"
"net/http"
"time"
)
func main() {
client := &http.Client{Timeout: 10 * time.Second}
req, _ := http.NewRequest("GET", "https://example.com/products", nil)
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; DataBot/1.0)")
req.Header.Set("Accept", "text/html")
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
fmt.Printf("Fetched %d bytes (status %d)\n", len(body), resp.StatusCode)
}Parsing HTML with goquery
goquery brings jQuery-style selectors to Go. Load an HTML document from a response body and use Find() with CSS selectors to locate elements. Use Text() to extract text content and Attr() for element attributes.
doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
log.Fatal(err)
}
doc.Find("div.product-card").Each(func(i int, s *goquery.Selection) {
title := s.Find("h2.product-title").Text()
price := s.Find("span.price").Text()
href, _ := s.Find("a").Attr("href")
fmt.Printf("%s: %s (%s)\n", title, price, href)
})Extracting Structured Data
Define a struct for your data, populate it during parsing, and encode to JSON or CSV. Go's encoding/json and encoding/csv packages handle serialization. The struct-based approach gives you compile-time type safety.
type Product struct {
Title string `json:"title"`
Price string `json:"price"`
URL string `json:"url"`
}
func extractProducts(doc *goquery.Document) []Product {
var products []Product
doc.Find("div.product-card").Each(func(i int, s *goquery.Selection) {
href, _ := s.Find("a").Attr("href")
products = append(products, Product{
Title: strings.TrimSpace(s.Find("h2").Text()),
Price: strings.TrimSpace(s.Find(".price").Text()),
URL: href,
})
})
return products
}
// Encode to JSON
data, _ := json.MarshalIndent(products, "", " ")
os.WriteFile("products.json", data, 0644)Concurrent Scraping with Goroutines
Go's goroutines make concurrent scraping straightforward. Use a semaphore pattern (buffered channel) to limit the number of simultaneous requests. This approach can process hundreds of URLs per second while respecting rate limits.
func scrapeMany(urls []string, concurrency int) []Product {
sem := make(chan struct{}, concurrency)
var mu sync.Mutex
var results []Product
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
sem <- struct{}{} // acquire
defer func() { <-sem }() // release
products, err := scrapePage(u)
if err != nil {
log.Printf("Error scraping %s: %v", u, err)
return
}
mu.Lock()
results = append(results, products...)
mu.Unlock()
}(url)
}
wg.Wait()
return results
}Handling JavaScript-Rendered Pages
Go's net/http fetches raw HTML — it does not execute JavaScript. For pages that load content dynamically (SPAs, infinite scroll, client-rendered dashboards), you need a rendering solution.
Options: run a headless browser with chromedp (Go bindings for Chrome DevTools Protocol), or use a cloud rendering API. Running browsers adds significant complexity — binary management, memory overhead, and detection challenges.
AlterLab's API handles rendering server-side. Send a POST request with render_js: true and receive fully rendered HTML — no browser management required.
Common Pitfalls and How to Avoid Them
The most common issues in Go web scraping:
**1. Not closing response bodies** — Always defer resp.Body.Close() after checking for errors. Unclosed bodies leak connections.
**2. No timeout on HTTP client** — The default http.Client has no timeout. Always set one: &http.Client{Timeout: 10 * time.Second}.
**3. Ignoring character encoding** — Some pages use non-UTF-8 encoding. Use golang.org/x/text/encoding to detect and convert.
**4. Unbounded concurrency** — Launching a goroutine per URL without limits can exhaust memory and trigger rate limiting. Use a semaphore pattern.
**5. Not handling redirects** — Go follows redirects by default (up to 10). Set a CheckRedirect function if you need to control this behavior.
Complete Go Scraper — Concurrent with CSV Output
Complete working scraper with concurrency, error handling, and CSV output.
package main
import (
"encoding/csv"
"fmt"
"log"
"net/http"
"os"
"strings"
"sync"
"time"
"github.com/PuerkitoBio/goquery"
)
type Article struct {
Title string
URL string
Date string
}
func fetchPage(client *http.Client, url string) (*goquery.Document, error) {
req, _ := http.NewRequest("GET", url, nil)
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; DataBot/1.0)")
req.Header.Set("Accept", "text/html")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return nil, fmt.Errorf("status %d", resp.StatusCode)
}
return goquery.NewDocumentFromReader(resp.Body)
}
func parseArticles(doc *goquery.Document) []Article {
var articles []Article
doc.Find("article.post").Each(func(i int, s *goquery.Selection) {
href, _ := s.Find("a").Attr("href")
articles = append(articles, Article{
Title: strings.TrimSpace(s.Find("h2").Text()),
URL: href,
Date: s.Find("time").AttrOr("datetime", ""),
})
})
return articles
}
func main() {
client := &http.Client{Timeout: 10 * time.Second}
var allArticles []Article
var mu sync.Mutex
var wg sync.WaitGroup
sem := make(chan struct{}, 5)
for page := 1; page <= 10; page++ {
wg.Add(1)
go func(p int) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
url := fmt.Sprintf("https://example.com/articles?page=%d", p)
log.Printf("Scraping page %d", p)
doc, err := fetchPage(client, url)
if err != nil {
log.Printf("Error on page %d: %v", p, err)
return
}
articles := parseArticles(doc)
mu.Lock()
allArticles = append(allArticles, articles...)
mu.Unlock()
time.Sleep(500 * time.Millisecond)
}(page)
}
wg.Wait()
f, _ := os.Create("output.csv")
defer f.Close()
w := csv.NewWriter(f)
w.Write([]string{"title", "url", "date"})
for _, a := range allArticles {
w.Write([]string{a.Title, a.URL, a.Date})
}
w.Flush()
log.Printf("Saved %d articles", len(allArticles))
}Or Skip the Complexity — Use AlterLab
AlterLab handles JavaScript rendering, IP rotation, and website compatibility automatically. One POST request returns rendered HTML — no browser management, no proxy configuration. Starts at $0.0002/request with 5,000 free to start.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
"net/http"
"strings"
"time"
"github.com/PuerkitoBio/goquery"
)
const apiKey = "YOUR_API_KEY" // Get free at alterlab.io
type ScrapeRequest struct {
URL string `json:"url"`
RenderJS bool `json:"render_js"`
}
func scrapeWithAlterLab(targetURL string, renderJS bool) (string, error) {
payload, _ := json.Marshal(ScrapeRequest{URL: targetURL, RenderJS: renderJS})
req, _ := http.NewRequest("POST", "https://api.alterlab.io/api/v1/scrape", bytes.NewReader(payload))
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 30 * time.Second}
resp, err := client.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
var result map[string]interface{}
json.NewDecoder(resp.Body).Decode(&result)
html, _ := result["html"].(string)
return html, nil
}
func main() {
html, err := scrapeWithAlterLab("https://example.com/products", true)
if err != nil {
log.Fatal(err)
}
doc, _ := goquery.NewDocumentFromReader(strings.NewReader(html))
doc.Find("div.product-card").Each(func(i int, s *goquery.Selection) {
title := strings.TrimSpace(s.Find("h2").Text())
price := strings.TrimSpace(s.Find(".price").Text())
fmt.Printf("%s: %s\n", title, price)
})
}Choosing Your Approach
net/http + goquery
Pros
- +Zero external dependencies for HTTP
- +Compiled binary — fast and portable
- +Native concurrency with goroutines
Cons
- −No JavaScript execution
- −Manual cookie/session management
- −More verbose than Python
Colly Framework
Pros
- +Built-in rate limiting and caching
- +Declarative callback-based API
- +Handles cookies and redirects
Cons
- −No JavaScript execution
- −Less flexible than raw net/http
- −Framework lock-in
AlterLab API
Pros
- +Handles static, JavaScript, and challenge pages
- +No browser management
- +Automatic IP rotation
- +5-tier auto-escalation
- +From $0.0002/request
Cons
- −Per-request cost
- −Requires network access
Frequently Asked Questions
More Go & Scraping Resources
Web Scraping with Python
Complete Python scraping guide: requests, BeautifulSoup, async patterns from beginner to production.
Web Scraping with Rust
Rust scraping guide: reqwest, scraper crate, and async patterns with tokio.
JavaScript Rendering API
Render SPAs and dynamic content with headless Chromium — no browser management.
Pricing
From $0.0002/request. No subscriptions. Balance never expires.
Your first scrape.
Sixty seconds.
$1 free credit — up to 5,000 scrapes. No credit card.
Just a POST request.
No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires