How to Feed Live Web Data into a Vector Database for RAG

TL;DR

To feed live web data into a vector database for RAG, scrape target pages with AlterLab, extract text, generate embeddings with an open-source model, and upsert the vectors into your store. Repeat on a schedule or via webhooks to keep the index fresh.

Why Real-Time Data Matters for RAG

Retrieval-augmented generation relies on the quality and freshness of the source documents. Stale indices lead to outdated answers, especially for news, pricing, or product availability. By continuously ingesting live web content, you keep the vector store aligned with the current state of the source sites.

Architecture Overview

The pipeline consists of four stages:

1. Ingestion – AlterLab retrieves HTML from target URLs.
2. Processing – Strip HTML, clean text, and chunk into embedding‑ready passages.
3. Embedding – Convert each passage to a vector using a model like all-MiniLM-L6-v2.
4. Storage – Upsert vectors and metadata into a vector database (e.g., Pinecone, Weaviate, Qdrant).

The loop can be triggered by a cron job, a message queue, or AlterLab’s webhook feature.

Step-by-Step Implementation

Below is a minimal Python script that demonstrates the full flow. It uses AlterLab’s Python SDK, the sentence-transformers library for embeddings, and Pinecone for vector storage. Adjust the API key, index name, and target URLs as needed.

import alterlab
from sentence_transformers import SentenceTransformer
import pinecone
import os

# Initialize services
alterlab_client = alterlab.Client(os.getenv("ALTERLAB_API_KEY"))  # highlighted
embed_model = SentenceTransformer("all-MiniLM-L6-v2")             # highlighted
pinecone.init(api_key=os.getenv("PINECONE_API_KEY"), environment="us-west1-gcp")  # highlighted
index = pinecone.Index("web-rag")                                 # highlighted

def scrape_and_upsert(url: str):
    # 1. Fetch page via AlterLab
    resp = alterlab_client.scrape(url, formats=["text"])          # highlighted
    raw_text = resp.text

    # 2. Simple cleaning – replace multiple whitespace, split into paragraphs
    paragraphs = [p.strip() for p in raw_text.split("\n") if len(p.strip()) > 50]

    # 3. Embed each paragraph
    vectors = embed_model.encode(paragraphs).tolist()

    # 4. Upsert with metadata
    to_upsert = []
    for i, (vec, para) in enumerate(zip(vectors, paragraphs)):
        to_upsert.append({
            "id": f"{url}_{i}",
            "values": vec,
            "metadata": {"source": url, "text": para[:500]}  # store snippet for citation
        })
    index.upsert(vectors=to_upsert)

# Example usage – replace with your target list
if __name__ == "__main__":
    target_urls = [
        "https://example.com/news",
        "https://example.com/products",
    ]
    for url in target_urls:
        scrape_and_upsert(url)
        print(f"Processed {url}")

Equivalent cURL Request

If you prefer to call the API directly, the same scrape operation looks like this:

curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: $ALTERLAB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/news",
    "formats": ["text"]
  }' | jq -r '.text' > news.txt

You would then pipe news.txt into your embedding and upsert steps.

Infographic: Try It Yourself

Step Flow: From Scrape to Vector Store

Keeping the Index Fresh

To maintain up‑to‑date vectors, schedule the script to run every hour or use AlterLab’s webhook to trigger a re‑scrape when a page changes. The webhook payload includes a change_detected flag, letting you avoid unnecessary recomputation.

Internal Resources

For more details on the AlterLab Python SDK, see the Python scraping API. The quickstart guide walks through installation and your first request. If you