Integration

AI Framework

LlamaIndex

Use AlterLab as a data connector in LlamaIndex to build RAG pipelines with live web content. Install the reader package, index pages into vector stores, and query with natural language.

Why AlterLab + LlamaIndex?

LlamaIndex excels at indexing and querying data. AlterLab handles the hard part — getting clean content from any website, including JavaScript-heavy SPAs and anti-bot-protected pages. Together, they make web-powered RAG pipelines simple.

Installation

Bash

pip install llama-index-readers-alterlab llama-index-llms-openai llama-index-embeddings-openai

The llama-index-readers-alterlab package provides a ready-to-use LlamaIndex reader. The examples below use OpenAI for LLM and embeddings, but you can substitute any LlamaIndex-compatible provider.

Basic Usage

Python

from llama_index_readers_alterlab import AlterLabReader

# Initialize the reader
reader = AlterLabReader(api_key="your_alterlab_key")

# Load a single page
documents = reader.load_data(["https://example.com/docs"])
print(f"Loaded {len(documents)} documents")
print(f"Content length: {len(documents[0].text)} chars")
print(f"Source: {documents[0].metadata['source']}")

# Load multiple pages
urls = [
    "https://example.com/docs/getting-started",
    "https://example.com/docs/api-reference",
    "https://example.com/docs/tutorials",
]
documents = reader.load_data(urls)
print(f"Loaded {len(documents)} documents")

Reader Options

Configure the reader for different scraping scenarios:

Python

from llama_index_readers_alterlab import AlterLabReader

# JavaScript-heavy SPA — use JS rendering
reader = AlterLabReader(
    api_key="your_alterlab_key",
    mode="js",
)
documents = reader.load_data(["https://app.example.com/dashboard"])

# Get text format instead of markdown
reader = AlterLabReader(
    api_key="your_alterlab_key",
    output_format="text",
)
documents = reader.load_data(["https://example.com"])

RAG Pipeline

Step 1: Index Web Content

Python

from llama_index_readers_alterlab import AlterLabReader
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Configure LlamaIndex defaults
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding()

# Load pages
reader = AlterLabReader(api_key="your_alterlab_key")
documents = reader.load_data([
    "https://docs.stripe.com/api/charges",
    "https://docs.stripe.com/api/customers",
    "https://docs.stripe.com/api/refunds",
])

# Build index
index = VectorStoreIndex.from_documents(documents)
print(f"Indexed {len(documents)} documents")

Step 2: Query

Python

# Create a query engine
query_engine = index.as_query_engine(similarity_top_k=3)

# Ask questions about the indexed content
response = query_engine.query("How do I create a charge in Stripe?")
print(response)

# The response includes source nodes with metadata
for node in response.source_nodes:
    print(f"  Source: {node.metadata.get('source')}")
    print(f"  Score: {node.score:.3f}")

Full Example

Complete end-to-end RAG pipeline that scrapes documentation and answers questions:

Python

from llama_index_readers_alterlab import AlterLabReader
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# 1. Configure
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding()

# 2. Load web pages
reader = AlterLabReader(api_key="your_alterlab_key")
documents = reader.load_data([
    "https://docs.stripe.com/api/charges",
    "https://docs.stripe.com/api/customers",
    "https://docs.stripe.com/api/refunds",
])

# 3. Index
index = VectorStoreIndex.from_documents(documents)

# 4. Query
engine = index.as_query_engine(similarity_top_k=3)

questions = [
    "How do I create a charge?",
    "What parameters does the refund endpoint accept?",
    "How do I list all customers?",
]

for q in questions:
    response = engine.query(q)
    print(f"Q: {q}")
    print(f"A: {response}\n")

Agent Tool

Beyond the AlterLabReader data connector, the same package ships AlterLabScrapeTool — a tool class for LlamaIndex agents that need to scrape pages on demand, rather than as a fixed pre-loading step.

Python

from llama_index.readers.alterlab import AlterLabScrapeTool

# Initialize the tool
tool = AlterLabScrapeTool(api_key="your_alterlab_key")

# Use it directly
result = tool.scrape("https://example.com/blog/ai-trends")
print(result)

# Release the underlying client when you're done
tool.close()

AlterLabScrapeTool holds a persistent client, so prefer using it as a context manager to guarantee cleanup:

Python

from llama_index.readers.alterlab import AlterLabScrapeTool

with AlterLabScrapeTool(api_key="your_alterlab_key") as tool:
    result = tool.scrape("https://example.com/blog/ai-trends")
    print(result)

Convert it to a FunctionTool with to_tool() to give a LlamaIndex agent the ability to fetch web pages autonomously:

Python

from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI
from llama_index.readers.alterlab import AlterLabScrapeTool

scrape_tool = AlterLabScrapeTool(api_key="your_alterlab_key")

agent = ReActAgent.from_tools(
    [scrape_tool.to_tool()],
    llm=OpenAI(model="gpt-4"),
)

response = agent.chat("Scrape https://example.com and summarize it")
print(response)

scrape_tool.close()

The constructor accepts the same scraping options as AlterLabReader — mode ("auto", "html", "js", "pdf", or "ocr"), output_format, and render_js — so agent scraping behaves consistently with your indexing pipeline.

LlamaIndex vs LangChain

Both frameworks work well with AlterLab. Here is when to use each:

Aspect	LlamaIndex	LangChain
Best for	Index-first RAG, knowledge bases, document QA	Multi-step chains, agents, complex orchestration
Index types	Vector, list, tree, keyword — built-in	Vector stores via integrations
Learning curve	Simpler for RAG-focused work	More flexible but more concepts to learn
AlterLab setup	`pip install llama-index-readers-alterlab`	`pip install langchain-alterlab` (see LangChain guide)

Tips & Best Practices

Use markdown format for the best RAG results. Markdown preserves document structure (headings, lists, code blocks) which improves chunking quality.
Set chunk overlap in your node parser. LlamaIndex defaults work well, but 200-token overlap helps with context continuity across chunks.
Use metadata filtering — store the source URL in document metadata so you can filter queries by domain or page type.
Persist your index to avoid re-scraping. Use index.storage_context.persist() to save locally, or connect to a persistent vector store like Pinecone or Weaviate.
Load many pages at once — pass all URLs to load_data() and the reader handles them efficiently.

Last updated: June 2026

Integration

AI Framework

LlamaIndex

Use AlterLab as a data connector in LlamaIndex to build RAG pipelines with live web content. Install the reader package, index pages into vector stores, and query with natural language.

Why AlterLab + LlamaIndex?

Installation

Bash

pip install llama-index-readers-alterlab llama-index-llms-openai llama-index-embeddings-openai

The llama-index-readers-alterlab package provides a ready-to-use LlamaIndex reader. The examples below use OpenAI for LLM and embeddings, but you can substitute any LlamaIndex-compatible provider.

Basic Usage

Python

from llama_index_readers_alterlab import AlterLabReader

# Initialize the reader
reader = AlterLabReader(api_key="your_alterlab_key")

# Load a single page
documents = reader.load_data(["https://example.com/docs"])
print(f"Loaded {len(documents)} documents")
print(f"Content length: {len(documents[0].text)} chars")
print(f"Source: {documents[0].metadata['source']}")

# Load multiple pages
urls = [
    "https://example.com/docs/getting-started",
    "https://example.com/docs/api-reference",
    "https://example.com/docs/tutorials",
]
documents = reader.load_data(urls)
print(f"Loaded {len(documents)} documents")

Reader Options

Configure the reader for different scraping scenarios:

Python

from llama_index_readers_alterlab import AlterLabReader

# JavaScript-heavy SPA — use JS rendering
reader = AlterLabReader(
    api_key="your_alterlab_key",
    mode="js",
)
documents = reader.load_data(["https://app.example.com/dashboard"])

# Get text format instead of markdown
reader = AlterLabReader(
    api_key="your_alterlab_key",
    output_format="text",
)
documents = reader.load_data(["https://example.com"])

RAG Pipeline

Step 1: Index Web Content

Python

from llama_index_readers_alterlab import AlterLabReader
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# Configure LlamaIndex defaults
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding()

# Load pages
reader = AlterLabReader(api_key="your_alterlab_key")
documents = reader.load_data([
    "https://docs.stripe.com/api/charges",
    "https://docs.stripe.com/api/customers",
    "https://docs.stripe.com/api/refunds",
])

# Build index
index = VectorStoreIndex.from_documents(documents)
print(f"Indexed {len(documents)} documents")

Step 2: Query

Python

# Create a query engine
query_engine = index.as_query_engine(similarity_top_k=3)

# Ask questions about the indexed content
response = query_engine.query("How do I create a charge in Stripe?")
print(response)

# The response includes source nodes with metadata
for node in response.source_nodes:
    print(f"  Source: {node.metadata.get('source')}")
    print(f"  Score: {node.score:.3f}")

Full Example

Complete end-to-end RAG pipeline that scrapes documentation and answers questions:

Python

from llama_index_readers_alterlab import AlterLabReader
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# 1. Configure
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding()

# 2. Load web pages
reader = AlterLabReader(api_key="your_alterlab_key")
documents = reader.load_data([
    "https://docs.stripe.com/api/charges",
    "https://docs.stripe.com/api/customers",
    "https://docs.stripe.com/api/refunds",
])

# 3. Index
index = VectorStoreIndex.from_documents(documents)

# 4. Query
engine = index.as_query_engine(similarity_top_k=3)

questions = [
    "How do I create a charge?",
    "What parameters does the refund endpoint accept?",
    "How do I list all customers?",
]

for q in questions:
    response = engine.query(q)
    print(f"Q: {q}")
    print(f"A: {response}\n")

Agent Tool

Python

from llama_index.readers.alterlab import AlterLabScrapeTool

# Initialize the tool
tool = AlterLabScrapeTool(api_key="your_alterlab_key")

# Use it directly
result = tool.scrape("https://example.com/blog/ai-trends")
print(result)

# Release the underlying client when you're done
tool.close()

AlterLabScrapeTool holds a persistent client, so prefer using it as a context manager to guarantee cleanup:

Python

from llama_index.readers.alterlab import AlterLabScrapeTool

with AlterLabScrapeTool(api_key="your_alterlab_key") as tool:
    result = tool.scrape("https://example.com/blog/ai-trends")
    print(result)

Convert it to a FunctionTool with to_tool() to give a LlamaIndex agent the ability to fetch web pages autonomously:

Python

from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI
from llama_index.readers.alterlab import AlterLabScrapeTool

scrape_tool = AlterLabScrapeTool(api_key="your_alterlab_key")

agent = ReActAgent.from_tools(
    [scrape_tool.to_tool()],
    llm=OpenAI(model="gpt-4"),
)

response = agent.chat("Scrape https://example.com and summarize it")
print(response)

scrape_tool.close()

LlamaIndex vs LangChain

Both frameworks work well with AlterLab. Here is when to use each:

Aspect	LlamaIndex	LangChain
Best for	Index-first RAG, knowledge bases, document QA	Multi-step chains, agents, complex orchestration
Index types	Vector, list, tree, keyword — built-in	Vector stores via integrations
Learning curve	Simpler for RAG-focused work	More flexible but more concepts to learn
AlterLab setup	`pip install llama-index-readers-alterlab`	`pip install langchain-alterlab` (see LangChain guide)

Tips & Best Practices

Use markdown format for the best RAG results. Markdown preserves document structure (headings, lists, code blocks) which improves chunking quality.
Set chunk overlap in your node parser. LlamaIndex defaults work well, but 200-token overlap helps with context continuity across chunks.
Use metadata filtering — store the source URL in document metadata so you can filter queries by domain or page type.
Persist your index to avoid re-scraping. Use index.storage_context.persist() to save locally, or connect to a persistent vector store like Pinecone or Weaviate.
Load many pages at once — pass all URLs to load_data() and the reader handles them efficiently.

Last updated: June 2026