AlterLabIntegration
AI Framework
LangChain
Use AlterLab as a document loader in LangChain to build RAG pipelines with live web data. Scrape pages, chunk content, embed into vector stores, and query with natural language.
Why AlterLab + LangChain?
Standard web loaders fail on JavaScript-heavy sites and get blocked by anti-bot systems. AlterLab handles JS rendering, anti-bot bypass, and returns clean text — ideal for LLM consumption.
Installation
Bash
pip install langchain-alterlab langchain-openai chromadbThe langchain-alterlab package provides a ready-to-use document loader for LangChain. The examples below also use ChromaDB for the vector store and OpenAI for embeddings, but you can substitute any LangChain-compatible alternatives.
Basic Usage
Document Loader
The langchain-alterlab package provides AlterLabLoader, a LangChain document loader. Each URL becomes a LangChain Document with page content and metadata.
Python
from langchain_alterlab import AlterLabLoader
loader = AlterLabLoader(
api_key="your_api_key",
urls=["https://example.com/blog/ai-trends"],
)
# Load pages as LangChain Documents
documents = loader.load()
doc = documents[0]
print(f"Loaded {len(doc.page_content)} chars from {doc.metadata['source']}")Loading Options
Pass AlterLab parameters to control how pages are scraped:
Python
from langchain_alterlab import AlterLabLoader
# JavaScript-heavy SPA — use JS rendering
loader = AlterLabLoader(
api_key="your_api_key",
urls=["https://app.example.com/dashboard"],
mode="js",
)
documents = loader.load()
# Use cost controls for budget-conscious pipelines
loader = AlterLabLoader(
api_key="your_api_key",
urls=["https://example.com"],
cost_controls={"max_tier": "2", "fail_fast": True},
)
documents = loader.load()RAG Pipeline
Build a complete Retrieval-Augmented Generation pipeline in three steps: load and chunk web content, embed into a vector store, then query with natural language.
Step 1: Load & Chunk
Python
from langchain_alterlab import AlterLabLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Load multiple pages
loader = AlterLabLoader(
api_key="your_api_key",
urls=[
"https://docs.example.com/getting-started",
"https://docs.example.com/api-reference",
"https://docs.example.com/tutorials",
],
)
documents = loader.load()
print(f"Loaded {len(documents)} documents")
# Split into chunks for embedding
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)
chunks = splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks")Step 2: Embed & Store
Python
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
# Create embeddings and store in ChromaDB
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
collection_name="web_docs",
persist_directory="./chroma_db",
)
print(f"Stored {len(chunks)} chunks in vector store")Step 3: Query
Python
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
)
answer = qa_chain.invoke("How do I authenticate with the API?")
print(answer["result"])Full RAG Example
A complete end-to-end example that scrapes a documentation site and answers questions about it:
Python
from langchain_alterlab import AlterLabLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA
# 1. Configure
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
# 2. Load pages as Documents
loader = AlterLabLoader(
api_key="your_alterlab_key",
urls=[
"https://docs.stripe.com/api/charges",
"https://docs.stripe.com/api/customers",
"https://docs.stripe.com/api/refunds",
],
)
documents = loader.load()
# 3. Chunk
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(documents)
# 4. Embed and store
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=OpenAIEmbeddings(),
)
# 5. Query
qa = RetrievalQA.from_chain_type(
llm=llm,
retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
)
questions = [
"How do I create a charge?",
"What parameters does the refund endpoint accept?",
"How do I list all customers?",
]
for q in questions:
answer = qa.invoke(q)
print(f"Q: {q}")
print(f"A: {answer['result']}\n")Batch Loading
For large document sets, use AlterLab's batch endpoint to load many pages concurrently:
Python
from langchain_alterlab import AlterLabLoader
# Load many URLs — the loader handles concurrency for you
loader = AlterLabLoader(
api_key="your_api_key",
urls=[
"https://example.com/page-1",
"https://example.com/page-2",
"https://example.com/page-3",
# ... add as many URLs as needed
],
)
documents = loader.load()
print(f"Loaded {len(documents)} documents")Structured Extraction
Combine AlterLab's AI extraction with LangChain for type-safe structured data:
Python
from langchain_alterlab import AlterLabLoader
# Use a custom extraction schema
loader = AlterLabLoader(
api_key="your_api_key",
urls=["https://example.com/article"],
extraction_schema={
"type": "object",
"properties": {
"title": {"type": "string"},
"author": {"type": "string"},
"published_date": {"type": "string"},
"summary": {"type": "string"},
},
},
)
documents = loader.load()
# Extracted data is available in document metadata
print(documents[0].metadata.get("extracted"))Tips & Best Practices
- Use markdown format for RAG pipelines. It preserves headings, lists, and structure while being cleaner than HTML for chunking.
- Set cost controls with
max_tierto limit per-page costs in large crawls. Tier 1-2 is sufficient for most documentation sites. - Enable caching when re-running pipelines during development. Cache hits are free and return instantly.
- Use batch scraping for loading 10+ pages. It is faster than sequential scraping and handles concurrency for you.
- Chunk on markdown headers by including
"\n## ", "\n### "in your text splitter separators. This creates semantically meaningful chunks.