Guide

New

Cloud Storage Export

Export scraping results directly to Amazon S3, Google Cloud Storage, or Azure Blob Storage. Connect once, then auto-export from batch and scheduled jobs.

Dashboard Available

You can also manage storage integrations from the Dashboard → Integrations → Storage page without writing any code.

How It Works

Connect

Add your cloud storage credentials via the API or dashboard. AlterLab validates access and encrypts credentials at rest.

Configure

Set a bucket, optional key prefix, and mark one integration as your default export destination.

Export

Enable storage_export on batch or scheduled scraping jobs. Results are automatically uploaded as JSONL, CSV, or JSON files.

Supported Providers

Provider	Value	Required Credentials
Amazon S3	s3	`access_key_id`, `secret_access_key`
Google Cloud Storage	gcs	`service_account_json`
Azure Blob Storage	azure	`connection_string`

Add a Storage Integration

POST

/api/v1/integrations/storage

Creates a new storage integration. Credentials are validated against the provider before being encrypted and stored.

Parameter	Type	Required	Description
name	string	Yes	Friendly name (e.g., "My S3 Bucket")
provider	string	Yes	`s3`, `gcs`, or `azure`
credentials	object	Yes	Provider-specific credentials (see examples below)
bucket	string	Yes	Bucket or container name
region	string	No	AWS region, GCS location, or Azure region
prefix	string	No	Default key prefix for uploaded files (e.g., `scraping/exports/`)
is_default	boolean	No	Set as default export destination (default: `false`)

Amazon S3

Create an IAM user with s3:PutObject, s3:GetObject, and s3:DeleteObject permissions on your target bucket.

Bash

curl -X POST https://api.alterlab.io/api/v1/integrations/storage \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production S3 Bucket",
    "provider": "s3",
    "credentials": {
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "bucket": "my-scraping-exports",
    "region": "us-east-1",
    "prefix": "alterlab/",
    "is_default": true
  }'

Google Cloud Storage

Create a service account with the Storage Object Admin role on your target bucket. Pass the full JSON key as a string.

Bash

curl -X POST https://api.alterlab.io/api/v1/integrations/storage \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GCS Data Lake",
    "provider": "gcs",
    "credentials": {
      "service_account_json": "{\"type\": \"service_account\", \"project_id\": \"my-project\", ...}"
    },
    "bucket": "my-gcs-bucket",
    "region": "us-central1",
    "prefix": "scraping/"
  }'

Azure Blob Storage

Use a connection string from your Azure Storage Account. The account needs Storage Blob Data Contributor role on the container.

Bash

curl -X POST https://api.alterlab.io/api/v1/integrations/storage \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Azure Exports",
    "provider": "azure",
    "credentials": {
      "connection_string": "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=...;EndpointSuffix=core.windows.net"
    },
    "bucket": "scraping-container",
    "region": "eastus"
  }'

Credentials Are Encrypted

All credentials are validated on submission and encrypted at rest using AES-256. They are never returned in API responses.

Test Your Connection

POST

/api/v1/integrations/storage/{integration_id}/test

Uploads a small test file to your bucket and deletes it, verifying read and write access.

Bash

curl -X POST https://api.alterlab.io/api/v1/integrations/storage/{integration_id}/test \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN"

Response:

JSON

{
  "success": true,
  "message": "Connection test passed",
  "error": null
}

You can also run a deeper validation that checks both credential validity and write access:

POST

/api/v1/integrations/storage/{integration_id}/validate

JSON

{
  "valid": true,
  "provider": "s3",
  "bucket_exists": true,
  "writable": true,
  "error": null
}

Manage Integrations

List Integrations

GET

/api/v1/integrations/storage

Bash

curl https://api.alterlab.io/api/v1/integrations/storage \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN"

Optional query parameters:

Parameter	Default	Description
include_inactive	`false`	Include disabled integrations
provider	`null`	Filter by provider: `s3`, `gcs`, `azure`

JSON

{
  "integrations": [
    {
      "id": "a1b2c3d4-...",
      "name": "Production S3 Bucket",
      "provider": "s3",
      "bucket": "my-scraping-exports",
      "region": "us-east-1",
      "prefix": "alterlab/",
      "is_active": true,
      "is_default": true,
      "validation_status": "valid",
      "total_exports": 142,
      "successful_exports": 140,
      "failed_exports": 2,
      "success_rate": 98.59,
      "total_bytes_uploaded": 52428800,
      "last_used_at": "2026-03-24T10:00:00Z",
      "created_at": "2026-03-01T08:00:00Z"
    }
  ],
  "total": 1
}

Update Settings

PUT

/api/v1/integrations/storage/{integration_id}

Update name, bucket, region, prefix, or active status. Does not modify credentials.

Bash

curl -X PUT https://api.alterlab.io/api/v1/integrations/storage/{integration_id} \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "prefix": "exports/2026/",
    "is_active": true
  }'

Rotate Credentials

PUT

/api/v1/integrations/storage/{integration_id}/credentials

Replace the stored credentials. The new credentials are validated before saving.

Bash

curl -X PUT https://api.alterlab.io/api/v1/integrations/storage/{integration_id}/credentials \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "credentials": {
      "access_key_id": "AKIANEWKEY7EXAMPLE",
      "secret_access_key": "newSecretKey/EXAMPLE"
    }
  }'

Set Default Destination

PUT

/api/v1/integrations/storage/{integration_id}/default

Mark an integration as the default export destination. Only one integration can be default at a time — setting a new default automatically clears the previous one.

Bash

curl -X PUT https://api.alterlab.io/api/v1/integrations/storage/{integration_id}/default \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN"

Delete Integration

DELETE

/api/v1/integrations/storage/{integration_id}

Permanently removes the integration and its encrypted credentials. Returns 204 No Content.

Bash

curl -X DELETE https://api.alterlab.io/api/v1/integrations/storage/{integration_id} \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN"

Auto-Export with Batch & Scheduler

Once you have a default storage integration, you can enable automatic export on batch and scheduled scraping jobs. Results are uploaded to your bucket as they complete.

Default Integration Required

Auto-export uses your default storage integration. Set one with the Set Default endpoint before enabling export on jobs.

With Batch Scraping

Add storage_export to your batch request to upload results when the batch completes:

Bash

curl -X POST https://api.alterlab.io/api/v1/batch \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      { "url": "https://example.com/page-1" },
      { "url": "https://example.com/page-2" }
    ],
    "storage_export": {
      "enabled": true,
      "format": "jsonl"
    }
  }'

With Scheduled Scraping

Add storage_export to your schedule so every execution uploads results automatically:

Bash

curl -X POST https://api.alterlab.io/api/v1/schedules \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "cron": "0 */6 * * *",
    "storage_export": {
      "enabled": true,
      "format": "csv"
    }
  }'

Supported export formats:

Format	Description
jsonl	One JSON object per line — ideal for streaming and BigQuery
csv	Comma-separated values — opens in Excel and Google Sheets
json	Single JSON array — simple for small datasets

Limits

Integration Limits

Maximum 5 storage integrations per account
Maximum 2 integrations per provider
Only one default integration at a time
Credentials are validated on create and credential update — invalid credentials are rejected
All operations require a valid session (JWT via Authorization header)

Python Example

Python

import alterlab

client = alterlab.AlterLab(api_key="your_api_key")

# Add an S3 integration
integration = client.create_storage_integration(
    name="My S3 Bucket",
    provider="s3",
    credentials={
        "access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    },
    bucket="my-scraping-exports",
    region="us-east-1",
    prefix="alterlab/",
    is_default=True,
)

print(f"Integration ID: {integration['id']}")
print(f"Status: {integration['validation_status']}")

# Test the connection
test = client.test_storage_integration(integration["id"])
print(f"Test passed: {test['success']}")

# List all integrations
integrations = client.list_storage_integrations()
for i in integrations["integrations"]:
    default = " (default)" if i["is_default"] else ""
    print(f"  {i['provider']}: {i['name']}{default} — {i['validation_status']}")

# Use with batch scraping
batch = client.batch_scrape(
    urls=[
        {"url": "https://example.com/page-1"},
        {"url": "https://example.com/page-2"},
    ],
    storage_export={"enabled": True, "format": "jsonl"},
)
print(f"Batch {batch['batch_id']} submitted with storage export")

Node.js Example

TYPESCRIPT

import AlterLab from "@alterlab/sdk";

const client = new AlterLab({ apiKey: "your_api_key" });

// Add an S3 integration
const integration = await client.createStorageIntegration({
  name: "My S3 Bucket",
  provider: "s3",
  credentials: {
    access_key_id: "AKIAIOSFODNN7EXAMPLE",
    secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  },
  bucket: "my-scraping-exports",
  region: "us-east-1",
  prefix: "alterlab/",
  isDefault: true,
});

console.log(`Integration ID: ${integration.id}`);
console.log(`Status: ${integration.validationStatus}`);

// Test the connection
const test = await client.testStorageIntegration(integration.id);
console.log(`Test passed: ${test.success}`);

// List all integrations
const { integrations } = await client.listStorageIntegrations();
for (const i of integrations) {
  const tag = i.isDefault ? " (default)" : "";
  console.log(`  ${i.provider}: ${i.name}${tag} — ${i.validationStatus}`);
}

// Use with batch scraping
const batch = await client.batchScrape({
  urls: [
    { url: "https://example.com/page-1" },
    { url: "https://example.com/page-2" },
  ],
  storageExport: { enabled: true, format: "jsonl" },
});
console.log(`Batch ${batch.batchId} submitted with storage export`);

Change Detection Spend Limits

Last updated: June 2026

Guide

New

Cloud Storage Export

Export scraping results directly to Amazon S3, Google Cloud Storage, or Azure Blob Storage. Connect once, then auto-export from batch and scheduled jobs.

Dashboard Available

You can also manage storage integrations from the Dashboard → Integrations → Storage page without writing any code.

How It Works

Connect

Add your cloud storage credentials via the API or dashboard. AlterLab validates access and encrypts credentials at rest.

Configure

Set a bucket, optional key prefix, and mark one integration as your default export destination.

Export

Enable storage_export on batch or scheduled scraping jobs. Results are automatically uploaded as JSONL, CSV, or JSON files.

Supported Providers

Provider	Value	Required Credentials
Amazon S3	s3	`access_key_id`, `secret_access_key`
Google Cloud Storage	gcs	`service_account_json`
Azure Blob Storage	azure	`connection_string`

Add a Storage Integration

POST

/api/v1/integrations/storage

Creates a new storage integration. Credentials are validated against the provider before being encrypted and stored.

Parameter	Type	Required	Description
name	string	Yes	Friendly name (e.g., "My S3 Bucket")
provider	string	Yes	`s3`, `gcs`, or `azure`
credentials	object	Yes	Provider-specific credentials (see examples below)
bucket	string	Yes	Bucket or container name
region	string	No	AWS region, GCS location, or Azure region
prefix	string	No	Default key prefix for uploaded files (e.g., `scraping/exports/`)
is_default	boolean	No	Set as default export destination (default: `false`)

Amazon S3

Create an IAM user with s3:PutObject, s3:GetObject, and s3:DeleteObject permissions on your target bucket.

Bash

curl -X POST https://api.alterlab.io/api/v1/integrations/storage \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production S3 Bucket",
    "provider": "s3",
    "credentials": {
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "bucket": "my-scraping-exports",
    "region": "us-east-1",
    "prefix": "alterlab/",
    "is_default": true
  }'

Google Cloud Storage

Create a service account with the Storage Object Admin role on your target bucket. Pass the full JSON key as a string.

Bash

curl -X POST https://api.alterlab.io/api/v1/integrations/storage \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GCS Data Lake",
    "provider": "gcs",
    "credentials": {
      "service_account_json": "{\"type\": \"service_account\", \"project_id\": \"my-project\", ...}"
    },
    "bucket": "my-gcs-bucket",
    "region": "us-central1",
    "prefix": "scraping/"
  }'

Azure Blob Storage

Use a connection string from your Azure Storage Account. The account needs Storage Blob Data Contributor role on the container.

Bash

curl -X POST https://api.alterlab.io/api/v1/integrations/storage \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Azure Exports",
    "provider": "azure",
    "credentials": {
      "connection_string": "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=...;EndpointSuffix=core.windows.net"
    },
    "bucket": "scraping-container",
    "region": "eastus"
  }'

Credentials Are Encrypted

All credentials are validated on submission and encrypted at rest using AES-256. They are never returned in API responses.

Test Your Connection

POST

/api/v1/integrations/storage/{integration_id}/test

Uploads a small test file to your bucket and deletes it, verifying read and write access.

Bash

curl -X POST https://api.alterlab.io/api/v1/integrations/storage/{integration_id}/test \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN"

Response:

JSON

{
  "success": true,
  "message": "Connection test passed",
  "error": null
}

You can also run a deeper validation that checks both credential validity and write access:

POST

/api/v1/integrations/storage/{integration_id}/validate

JSON

{
  "valid": true,
  "provider": "s3",
  "bucket_exists": true,
  "writable": true,
  "error": null
}

Manage Integrations

List Integrations

GET

/api/v1/integrations/storage

Bash

curl https://api.alterlab.io/api/v1/integrations/storage \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN"

Optional query parameters:

Parameter	Default	Description
include_inactive	`false`	Include disabled integrations
provider	`null`	Filter by provider: `s3`, `gcs`, `azure`

JSON

{
  "integrations": [
    {
      "id": "a1b2c3d4-...",
      "name": "Production S3 Bucket",
      "provider": "s3",
      "bucket": "my-scraping-exports",
      "region": "us-east-1",
      "prefix": "alterlab/",
      "is_active": true,
      "is_default": true,
      "validation_status": "valid",
      "total_exports": 142,
      "successful_exports": 140,
      "failed_exports": 2,
      "success_rate": 98.59,
      "total_bytes_uploaded": 52428800,
      "last_used_at": "2026-03-24T10:00:00Z",
      "created_at": "2026-03-01T08:00:00Z"
    }
  ],
  "total": 1
}

Update Settings

PUT

/api/v1/integrations/storage/{integration_id}

Update name, bucket, region, prefix, or active status. Does not modify credentials.

Bash

curl -X PUT https://api.alterlab.io/api/v1/integrations/storage/{integration_id} \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "prefix": "exports/2026/",
    "is_active": true
  }'

Rotate Credentials

PUT

/api/v1/integrations/storage/{integration_id}/credentials

Replace the stored credentials. The new credentials are validated before saving.

Bash

curl -X PUT https://api.alterlab.io/api/v1/integrations/storage/{integration_id}/credentials \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "credentials": {
      "access_key_id": "AKIANEWKEY7EXAMPLE",
      "secret_access_key": "newSecretKey/EXAMPLE"
    }
  }'

Set Default Destination

PUT

/api/v1/integrations/storage/{integration_id}/default

Mark an integration as the default export destination. Only one integration can be default at a time — setting a new default automatically clears the previous one.

Bash

curl -X PUT https://api.alterlab.io/api/v1/integrations/storage/{integration_id}/default \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN"

Delete Integration

DELETE

/api/v1/integrations/storage/{integration_id}

Permanently removes the integration and its encrypted credentials. Returns 204 No Content.

Bash

curl -X DELETE https://api.alterlab.io/api/v1/integrations/storage/{integration_id} \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN"

Auto-Export with Batch & Scheduler

Once you have a default storage integration, you can enable automatic export on batch and scheduled scraping jobs. Results are uploaded to your bucket as they complete.

Default Integration Required

Auto-export uses your default storage integration. Set one with the Set Default endpoint before enabling export on jobs.

With Batch Scraping

Add storage_export to your batch request to upload results when the batch completes:

Bash

curl -X POST https://api.alterlab.io/api/v1/batch \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      { "url": "https://example.com/page-1" },
      { "url": "https://example.com/page-2" }
    ],
    "storage_export": {
      "enabled": true,
      "format": "jsonl"
    }
  }'

With Scheduled Scraping

Add storage_export to your schedule so every execution uploads results automatically:

Bash

curl -X POST https://api.alterlab.io/api/v1/schedules \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "cron": "0 */6 * * *",
    "storage_export": {
      "enabled": true,
      "format": "csv"
    }
  }'

Supported export formats:

Format	Description
jsonl	One JSON object per line — ideal for streaming and BigQuery
csv	Comma-separated values — opens in Excel and Google Sheets
json	Single JSON array — simple for small datasets

Limits

Integration Limits

Maximum 5 storage integrations per account
Maximum 2 integrations per provider
Only one default integration at a time
Credentials are validated on create and credential update — invalid credentials are rejected
All operations require a valid session (JWT via Authorization header)

Python Example

Python

import alterlab

client = alterlab.AlterLab(api_key="your_api_key")

# Add an S3 integration
integration = client.create_storage_integration(
    name="My S3 Bucket",
    provider="s3",
    credentials={
        "access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    },
    bucket="my-scraping-exports",
    region="us-east-1",
    prefix="alterlab/",
    is_default=True,
)

print(f"Integration ID: {integration['id']}")
print(f"Status: {integration['validation_status']}")

# Test the connection
test = client.test_storage_integration(integration["id"])
print(f"Test passed: {test['success']}")

# List all integrations
integrations = client.list_storage_integrations()
for i in integrations["integrations"]:
    default = " (default)" if i["is_default"] else ""
    print(f"  {i['provider']}: {i['name']}{default} — {i['validation_status']}")

# Use with batch scraping
batch = client.batch_scrape(
    urls=[
        {"url": "https://example.com/page-1"},
        {"url": "https://example.com/page-2"},
    ],
    storage_export={"enabled": True, "format": "jsonl"},
)
print(f"Batch {batch['batch_id']} submitted with storage export")

Node.js Example

TYPESCRIPT

import AlterLab from "@alterlab/sdk";

const client = new AlterLab({ apiKey: "your_api_key" });

// Add an S3 integration
const integration = await client.createStorageIntegration({
  name: "My S3 Bucket",
  provider: "s3",
  credentials: {
    access_key_id: "AKIAIOSFODNN7EXAMPLE",
    secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  },
  bucket: "my-scraping-exports",
  region: "us-east-1",
  prefix: "alterlab/",
  isDefault: true,
});

console.log(`Integration ID: ${integration.id}`);
console.log(`Status: ${integration.validationStatus}`);

// Test the connection
const test = await client.testStorageIntegration(integration.id);
console.log(`Test passed: ${test.success}`);

// List all integrations
const { integrations } = await client.listStorageIntegrations();
for (const i of integrations) {
  const tag = i.isDefault ? " (default)" : "";
  console.log(`  ${i.provider}: ${i.name}${tag} — ${i.validationStatus}`);
}

// Use with batch scraping
const batch = await client.batchScrape({
  urls: [
    { url: "https://example.com/page-1" },
    { url: "https://example.com/page-2" },
  ],
  storageExport: { enabled: true, format: "jsonl" },
});
console.log(`Batch ${batch.batchId} submitted with storage export`);

Change Detection Spend Limits

Last updated: June 2026