Ensuring Backup Integrity: Immutable Logical Contracts and Production Table Classification at AlterLab

TL;DR

AlterLab now stores an immutable snapshot of its backup policy and redaction rules with every logical backup generation. When validating a backup or performing a restore, the system compares the data against that generation’s specific contract rather than the latest policy. Additionally, all production‑recovery tables are classified and, where needed, fields like payloads in jobs_archive are redacted to ensure restores are both accurate and safe.

Introduction

AlterLab’s scraping API generates a continuous stream of job metadata, scrape results, and usage logs. To protect this data we maintain a nightly backup pipeline that creates logical copies of the underlying store. Early in 2024 we noticed a recurring issue: whenever the backup policy or redaction rules were updated, previously valid backup generations would fail validation. This blocked the promotion of new backups and created anxiety around restore reliability. The solution was two‑fold: preserve the exact contract that governed each backup, and formally classify which production tables belong in the backup set.

The Problem: Backup Drift and Policy Evolution

Our backup system originally validated each retained generation by comparing its contents to the current backup‑policy.yaml and redaction‑registry.json files. Any change—even a harmless comment or a new optional field—altered the file hashes and caused the validation step to reject older generations. Because the validation failure prevented the next backup from being published, a single policy tweak could halt the entire nightly pipeline.

At the same time, we observed that certain tables were being retained unintentionally. Tables such as credit_balance_cache_snap_20260725 (a temporary cache) and jobs_archive (which stores verbose job payloads) were not part of the intended backup set. Their presence increased backup size and introduced noise during restore validation, yet the live‑schema gate had no way to distinguish them from essential tables.

Solution Overview: Immutable Logical Contracts

What is a Logical Contract?

A logical contract is a version‑controlled snapshot of two files:

• backup-policy.yaml – defines which tables, columns, and retention rules apply.
• redaction-registry.json – lists fields that must be stripped or transformed before storage.

When a backup generation is created, we compute a hash of these files and store the hash alongside the backup manifest. The manifest also records the exact versions of the policy and redaction files used.

How We Store Snapshots

The backup workflow now includes an extra step before uploading a generation:

1. Read the live backup-policy.yaml and redaction-registry.json.
2. Compute SHA‑256 hashes for each.
3. Write the hashes into the generation’s metadata block.
4. Upload the data payload and the metadata block together.

Because the contract is immutable, future policy edits do not alter the hash stored with an existing generation.

Validating Retained Generations Against Historical Contracts

The Validation Process

When the retention job runs, it does not compare the generation to the current policy files. Instead:

1. Load the generation’s manifest to retrieve the stored policy and redaction hashes.
2. Retrieve the exact versions of those files from our internal version store (a simple append‑only log).
3. Run the validation checks against those retrieved versions.
4. If the checks pass, the generation is considered valid for the next retention cycle.

This approach means that a policy change only affects newly created generations. Older generations continue to validate against the rules that were in place when