Importing SQLite Database Snapshots

Overview

A new gateway builds its index of the Arweave network block by block. That can take days or weeks, depending on your hardware and on how much data you index. You can skip most of it in one of two ways:

Import a SQLite snapshot. The 2025-04-23 snapshot holds transactions and the data items uploaded through ArDrive products, including Turbo, up to block 1645229.
Bootstrap from L1 bands (Release 85). Your gateway imports the Arweave base layer (blocks, transactions, tags and owners) from signed bands that another gateway publishes, up to the current tip. Data items are not included.

Looking to resolve data items faster without a full database? Index Sharing lets your gateway subscribe to another gateway's root transaction indexes, about 21 GB today and kept current, instead of importing a snapshot.

Note

The below instructions are designed to be used in a linux environment. Windows and MacOS users must modify the instructions to use the appropriate package manager/ command syntax for their platform.

Unless otherwise specified, all commands should be run from the root directory of the gateway.

Import a SQLite Snapshot

SQLite snapshots are large and hard to update in steps, so ar.io distributes them over BitTorrent.

IMPORTANT

Importing a snapshot deletes your existing database and replaces it with the snapshot.

Install a Torrent Client

Any BitTorrent client works. For example, transmission-cli:

# Ubuntu/Debian
sudo apt-get install transmission-cli

# CentOS/RHEL
sudo yum install transmission-cli

# macOS
brew install transmission-cli

Download the Snapshot

transmission-cli "magnet:?xt=urn:btih:62ca6e05248e6df59fac9e38252e9c71951294ed&dn=2025-04-23-sqlite.tar.gz&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=http%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=https%3A%2F%2Ftracker.bt4g.com%3A443%2Fannounce"

This downloads 2025-04-23-sqlite.tar.gz, about 42.8 GB. Check that it arrived whole with ls -lh 2025-04-23-sqlite.tar.gz.

Seeding the torrent after the download is not required, but it keeps the snapshot available for other operators.

Extract the Archive

tar -xzf 2025-04-23-sqlite.tar.gz
ls -la 2025-04-23-sqlite/

This creates a directory named after the file, without .tar.gz, that holds the SQLite database files. If you use a different snapshot, replace the file name.

Stop the Gateway

docker compose down

If you start your gateway with your own -f files, pass the same ones to every docker compose command on this page.

Replace the Database

Back up your existing database (optional), then move the snapshot into data/sqlite:

mkdir sqlite-backup
mv data/sqlite/* sqlite-backup/
mv 2025-04-23-sqlite/* data/sqlite/

If you skip the backup, delete the old files with rm data/sqlite/* before the last command.

Start the Gateway

docker compose up -d

Verify the Import

Watch the gateway logs for the block height being imported:

docker compose logs -f core

The 2025-04-23 snapshot was taken at block 1645229. If the import worked, the gateway imports blocks from 1645230 upward. The Grafana extension also shows the last block imported.

Bootstrap from L1 Bands

Release Requirement: L1 bands, and the ar-io-node CLI that imports them, arrive in gateway Release 85, which is not released yet. Your gateway and the publisher you subscribe to must both run it.

An L1 band (parquet-l1) holds the Arweave base layer for a range of block heights, as Parquet files: blocks, transactions, tags and owners. Gateways that publish them sign every file, and your gateway checks each one before it installs the band. index-l1-import then fills your gateway's core.db from the installed bands, so it starts near the tip instead of at block 0.

L1 bands carry no data items. Your gateway still unbundles the bundles your filters select, and it does that for imported history only if you also set BACKFILL_BUNDLE_RECORDS.

Prerequisites

A gateway on Release 85. The core service migrates its database when it starts, and the import refuses a core.db that has not been migrated
A core.db with no unstable blocks near the tip (the new_* tables). The import refuses one that holds them, so run it on a new gateway before its block importer has indexed the tip
Room on disk: about 15 GB for the bands, and about 120 GB for a core.db that covers the whole chain. The import refuses to start without the space it needs
Time: a full chain takes many hours. The import logs its progress once a minute

L1 bands arrive through Index Sharing. A subscription takes them only when it names them, so add parquet-l1 to the name list in INDEX_SWARM_SUBSCRIBE:

INDEX_SWARM_SUBSCRIBE='[{"publisher":"34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5","name":["root-tx-index","parquet-l1"]}]'

Then recreate the index-swarm sidecar and wait until ./tools/index-swarm-status shows the bands installed under data/indexes/installed/parquet-l1.

Stop the Gateway

The import refuses a core.db that another process is writing to. Stop only the core service, with the same -f files your gateway was started with:

docker compose stop core

Import the Bands

Run it from the gateway's directory. The tool runs in the core image, so the host needs only Docker:

./tools/ar-io-node index-l1-import \
  --bands-dir data/indexes/installed/parquet-l1 \
  --core-db data/sqlite/core.db \
  --cache-mib 8192

Bands import in height order, lowest first. --cache-mib sets SQLite's page cache for the run (default 1024 MiB); give it what your machine can spare, since a larger cache keeps a long import fast. An interrupted run is safe: run the command again and it carries on from the last band that landed. --max-bands splits the work across several runs.

If your gateway started above block 0 (START_HEIGHT), add --from 0 --to <height below what it holds> to fill in the history underneath. Check that the result's holes field is absent or empty before you start the gateway.

Start the Gateway

docker compose up -d --no-deps core

The block importer continues from the highest height imported.

To check a set of bands, or your gateway's own core.db, against the chain, use ./tools/ar-io-node index-l1-verify. Every option, and the JSON each command prints, is in the ar-io-node CLI reference.

How is this guide?

Importing SQLite Database Snapshots

On this page