Downloading Indexes with BitTorrent

Gateways that publish shared indexes offer each band as a torrent, so you don't need a gateway to take a copy. Any BitTorrent client can download one, and peers don't charge for what they send. This guide downloads a band from turbo-gateway.com with qBittorrent and with aria2 on the command line, then checks it against the publisher's signature. To read single IDs rather than whole bands, see Reading Shared Indexes.

What You Need

• Disk. About 21 GB for the five root-TX bands turbo-gateway.com publishes, or a few hundred MB to try the tip band. aria2 needs up to 64 MB more per band for pad files while it downloads (see below).
• Network. Nothing to open to download: your client connects out to the peers. To seed well, forward or open your client's listening port, so that peers can connect to you as well. Some networks and hosting providers block or restrict BitTorrent traffic.

Find the Torrents

A publisher lists its bands in one signed JSON document at /ar-io/indexes. Each band that is seeded has a torrent entry, and its magnet link is part of what the publisher signed:

curl -s https://turbo-gateway.com/ar-io/indexes \
  | jq -r '.indexes[] | .name as $index | .bands[] | "\($index) \(.id)\n\(.torrent.magnet)\n"'

root-tx-index b0-h2010500-tip-20261010
magnet:?xt=urn:btih:2c35385ba96632bcfe39566d8f336e8c9b462f18&xt=urn:btmh:12201710ca02d31b9b4c7305c4506a67fdaaec634cb7200c2d6cce70725ed13686b1&dn=bfa9f0269c7b03f8&tr=http%3A%2F%2F167.235.37.218%3A6969%2Fannounce

root-tx-index b1-h1950000-tip-20260929
magnet:?xt=urn:btih:57f33f3cdda02de27eb42f84e62b3b67dccc758c&xt=urn:btmh:122027a8171c4e95fe7eeb86c4a1efe9c2190861f7f7e183cdc877549f87a2abc07a&dn=0a27456b192621a7&tr=http%3A%2F%2F167.235.37.218%3A6969%2Fannounce
...

turbo-gateway.com publishes two indexes. This guide downloads the root-TX bands (root-tx-index): five bands, about 21 GB in all. The band id shows the block heights each covers: b0-h2010500-tip runs from height 2,010,500 to the chain tip, and b4-h0-1349999 covers the start of the chain. The tip band is the smallest and is rebuilt often. The older bands are 2 to 8 GB and rarely change. Start with b0 to try it out.

The list also shows the 24 L1 bands (parquet-l1), about 15 GB, which come with gateway Release 85, not released yet. They download the same way. The .cdb partitions described below belong to root-TX bands only; an L1 band holds Parquet files and a band.json. See Query a Parquet Dataset.

The magnets on this page were current when it was written. Bands are rebuilt and replaced, so take yours from the publication.

Download with qBittorrent

Download with aria2

aria2 downloads a magnet link from the command line and stops once the download finishes:

aria2c --seed-time=0 --bt-save-metadata=false \
  'magnet:?xt=urn:btih:2c35385ba96632bcfe39566d8f336e8c9b462f18&xt=urn:btmh:12201710ca02d31b9b4c7305c4506a67fdaaec634cb7200c2d6cce70725ed13686b1&dn=bfa9f0269c7b03f8&tr=http%3A%2F%2F167.235.37.218%3A6969%2Fannounce'

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
ece690|OK  |       0B/s|[MEMORY][METADATA]bfa9f0269c7b03f8
710e97|OK  |    12MiB/s|/data/bfa9f0269c7b03f8/00.cdb (513more)

Unlike qBittorrent, aria2 (1.37) writes the pad files, up to 64 MB of zeros, into a hidden .pad/ folder inside the download. Delete it once the download is done:

rm -r bfa9f0269c7b03f8/.pad

To seed after downloading, which helps the next person, drop --seed-time=0.

Check the Download

The client checked every piece against the info hash. That proves you got the torrent the link names. It doesn't prove the files are the publisher's index. For that, check the publication's signature against the key the gateway registry holds for the publisher, then check each file's size and SHA-256 against the publication.

curl -s https://turbo-gateway.com/ar-io/indexes > publication.json

// Check a band downloaded over BitTorrent against a verified publication.
// Usage: node verify-band.mjs <publication.json> <band directory>
import crypto from "node:crypto";
import { createReadStream } from "node:fs";
import { readFile, stat } from "node:fs/promises";
import path from "node:path";

const [, , publicationFile, bandDir] = process.argv;
const doc = JSON.parse(await readFile(publicationFile, "utf8"));

async function sha256File(file) {
  const hash = crypto.createHash("sha256");
  for await (const chunk of createReadStream(file)) hash.update(chunk);
  return hash.digest("hex");
}

// The band whose files are in bandDir: matched by torrent name, which is
// the directory name a torrent client downloads into.
const bands = doc.indexes.flatMap((index) => index.bands);
const torrentName = (b) => new URLSearchParams(b.torrent?.magnet.split("?")[1]).get("dn");
const band = bands.find((b) => torrentName(b) === path.basename(bandDir));
if (band === undefined) throw new Error(`no band in the publication for ${path.basename(bandDir)}`);
console.log(`band ${band.id}, infohash v1 ${band.torrent.infohashV1}`);

// Every file: size and SHA-256 as signed.
let bad = 0;
for (const file of band.files) {
  const p = path.join(bandDir, file.name);
  const size = await stat(p).then((s) => s.size, () => undefined);
  if (size !== file.size || (await sha256File(p)) !== file.sha256) {
    bad += 1;
    console.log(`MISMATCH ${file.name}`);
  }
}
console.log(bad === 0 ? `all ${band.files.length} files match: ok` : `${bad} files do not match`);
process.exit(bad === 0 ? 0 : 1);

node verify-band.mjs publication.json ./bfa9f0269c7b03f8

band b0-h2010500-tip-20261010, infohash v1 2c35385ba96632bcfe39566d8f336e8c9b462f18
all 257 files match: ok

MISMATCH 00.cdb
MISMATCH manifest.json
2 files do not match

The check is against the publication as it is now, so check a band soon after you download it. Once the publisher replaces a band, it drops out of the publication.

What to Do with It

• Keep seeding. Leave the torrent running in your client. Every seeder makes the next download faster, for gateways as well as for people.

• Look up IDs. A root-TX band is a partitioned CDB64 index from data item ID to root transaction. See Look Up One ID to read it.

• Let a gateway do it. A gateway can subscribe to a publisher, download and check every band by itself, keep them current, and seed them:

  ./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --torrent --restart

  See Index Sharing for the full setup.