# Admin (/apis/ar-io-node/admin) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Access several password protected features and functions specific to your AR.IO Gateway. # ArNS (/apis/ar-io-node/arns) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Resolve ArNS names (the ar.io Arweave Name System) and serve their data # Blocks (/apis/ar-io-node/blocks) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get current or historical Arweave block information # Chunks (/apis/ar-io-node/chunks) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Upload Arweave data chunks or get existing chunk offset information # Data (/apis/ar-io-node/data) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Core data retrieval operations for accessing transaction and data item content. Supports manifest resolution, range requests, caching, and verification status. These endpoints serve as the primary interface for retrieving data from the Permaweb. # Farcaster Frames (/apis/ar-io-node/farcaster-frames) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Retrieve and interact with Farcaster Frames using Arweave transactions. # Gateway (/apis/ar-io-node/gateway) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Operations related to the AR.IO Gateway server itself, including health checks, metrics, and gateway-specific information # Index Querying (/apis/ar-io-node/index-querying) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get data from the AR.IO Gateway index using GQL # ar.io Gateway APIs (/apis/ar-io-node) import { Server, Network, Route, Database, Search, FileText, } from "lucide-react"; The ar.io Gateway is the core software for the ar.io network, serving the essential responsibility of gateways for accessing, caching, and querying data stored on Arweave. It provides a robust, decentralized infrastructure for interacting with the permanent web. ## Core Responsibilities The ar.io Gateway handles fundamental operations for the Arweave ecosystem: - **Data Access** - Retrieve transaction data, files, and metadata from Arweave - **Caching** - Intelligent caching strategies for improved performance and availability - **Data Querying** - Powerful search and indexing capabilities for Arweave data - **ArNS Resolution** - Resolve human-readable names to Arweave transaction IDs - **Network Management** - Coordinate with other gateways in the ar.io network ## Advanced Features Beyond basic gateway functionality, ar.io Gateway includes sophisticated capabilities: - **Parquet Generation** - Convert Arweave data into optimized Parquet format for analytics - **Data Verification** - Cryptographic verification of data integrity and authenticity - **Index Querying** - Advanced search and filtering across Arweave datasets - **Farcaster Frames** - Support for Farcaster protocol integration - **Admin Controls** - Comprehensive gateway management and configuration ## APIs Categories } title="Data Access" description="Retrieve transaction data, files, and metadata from Arweave" href="/apis/ar-io-node/data" /> } title="ArNS Resolution" description="Resolve human-readable names to Arweave transaction IDs" href="/apis/ar-io-node/arns" /> } title="Transactions & Blocks" description="Access transaction details, block information, and network data" href="/apis/ar-io-node/transactions" /> } title="Index Querying" description="Advanced search and filtering capabilities across Arweave data" href="/apis/ar-io-node/index-querying" /> } title="Network & Gateway" description="Gateway status, network information, and peer coordination" href="/apis/ar-io-node/network" /> } title="Admin & Management" description="Gateway configuration, pricing, and administrative controls" href="/apis/ar-io-node/admin" /> ## Get Involved with ar.io Gateways } title="Run a Gateway" description="Join the ar.io network by operating your own gateway and earn rewards" href="/build/run-a-gateway/quick-start" /> } title="Leverage Gateways with Wayfinder" description="Use Wayfinder SDK to access data through the distributed gateway network" href="/sdks/wayfinder" /> } title="Join the Network" description="Learn about the ar.io network and how to participate in the ecosystem" href="https://ar.io/ecosystem/" /> ## Getting Started 1. **Explore the APIs endpoints** - Review the comprehensive APIs documentation 2. **Test with sample requests** - Try out the interactive examples 3. **Choose your integration approach** - Direct APIs calls or SDK usage 4. **Consider running a gateway** - Contribute to the network infrastructure The ar.io Gateway APIs provide the foundation for building robust, decentralized applications on Arweave with reliable data access and advanced querying capabilities. # Indexes (/apis/ar-io-node/indexes) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Verifiable Index Sharing: the signed publication of the root transaction indexes this gateway offers to other gateways and clients, and their files by name or by SHA-256. See [Index Sharing](https://docs.ar.io/learn/gateways/index-sharing). # Network (/apis/ar-io-node/network) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get Arweave node info, peers and network status # Pricing (/apis/ar-io-node/pricing) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get the price (in winston) for an amount of bytes # Rate Limiting (/apis/ar-io-node/rate-limiting) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Query and manage rate limit bucket balances. Supports both public balance queries and top-ups via x402 payments or admin authentication. **⚠️ EXPERIMENTAL FEATURES**: The rate limiter and x402 payment protocol are experimental features subject to change. API endpoints, parameters, behavior, and configuration options (environment variables) may evolve in future releases as these systems continue to be developed. See [docs/x402-and-rate-limiting.md](https://github.com/ar-io/ar-io-node/blob/develop/docs/x402-and-rate-limiting.md) for comprehensive documentation. # Transactions (/apis/ar-io-node/transactions) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Submit a new Arweave transaction or get existing transaction information # Wallets (/apis/ar-io-node/wallets) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Get Arweave wallet balance and last transaction information # APIs Reference (/apis) Explore the REST APIs available in the ar.io ecosystem. Our services are built with a commitment to open source principles, and all repositories are publicly available under AGPL-3 licenses. ## Available Services } title="ar.io Gateway" description="The core gateway software providing access to data on Arweave. Includes data retrieval, ArNS resolution, and network management." href="/apis/ar-io-node" /> } title="Turbo" description="Turbo, an ar.io gateway run by the ArDrive team: these APIs bundle your files and issue the credits you pay with." href="/apis/turbo" /> ## ar.io Gateway APIs The ar.io Gateway serves as the primary interface for accessing Arweave data through the ar.io network. Key endpoints include: - **Data Access** - Retrieve transaction data and files from Arweave - **ArNS Resolution** - Resolve human-readable names to Arweave transaction IDs - **Network Information** - Query gateway health, pricing, and network status - **Transaction Management** - Submit and track transactions - **Admin Functions** - Gateway administration and configuration ## Turbo APIs Turbo is an ar.io gateway run by the ArDrive team. Its upload and payment APIs offer: - **Data Upload** - Fast, reliable uploads with instant confirmation - **Payment Processing** - Transparent pricing and payment management - **Upload Tracking** - Monitor upload status and metadata - **Credit Management** - Handle payment credits and billing ## Open Source Commitment We believe strongly in open source development. All ar.io services are: - **Publicly Available** - Source code is open and accessible - **AGPL-3 Licensed** - Ensuring software freedom and transparency - **Community Driven** - Built with input from the developer community - **Auditable** - Code can be reviewed and verified by anyone ## Getting Started 1. **Choose your service** - Select the APIs that fit your needs 2. **Review the documentation** - Each service has comprehensive APIs documentation 3. **Test endpoints** - Use the interactive examples to explore functionality 4. **Integrate** - Implement the APIs in your applications For SDK alternatives to these REST APIs, visit our [SDK documentation](/sdks). ## Explore More } title="SDK Documentation" description="Use our TypeScript SDKs for easier integration and development" href="/sdks" /> } title="Quick Start - Upload" description="Start uploading data to Arweave with our upload guides" href="/build/upload" /> } title="Quick Start - Access" description="Learn how to retrieve and query data from Arweave" href="/build/access" /> } title="Run a Gateway" description="Deploy your own ar.io gateway and access these APIs directly" href="/build/run-a-gateway" /> # Turbo APIs (/apis/turbo) Turbo is an ar.io gateway run by the ArDrive team: it serves permanent data like any ar.io gateway, and adds paid uploads. These APIs are the HTTP interface to the upload and payment half. ## Base URL Both services are served together from `https://turbo.ardrive.io`, a single consolidated host that amalgamates the upload service (`upload.ardrive.io`) and the payment service (`payment.ardrive.io`). The two are mounted at different prefixes, which is why the examples on the reference pages differ: | Service | Base URL | Example | | --- | --- | --- | | Upload | `https://turbo.ardrive.io` | `GET /price/:token/:byteCount` | | Payment | `https://turbo.ardrive.io/v1` | `GET /v1/price/bytes/:byteCount` | Upload endpoints are served at the root, **not** under `/v1` — prefixing them returns a 404. Payment endpoints require the `/v1` prefix. ## Services } title="Upload Service" description="Fast, reliable data uploads to Arweave with instant confirmation and metadata management" href="/apis/turbo/upload-service/upload" /> } title="Payment Service" description="Transparent pricing, payment processing, and credit management for Turbo uploads" href="/apis/turbo/payment-service/payments" /> ## Upload Service The Turbo Upload Service provides high-performance data uploads to the Arweave network with features including: - **Fast Uploads** - Optimized upload processing for quick data submission - **Instant Confirmation** - Immediate upload confirmations and transaction IDs - **Metadata Management** - Comprehensive data tagging and organization - **Account Management** - User account and upload history tracking - **Service Information** - Real-time service status and capabilities Key endpoints include account management, upload processing, pricing information, and transaction data retrieval. ## Payment Service The Turbo Payment Service handles all financial aspects of data uploads with transparent and flexible payment options: - **Transparent Pricing** - Clear, upfront costs for all upload operations - **Multiple Currencies** - Support for various payment methods and currencies - **Credit Management** - Prepaid credits and balance tracking - **Payment Processing** - Secure payment handling and transaction management - **Approval Workflows** - Payment authorization and confirmation flows Key endpoints include balance management, payment processing, pricing calculations, and credit redemption. ## Getting Started with Turbo 1. **Choose your service** - Upload for data submission, Payment for financial operations 2. **Review the APIs documentation** - Detailed endpoint specifications and examples 3. **Test with sample data** - Try uploads and payment flows with test data 4. **Integrate into your application** - Implement the APIs in your workflow ## Use the Turbo SDK For a more convenient integration experience, consider using the Turbo SDK instead of direct API calls: } title="Interact with Turbo via the SDK" description="Use the Turbo SDK for simplified integration with built-in error handling, retries, and TypeScript support" href="/sdks/turbo-sdk" /> The SDK provides a higher-level interface with built-in error handling, automatic retries, and full TypeScript support, making it easier to integrate Turbo services into your applications. # Approvals (/apis/turbo/payment-service/approvals) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Credit sharing and approval management # ArNS (/apis/turbo/payment-service/arns) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Arweave Name System (ArNS) registration and management # Balance (/apis/turbo/payment-service/balance) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Account balance and credit management # Currencies (/apis/turbo/payment-service/currencies) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Supported currencies and exchange rates # Info (/apis/turbo/payment-service/info) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Service information and metadata # Payments (/apis/turbo/payment-service/payments) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Payment processing and top-up operations # Pricing (/apis/turbo/payment-service/pricing) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Pricing and cost calculation endpoints # Protected (/apis/turbo/payment-service/protected) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Protected routes for inter-service communication (requires PRIVATE_ROUTE_SECRET) # Redemption (/apis/turbo/payment-service/redemption) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Credit redemption and gift processing # System (/apis/turbo/payment-service/system) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} System health and monitoring endpoints # Webhooks (/apis/turbo/payment-service/webhooks) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Webhook endpoints for external payment providers # x402 (/apis/turbo/payment-service/x402) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} x402 protocol payment endpoints (EIP-3009 USDC payments) # Account (/apis/turbo/upload-service/account) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Account balance and wallet information # Pricing (/apis/turbo/upload-service/pricing) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Pricing calculation endpoints # Service Info (/apis/turbo/upload-service/service-info) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Service information and health endpoints # Transaction Data (/apis/turbo/upload-service/transaction-data) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Transaction status and metadata retrieval # Upload (/apis/turbo/upload-service/upload) {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} Data item upload endpoints (single and multi-part) # Ar.io Name System (ArNS) (/build/access/arns) ArNS provides **human-readable URLs** for your Arweave data, making it easy to share and remember permanent addresses. ## What is ArNS? ArNS is a naming system that allows you to register human-readable names that point to your Arweave transactions. Instead of sharing long transaction IDs, you can use memorable URLs. **Example:** - **Before:** `https://turbo-gateway.com/bVLEkL1SOPFCzIYi8T_QNnh17VlDp4RylU6YTwCMVRw` - **After:** `https://myapp.ar.io` **Learn More:** For detailed information about ArNS architecture and how it works, see our [ArNS Documentation](/learn/arns). ## Get an ArNS Name The easiest way to get an ArNS name is via [arns.ar.io](https://arns.ar.io), which supports multiple payment methods: - **Fiat payments** - Credit cards and bank transfers - **Turbo Credits** - Use existing Turbo credits - **ARIO tokens** - Pay with ARIO cryptocurrency **Alternative registration method:** - **ar.io SDK** - Programmatic registration using the `buyRecord` API ### Using the ar.io SDK For developers, you can register ArNS names programmatically: ```js const ario = ARIO.mainnet(); // Buy a record with Turbo Credits or ARIO tokens const result = await ario.buyRecord({ name: 'my-domain', years: 1, // Payment method: 'turbo-credits' or 'ario-tokens' }); console.log('Record purchased:', result); ``` **Learn More:** For a complete list of ar.io SDK APIs, see the [ArNS SDK Documentation](/sdks/ar-io-sdk/ar-io-name-system-arns). ## Fetching Data via ArNS Once you've set up your ArNS name, fetch data using standard HTTP requests: ```js // Fetch content from your ArNS name const response = await fetch("https://my-data.ar.io"); if (!response.ok) { throw new Error(`HTTP error! status: ${response.status}`); } const data = await response.text(); console.log(data); ``` ## Why Use ArNS? ArNS provides significant advantages for accessing data on Arweave: **Decentralized Data Index** - ArNS creates a decentralized index of data accessible through any gateway in the ar.io network - No single point of failure - names resolve across all participating gateways - Censorship-resistant access to your content **Flexible Data Management** - **Permanent references** - Keep stable URLs even when updating underlying data - **Replaceable data** - Point names to new transaction IDs as content evolves - **Undernames** - Organize related content under a single name using underscores (e.g., `v2_myapp.ar.io`, `docs_myapp.ar.io`) **Supporting Network Decentralization** - ArNS purchases contribute to the protocol balance - Fees reward ar.io gateway operators for participating in the network - This economic model preserves decentralized access to data on Arweave - Your name registration helps maintain the infrastructure that serves your content ## Next Steps } > Register your own human-readable name on ArNS. } > Learn how to participate in the ar.io ecosystem. } > Advanced gateway routing for production applications. # Fetch Data (via REST API) (/build/access/fetch-data) The simplest way to access data on Arweave is through **HTTP requests** to gateways. This method works in any web browser and requires no additional setup. ## Fetching Data from Gateways Gateways are the most performant way to fetch data from Arweave, providing significant advantages over accessing Arweave nodes directly. **Why Gateways Are Faster:** - **Content Caching** - Pre-cached data for instant retrieval - **Data Indexing** - Fast search and query capabilities - **Network Optimization** - Distributed infrastructure for better performance - **Content Delivery** - Optimized serving with compression and CDN features ## REST APIs for Fetching Data Gateways support multiple API endpoints for accessing data: ### Standard Endpoint Access any transaction using this URL structure: ``` https:/// ``` **Examples:** - `https://turbo-gateway.com/bVLEkL1SOPFCzIYi8T_QNnh17VlDp4RylU6YTwCMVRw` - `https://turbo-gateway.com/FguFk5eSth0wO8SKfziYshkSxeIYe7oK9zoPN2PhSc0` ### Raw Data Endpoint For raw data access that bypasses manifest path resolution: ``` https:///raw/ ``` This endpoint returns the raw data bytes without resolving manifest paths, useful when you need the exact stored data. ## Providing Bundle Location Hints An ANS-104 data item is stored inside a bundle whose root transaction is written to Arweave. A gateway normally resolves the data item ID to that root transaction and the item's byte range automatically. If you already know this location, you can include it as request headers to avoid the lookup step: | Request header | Value | | --- | --- | | `X-AR-IO-Root-Transaction-Id` | ID of the root Arweave transaction containing the data item | | `X-AR-IO-Data-Item-Offset` | Byte offset of the complete data item within the root bundle | | `X-AR-IO-Data-Item-Size` | Size of the complete data item in bytes | For example, when all three values are known: ```bash curl --fail --location \ --header "X-AR-IO-Root-Transaction-Id: " \ --header "X-AR-IO-Data-Item-Offset: " \ --header "X-AR-IO-Data-Item-Size: " \ --output \ "https:///" ``` If only the root transaction ID is known, it can be supplied on its own. The gateway will use it to locate the bundle and determine the data item's offset and size: ```bash curl --fail --location \ --header "X-AR-IO-Root-Transaction-Id: " \ --output \ "https:///" ``` Location hints are an optional retrieval optimization, not a different data address. Keep requesting the data item ID in the URL. The gateway validates the resolved data item against that ID and returns an error if the hints do not identify the requested item. You can inspect a successful `HEAD` or `GET` response for the same `X-AR-IO-Root-Transaction-Id`, `X-AR-IO-Data-Item-Offset`, and `X-AR-IO-Data-Item-Size` headers. Save those values when you need to make later requests without repeating root-transaction discovery. **Learn More:** For complete API documentation and testing, see the [ar.io Node Data APIs](/apis/ar-io-node/data). ## Sandboxing Ar.io gateways implement security measures by redirecting requests to sandbox subdomains for enhanced browser security. **Why Redirects Happen:** - **Security Isolation** - Content is served from isolated sandbox environments - **CSP Protection** - Prevents cross-site scripting attacks - **Resource Isolation** - Limits potential security vulnerabilities - **Browser Sandboxing** - Leverages same-origin policy for enhanced security **What to Expect:** - Initial request: `https://turbo-gateway.com/transaction-id` - Redirects to: `https://sandbox.turbo-gateway.com/transaction-id` (or similar) - Final content served from sandbox subdomain **Important:** Always follow redirects in your applications - the final sandbox URL contains the actual content. **Learn More:** For detailed information about how browser sandboxing works and why it's important for security, see our [Browser Sandboxing](/build/advanced/sandboxing) documentation. ## Using in Applications **JavaScript Example with Fetch:** ```js // Fetch data from Arweave (follows redirects automatically) const response = await fetch("https://turbo-gateway.com/your-transaction-id", { redirect: "follow", // Follow redirects automatically }); if (!response.ok) { throw new Error(`HTTP error! status: ${response.status}`); } const data = await response.text(); console.log(data); ``` **HTML Example:** ```html ``` ## Manifests For organized file collections, use manifests to create friendly path-based URLs: ``` https://turbo-gateway.com//path/to/file ``` **Example:** - `https://turbo-gateway.com/X8Qm…AOhA/index.html` - `https://turbo-gateway.com/X8Qm…AOhA/styles.css` - `https://turbo-gateway.com/X8Qm…AOhA/assets/logo.png` [Learn more about manifests](/build/upload/manifests) ## Next Steps } > Discover data by searching with tags, metadata, and filters. }> Set up a gateway to serve and cache your specific data. }> Start uploading your data to Arweave's permanent storage. } > Automatically route requests to the best performing gateway. # Find Data (via GraphQL) (/build/access/find-data) Use **GraphQL** to **find and identify** Arweave data with powerful search and filtering capabilities. GraphQL is used for discovery - you query to get transaction IDs, then use those IDs to fetch the actual data. **GraphQL is for Discovery, Not Direct Access** GraphQL finds data, it doesn't access it directly. Use GraphQL to get transaction IDs, then use those IDs with the REST API to fetch the actual data. ## How GraphQL Works GraphQL on Arweave follows a two-step process: 1. **Find** - Query GraphQL to discover transactions by tags, metadata, owner, or other criteria 2. **Fetch** - Use the transaction IDs from your query results to retrieve the actual data via the REST API This separation allows for powerful data discovery while keeping data retrieval fast and efficient. ## GraphQL Providers - **turbo-gateway.com** - `https://turbo-gateway.com/graphql` - Comprehensive indexing of all Arweave data - **Goldsky** - `https://arweave-search.goldsky.com/graphql` - High-performance GraphQL service with full data coverage **ar.io Gateways:** ar.io gateways support the `/graphql` endpoint, but they only return data they've indexed. If you're uploading data and want it unbundled and indexed, you can run a gateway and configure it to unbundle your data, or post data items/bundles via the gateway's APIs (recommended). [Learn more](/build/run-a-gateway/manage/filters). ## Quick Start The easiest way to get started is using the interactive GraphQL playground: 1. Navigate to [https://turbo-gateway.com/graphql](https://turbo-gateway.com/graphql) in your browser 2. Enter your GraphQL query in the interface 3. Press the "play" button to execute and see results ## Basic Query Structure Try this example query in the playground - it fetches the most recent 10 HTML pages from "MyApp": ```graphql query { transactions( tags: [ { name: "Content-Type", values: ["text/html"] } { name: "App-Name", values: ["MyApp"] } ] sort: HEIGHT_DESC first: 10 ) { edges { node { id tags { name value } data { size } } } } } ``` ## Example Queries Here's how to find videos using GraphQL: ```js const query = ` query { transactions( tags: [{ name: "Content-Type", values: ["video/mp4"] }] first: 10 ) { edges { node { id tags { name value } data { size } } } } } `; const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ query }), }); const data = await response.json(); const videos = data.data.transactions.edges; // This returns transaction IDs that you can use with HTTP requests console.log( "Found video IDs:", videos.map((v) => v.node.id) ); ``` ```js const query = ` query { transactions(owners: ["your-wallet-address"], first: 10) { edges { node { id tags { name value } } } } } `; const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ query }), }); const data = await response.json(); const transactions = data.data.transactions.edges; console.log("Found transactions:", transactions.map(t => t.node.id)); ``` ```js const query = ` query { transactions(block: { min: 1000000, max: 1100000 }, first: 10) { edges { node { id block { height } } } } } `; const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ query }), }); const data = await response.json(); const transactions = data.data.transactions.edges; console.log("Found transactions in block range:", transactions.map(t => ({ id: t.node.id, height: t.node.block.height }))); ``` ```js // First page const query = ` query { transactions( tags: [{ name: "App-Name", values: ["MyApp"] }] first: 10 ) { pageInfo { hasNextPage } edges { cursor node { id tags { name value } } } } } `; const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ query }), }); const data = await response.json(); const { edges, pageInfo } = data.data.transactions; // Next page using cursor if (pageInfo.hasNextPage) { const nextQuery = ` query($cursor: String) { transactions( tags: [{ name: "App-Name", values: ["MyApp"] }] after: $cursor first: 10 ) { pageInfo { hasNextPage } edges { cursor node { id tags { name value } } } } } `; const nextResponse = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ query: nextQuery, variables: { cursor: edges[edges.length - 1].cursor }, }), }); const nextData = await nextResponse.json(); console.log(nextData.data); } ``` ## Pagination Different GraphQL providers may enforce varying limits on the maximum number of items returned per query. For larger datasets, use cursor-based pagination techniques to navigate through the complete set of results. **How Pagination Works:** - Use `first` parameter to specify page size - Use `pageInfo.hasNextPage` to check if more results exist - Use `cursor` from the last item with `after` parameter for the next page ```js let allTransactions = []; let hasNextPage = true; let cursor = null; while (hasNextPage) { const query = ` query($cursor: String) { transactions( tags: [{ name: "App-Name", values: ["MyApp"] }] first: 100 ${cursor ? "after: $cursor" : ""} ) { pageInfo { hasNextPage } edges { cursor node { id tags { name value } } } } } `; const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ query, variables: cursor ? { cursor } : {}, }), }); const data = await response.json(); const { edges, pageInfo } = data.data.transactions; allTransactions.push(...edges); hasNextPage = pageInfo.hasNextPage; cursor = edges.length > 0 ? edges[edges.length - 1].cursor : null; console.log( `Loaded ${edges.length} transactions. Total: ${allTransactions.length}` ); } console.log(`Found ${allTransactions.length} total transactions`); ``` ## Query Optimization Tips Follow these guidelines for optimal performance: **Specificity:** - Use the most precise tags possible to narrow search scope - Query with essential tags only to reduce processing time **Schema Design:** - Design your app's schema to reflect query patterns - Use tags that encapsulate frequent combinations of criteria **Include Non-tag Fields:** - Add fields like `owner` to refine your search - This makes queries more efficient and targeted **Order Your Tags:** - Arrange tags from most specific to most general - This leverages Arweave's indexing more effectively **Example Optimized Query:** ```js // Well-optimized query with specific tags and useful fields const query = ` query { transactions( tags: [ { name: "App-Name", values: ["MyApp"] } { name: "Content-Type", values: ["application/json"] } { name: "Version", values: ["1.0"] } ] owners: ["your-wallet-address"] first: 20 ) { edges { node { id data { size type } tags { name value } block { height timestamp } owner { address } } } } } `; ``` ## Next Steps } > Learn how to retrieve the actual data using transaction IDs. }> Set up a gateway to index and serve your specific data. }> Start uploading your data to Arweave's permanent storage. } > Automatically route requests to the best performing gateway. # Access Data (/build/access) Once data is stored on Arweave, it's permanently available. Here's how to access it efficiently for your applications. ## Access Methods Different methods serve different needs. Each provides unique capabilities for retrieving data from Arweave. Search and discover data on Arweave Query by tags and metadata Filter by app, owner, timestamp Get transaction IDs for fetching } href="/build/access/find-data" icon={} /> Retrieve data bytes from Arweave REST API endpoints GET turbo-gateway.com/[txId] Returns raw data/files } href="/build/access/fetch-data" icon={} /> Assign names to data and apps Create names like ardrive.ar.io Point to any Arweave data Update targets as needed } href="/build/access/arns" icon={} /> ## Common Access Patterns **Finding Data** - Search for data by tags, owner, or timestamp - Discover content from specific applications - Get transaction IDs for data retrieval **Fetching Data** - Retrieve the actual files/data using transaction IDs - Access data via REST API: `GET turbo-gateway.com/[txId]` - Stream large files efficiently **Naming with ArNS** - Register memorable names for your apps and data - Create permanent links like `ardrive.ar.io` - Update where names point without changing the URL ## Quick Example: Find and Fetch ### Find Data Use GraphQL to search for data and get transaction IDs: ```graphql query { transactions( tags: [{ name: "App-Name", values: ["ArDrive"] }] first: 1 ) { edges { node { id } } } } ``` ### Fetch Data Use the transaction ID to retrieve the actual data: ```bash curl https://turbo-gateway.com/[transaction-id-from-above] ``` ## Additional Access Options # Wayfinder (/build/access/wayfinder) Wayfinder is a client-side routing and verification protocol that provides **decentralized, cryptographically verified access** to data stored on Arweave via ar.io. ## What is Wayfinder? Wayfinder solves the challenge of reliable data access on the permaweb by: - **Intelligent Routing** - Automatically selects the best gateway for each request - **Data Verification** - Cryptographically verifies data integrity - **Decentralized Access** - Eliminates single points of failure - **Seamless Integration** - Works behind the scenes for fast, reliable access **Learn More:** For detailed information about Wayfinder architecture and how it works, see our [Wayfinder Documentation](/learn/wayfinder). ## Get Started **Installation:** ```npm npm install @ar.io/wayfinder-core @ar.io/sdk ``` **Basic Usage:** ```js // Create wayfinder with default settings const wayfinder = createWayfinderClient({ ario: ARIO.mainnet(), }); // Fetch data using ar:// protocol try { const response = await wayfinder.request("ar://transaction-id"); const data = await response.text(); console.log("Data:", data); } catch (error) { console.error("Failed to fetch data:", error); } ``` **Full API Reference:** For complete documentation of all Wayfinder core APIs, see the [Wayfinder Core SDK Reference](/sdks/wayfinder/wayfinder-core). ## React Integration For React applications, use the wayfinder-react package: ```npm npm install @ar.io/wayfinder-react @ar.io/sdk ``` ```jsx function App() { return ( ); } function YourComponent() { const request = useWayfinderRequest(); const [data, setData] = useState(null); useEffect(() => { (async () => { const response = await request(`ar://${txId}`, { verificationSettings: { enabled: true, strict: true, }, }); const data = await response.arrayBuffer(); setData(data); })(); }, [request, txId]); return {data && {data}}; } ``` **Full API Reference:** For complete documentation of all Wayfinder React APIs, see the [Wayfinder React SDK Reference](/sdks/wayfinder/wayfinder-react). ## Why Use Wayfinder? Wayfinder eliminates centralized points of failure by distributing data access across ar.io, reducing dependency on arweave.net and providing advanced capabilities for production applications: **Maximum Reliability** - Intelligent gateway selection eliminates single points of failure - Automatic failover ensures data is always accessible - Built-in retry mechanisms handle network issues gracefully **Data Verification** - Cryptographic verification ensures data integrity - Multiple verification strategies protect against tampering - Trust but verify approach validates all responses **Performance Optimization** - Fastest ping routing selects optimal gateways - Round-robin distribution balances load across the network - Caching strategies reduce latency for frequently accessed data **Production Ready** - Developer-friendly APIs with React integration - Comprehensive error handling and logging - Configurable routing and verification strategies ## Next Steps } > Start building with the Wayfinder SDK. } > Use REST API for basic data retrieval. }> Use GraphQL to search for data. } > Create memorable names for your Arweave data. # Creating Drives (/build/advanced/arfs/creating-drives) To properly create a new drive, two new entities need to be created: a new Drive entity and a new Folder entity to serve as the root folder of that drive. ## New Drive Entity - The user must specify a `name` of the drive which is stored within the Drive Entity's metadata JSON. - ArDrive generates a new unique uuidv4 for the drive entity's `Drive-Id`. - ArDrive also generates a new unique uuidv4 for the drive entity's `rootFolderId`, which will refer to the `Folder-Id` of the new folder entity that will be created. - This `rootFolderId` is stored within the Drive Entity's metadata JSON. - Drive Entity Metadata transactions must have `Entity-Type: "drive"`. - ArDrive will that the current local system time as seconds since Unix epoch for the Drive Entity's `Unix-Time`. - The Drive Entity's `Drive-Privacy` must also be set to `public` or `private` in order for its subfolders and files to have the correct security settings. - If the drive is private: - Its `Cipher` tag must be filled out with the correct encryption algorithm (currently `AES256-GCM`). - Its `Cipher-IV` tag must be filled out with the generated Initialization Vector for the private drive. - The ArFS client must derive the Drive Key and encrypt the Drive Entity's metadata JSON using the assigned `Cipher` and `Cipher-IV`. ## New Root Folder Entity - The `name` of the drive and folder entities must be the same. - This `name` is stored within the Folder Entity's metadata JSON. - The Folder Entity's `Folder-Id` must match the `rootFolderId` previously created for the Drive Entity. - The Folder Entity's `Drive-Id` must match the `Drive-Id` previously created for the Drive Entity. - The Folder Entity must not include a `Parent-Folder-Id` tag. - This is how it is determined to be the root folder for a drive. - Folder Entity metadata transactions must have `Entity-Type: 'folder'`. - The client gets the user's local time for the `Unix-Time` tag, represented as seconds since Unix Epoch. - Public folders must have the content type `Content-Type: "application/json"`. - If the folder is private - Its `Cipher` tag must be filled out with the correct encryption algorithm (currently `AES256-GCM`). - Its `Cipher-IV` tag must be filled out with the generated Initialization Vector for the private folder. - Its content type must be `Content-Type: "application/octet-stream"`. - The ArFS client must encrypt the Drive Entity's metadata JSON using the assigned `Cipher` and `Cipher-IV`. ## Creating Files Files in ArFS require two separate transactions: 1. **File Metadata Transaction** - Contains file information and references 2. **File Data Transaction** - Contains the actual file data ### File Metadata Transaction ```json ArFS: "0.15", Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", Drive-Id: "", Entity-Type: "file", File-Id: "", Parent-Folder-Id: "", Unix-Time: "" Metadata JSON { "name": "", "size": , "lastModifiedDate": , "dataTxId": "", "dataContentType": "", "isHidden": false, "pinnedDataOwner": "" } ``` ### File Data Transaction ```json Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", { File Data - Encrypted if private } ``` ## Creating Folders Folders are simpler than files as they only require a metadata transaction: ```json ArFS: "0.15", Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", Drive-Id: "", Entity-Type: "folder", Folder-Id: "", Parent-Folder-Id?: "", Unix-Time: "" Metadata JSON { "name": "", "isHidden": false } ``` ## Creating Snapshots Snapshots provide a way to quickly synchronize drive state by rolling up all metadata into a single transaction: ```json ArFS: "0.15", Drive-Id: "", Entity-Type: "snapshot", Snapshot-Id: "", Content-Type: "", Block-Start: "", Block-End: "", Data-Start: "", Data-End: "", Unix-Time: "" ``` ## Implementation Example Here's a practical example of creating a complete drive structure: ```mermaid sequenceDiagram participant User participant Client participant Wallet participant Arweave User->>Client: Create drive "My Project" Client->>Client: Generate drive UUID Client->>Client: Generate root folder UUID Client->>Wallet: Request signature (if private) Wallet->>Client: Return signature Client->>Client: Derive drive key (if private) Client->>Client: Encrypt metadata (if private) Client->>Arweave: Upload drive entity Client->>Arweave: Upload root folder entity User->>Client: Create folder "Documents" Client->>Client: Generate folder UUID Client->>Client: Encrypt folder metadata (if private) Client->>Arweave: Upload folder entity User->>Client: Upload file "readme.txt" Client->>Client: Generate file UUID Client->>Client: Encrypt file metadata (if private) Client->>Client: Encrypt file data (if private) Client->>Arweave: Upload file metadata Client->>Arweave: Upload file data ``` ## Best Practices ### Naming Conventions - Use descriptive names for drives, folders, and files - Avoid special characters that might cause issues - Keep names under 255 characters - Use consistent casing ### Organization - Create logical folder structures - Use meaningful folder names - Implement proper versioning - Document your structure ### Performance - Batch operations when possible - Use efficient queries - Implement caching - Consider file sizes ### Security - Use strong passwords for private drives - Implement proper key management - Follow encryption best practices - Regular security audits ## Error Handling When creating ArFS entities, handle these common scenarios: ### Transaction Failures - Implement retry logic for failed uploads - Validate data before uploading - Check transaction confirmation status ### Validation Errors - Verify required tags are present - Check data format compliance - Validate UUID formats ### Network Issues - Implement timeout handling - Provide user feedback - Graceful degradation ## Next Steps Now that you know how to create ArFS entities, learn how to work with them: - [Reading Data](/build/advanced/arfs/reading-data) - Query and retrieve your ArFS data - [Privacy & Encryption](/build/advanced/arfs/privacy) - Secure your data with private drives - [Upgrading Private Drives](/build/advanced/arfs/upgrading-drives) - Update legacy drives to v0.15 # Data Model (/build/advanced/arfs/data-model) Because of Arweave's permanent and immutable nature, traditional file structure operations such as renaming and moving files or folders cannot be accomplished by simply updating onchain data. ArFS works around this by defining an append-only transaction data model based on the metadata tags found in the Arweave [Transaction Headers.](https://docs.arweave.org/developers/server/http-api#transaction-format) This model uses a bottom-up reference method, which avoids race conditions in file system updates. Each file contains metadata that refers to the parent folder, and each folder contains metadata that refers to its parent drive. A top-down data model would require the parent model (i.e. a folder) to store references to its children. These defined entities allow the state of the drive to be constructed by a client to look and feel like a file system: - Drive Entities contain folders and files - Folder Entities contain other folders or files - File Entities contain both the file data and metadata - Snapshot entities contain a state rollups of all entities' (such as drive, folder, file and snapshot) metadata within a drive ## Entity Relationships The following diagram shows the high level relationships between drive, folder, and file entities, and their associated data. More detailed information about each Entity Type can be found in the ArFS specification documentation. ```mermaid graph TD A[Drive Entity] --> B[Root Folder] B --> C[Subfolder 1] B --> D[Subfolder 2] B --> E[File 1] C --> F[File 2] C --> G[File 3] D --> H[File 4] D --> I[Subfolder 3] I --> J[File 5] A --> K[Drive Metadata] B --> L[Folder Metadata] C --> M[Folder Metadata] D --> N[Folder Metadata] I --> O[Folder Metadata] E --> P[File Metadata + Data] F --> Q[File Metadata + Data] G --> R[File Metadata + Data] H --> S[File Metadata + Data] J --> T[File Metadata + Data] U[Snapshot Entity] --> V[Complete Drive State] V --> A V --> B V --> C V --> D V --> I V --> E V --> F V --> G V --> H V --> J ``` As you can see, each file and folder contains metadata which points to both the parent folder and the parent drive. The drive entity contains metadata about itself, but not the child contents. So clients must build drive states from the lowest level and work their way up. ## Metadata Format Metadata stored in any Arweave transaction tag will be defined in the following manner: ```json { "name": "Example-Tag", "value": "example-data" } ``` Metadata stored in the Transaction Data Payload will follow JSON formatting like below: ```json { "exampleField": "exampleData" } ``` Fields with a `?` suffix are optional. ```json { "name": "My Project", "description": "This is a sample project.", "version?": "1.0.0", "author?": "John Doe" } ``` Enumerated field values (those which must adhere to certain values) are defined in the format "value 1 | value 2". All UUIDs used for Entity-Ids are based on the [Universally Unique Identifier](https://en.wikipedia.org/wiki/Universally_unique_identifier) standard. There are no requirements to list ArFS tags in any specific order. ## Building Drive State To construct the current state of a drive, clients must: 1. **Query for all entities** associated with a specific `Drive-Id` 2. **Sort by block height** to establish chronological order 3. **Process entities bottom-up** starting with files and folders 4. **Build the hierarchy** by following parent-child relationships 5. **Handle conflicts** by using the most recent entity version ### Example Drive State Construction ```mermaid sequenceDiagram participant Client participant Gateway participant Arweave Client->>Gateway: Query Drive-Id: abc123 Gateway->>Client: Return all entities Client->>Client: Sort by block height Client->>Client: Process files first Client->>Client: Process folders Client->>Client: Process drive metadata Client->>Client: Build hierarchy tree Client->>Client: Resolve conflicts Client->>Client: Return complete drive state ``` ## Entity Lifecycle Each ArFS entity follows a specific lifecycle pattern: ### Creation 1. Generate unique UUID for entity 2. Create metadata transaction with required tags 3. For files: create separate data transaction 4. Upload to Arweave network ### Updates 1. Create new entity with same ID 2. Update metadata as needed 3. Upload new transaction 4. Client processes both versions and uses latest ### Hiding 1. Mark entity as hidden (`isHidden: true`) 2. Upload new transaction 3. Entity remains in history but hidden from UI Nothing is deleted: because Arweave transactions are immutable, hiding an entity is reversible. Every prior revision stays on Arweave, and the entity reappears if a later revision sets `isHidden: false`. ## Data Integrity ArFS ensures data integrity through: - **Immutable transactions** - Once uploaded, data cannot be modified - **Cryptographic signatures** - All transactions are signed by the owner - **Version tracking** - Multiple versions of entities can exist - **Conflict resolution** - Clients use block height and timestamps to resolve conflicts ## Performance Considerations For large drives, consider these optimization strategies: - **Use snapshots** for quick state reconstruction - **Implement caching** for frequently accessed data - **Batch operations** when possible - **Query by date ranges** to limit data transfer ## Next Steps Now that you understand the ArFS data model, learn how to work with it: - [Privacy & Encryption](/build/advanced/arfs/privacy) - Secure your data with private drives - [Creating Drives](/build/advanced/arfs/creating-drives) - Start building with ArFS - [Reading Data](/build/advanced/arfs/reading-data) - Query and retrieve your data # Entity Types (/build/advanced/arfs/entity-types) ## Overview Arweave transactions provide for a separation between data and metadata about that data via the use of headers. Key-value tags in the headers provide for expressive description about the data as well as searchability via gateway GraphQL APIs. ArFS adds an additional layer of separation between data and metadata by using separate transactions for ArFS metadata and, where applicable, ArFS file data. But it also makes use of tags and data separation within an ArFS metadata transaction by including data critical to tracking drive composition in the tags space of ArFS metadata transactions and having most of the other metadata encoded as JSON in the data body of the metadata transaction. In the case of private entities, JSON data and file data payloads are always encrypted according to the protocol processes defined below. - Drive entities require a single metadata transaction, with standard Drive tags and encoded JSON with secondary metadata. - Folder entities require a single metadata transaction, with standard Folder tags and an encoded JSON with secondary metadata. - File entities require a metadata transaction, with standard File tags and an encoded Data JSON with secondary metadata relating to the file. - File entities also require a second data transaction, which includes a limited set of File tags and the actual file data itself. - Snapshot entities require a single transaction, which contains a Data JSON with all of the Drive's rolled up ArFS metadata and standard Snapshot GQL tags that identify the Snapshot. ArFS v0.14 introduces the `isHidden` property. `isHidden` is a boolean (true/false) that tells clients if they should display the file or folder. Hidden files still exist and will be included in [snapshots](#snapshot), but should not be rendered by clients. If `isHidden` is not present, its value should be assumed false. ArFS v0.15 introduces the `Signature-Type` metadata property on Drive entities, and a new entity type `DriveSignature`. ## Drive A drive is the highest level logical grouping of folders and files. All folders and files must be part of a drive, and reference the Drive ID of that drive. When creating a Drive, a corresponding "root" folder must be created as well. This separation of drive and folder entity enables features such as folder view queries, renaming, and linking. ```json ArFS: "0.15", Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", Drive-Id: "", Drive-Privacy: "", Drive-Auth-Mode?: "password", Entity-Type: "drive", Signature-Type?: "1", Unix-Time: "" Metadata JSON { "name": "", "rootFolderId": "", "isHidden": false } ``` ## Drive-Signature ArFS versions prior to v0.15 applied encryption to drive contents with a signing scheme that, while secure, is now deprecated in modern Arweave software wallets. ArFS v0.15 introduces an updated signing scheme compatible with these wallets and as well as "Drive Signatures", a new entity type to help bridge the signature derivation schemes across ArFS versions. A drive signature uses the v0.15 encryption scheme to encrypt and store the pre-v0.15 wallet signature for a private drive that is necessary for deriving the "drive key" for that drive. This allows for continued access of historical drive contents into the future. ```json ArFS: "0.15", Entity-Type: "drive-signature", Signature-Format: "1", Cipher?: "AES256-GCM", Cipher-IV: "" {data: } ``` The encrypted "type 1" signature for the drive must be provided in the `data` field of the transaction creating the drive-signature entity. ## Folder A folder is a logical grouping of other folders and files. Folder entity metadata transactions without a parent folder id are considered the Drive Root Folder of their corresponding Drives. All other Folder entities must have a parent folder id. Since folders do not have underlying data, there is no Folder data transaction required. ```json ArFS: "0.15", Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", Drive-Id: "", Entity-Type: "folder", Folder-Id: "", Parent-Folder-Id?: "", Unix-Time: "" Metadata JSON { "name": "", "isHidden": false } ``` ## File A File contains uploaded data, like a photo, document, or movie. In the Arweave File System, a single file is broken into 2 parts - its metadata and its data. A File entity metadata transaction does not include the actual File data. Instead, the File data must be uploaded as a separate transaction, called the File Data Transaction. The File JSON metadata transaction contains a reference to the File Data Transaction ID so that it can retrieve the actual data. This separation allows for file metadata to be updated without requiring the file itself to be reuploaded. It also ensures that private files can have their JSON Metadata Transaction encrypted as well, ensuring that no one without authorization can see either the file or its metadata. ```json ArFS: "0.15", Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", Drive-Id: "", Entity-Type: "file", File-Id: "", Parent-Folder-Id: "", Unix-Time: "" Metadata JSON { "name": "", "size": , "lastModifiedDate": , "dataTxId": "", "dataContentType": "", "isHidden": false, "pinnedDataOwner": "", # Optional } ``` ### Pinning Files Since the version v0.13, ArFS supports Pins. Pins are files whose data may be any transaction uploaded to Arweave, that may or may not be owned by the wallet that created the pin. When a new File Pin is created, the only created transaction is the Metadata Transaction. The `dataTxId` field will point it to any transaction in Arweave, and the optional `pinnedDataOwner` field is gonna hold the address of the wallet that owns the original copy of the data transaction. ### File Data Transaction Example The File Data Transaction contains limited information about the file, such as the information required to decrypt it, or the Content-Type (mime-type) needed to view in the browser. ```json Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", { File Data - Encrypted if private } ``` ### File Metadata Transaction Example The File Metadata Transaction contains the GQL Tags necessary to identify the file within a drive and folder. Its data contains the JSON metadata for the file. This includes the file name, size, last modified date, data transaction id, and data content type. ```json ArFS: "0.15", Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", Drive-Id: "", Entity-Type: "file", File-Id: "", Parent-Folder-Id: "", Unix-Time: "", { File JSON Metadata - Encrypted if private } ``` ## Snapshot ArFS applications generate the latest state of a drive by querying for all ArFS transactions made relating to a user's particular `Drive-Id`. This includes both paged queries for indexed ArFS data via GQL, as well as the ArFS JSON metadata entries for each ArFS transaction. For small drives (less than 1000 files), a few thousand requests for very small volumes of data can be achieved relatively quickly and reliably. For larger drives, however, this results in long sync times to pull every piece of ArFS metadata when the local database cache is empty. This can also potentially trigger rate-limiting related ArWeave Gateway delays. Once a drive state has been completely, and accurately generated, in can be rolled up into a single snapshot and uploaded as an Arweave transaction. ArFS clients can use GQL to find and retrieve this snapshot in order to rapidly reconstitute the total state of the drive, or a large portion of it. They can then query individual transactions performed after the snapshot. This optional method offers convenience and resource efficiency when building the drive state, at the cost of paying for uploading the snapshot data. Using this method means a client will only have to iterate through a few snapshots instead of every transaction performed on the drive. ### Snapshot Entity Tags Snapshot entities require the following tags. These are queried by ArFS clients to find drive snapshots, organize them together with any other transactions not included within them, and build the latest state of the drive. ```json ArFS: "0.15", Drive-Id: "", Entity-Type: "snapshot", Snapshot-Id: "", Content-Type: "", Block-Start: "", Block-End: "", Data-Start: "" ``` ### Snapshot Entity Data A JSON data object must also be uploaded with every ArFS Snapshot entity. This data contains all ArFS Drive, Folder, and File metadata changes within the associated drive, as well as any previous Snapshots. The Snapshot Data contains an array `txSnapshots`. Each item includes both the GQL and ArFS metadata details of each transaction made for the associated drive, within the snapshot's start and end period. Each `txSnapshot` has exactly two fields, `gqlNode` and `jsonMetadata`, as siblings. `gqlNode` uses the same GQL tags interface returned by the Arweave Gateway. It includes all of the important `block`, `owner`, `tags`, and `bundledIn` information needed by ArFS clients. `jsonMetadata` is a **string**, not an object: the correlated Data JSON for that ArFS entity, serialized and escaped. It is `null` for an entry recorded for its GQL node alone, with no metadata body — a client must handle that rather than assume a string is always present. For private drives, `jsonMetadata` contains the JSON-string-escaped encrypted text of the associated file or folder. This encrypted text uses the file's existing `Cipher` and `Cipher-IV`. This ensures clients can decrypt this information quickly using the existing ArFS privacy protocols. ```json { "txSnapshots": [ { "gqlNode": { "id": "bWCvIc3cOzwVgquD349HUVsn5Dd1_GIri8Dglok41Vg", "owner": { "address": "hlWRbyJ6WUoErm3b0wqVgd1l3LTgaQeLBhB36v2HxgY" }, "bundledIn": { "id": "39n5evzP1Ip9MhGytuFm7F3TDaozwHuVUbS55My-MBk" }, "block": { "height": 1062005, "timestamp": 1669053791 }, "tags": [ { "name": "Content-Type", "value": "application/json" }, { "name": "ArFS", "value": "0.11" }, { "name": "Entity-Type", "value": "drive" }, { "name": "Drive-Id", "value": "f27abc4b-ed6f-4108-a9f5-e545fc4ff55b" }, { "name": "Drive-Privacy", "value": "public" }, { "name": "App-Name", "value": "ArDrive-App" }, { "name": "App-Platform", "value": "Web" }, { "name": "App-Version", "value": "1.39.0" }, { "name": "Unix-Time", "value": "1669053323" } ] }, "jsonMetadata": "{\"name\":\"november\",\"rootFolderId\":\"71dfc1cb-5368-4323-972a-e9dd0b1c63a0\", \"isHidden\":false}" } ] } ``` ## Schema Diagrams The following diagrams show complete examples of Drive, Folder, and File entity Schemas. ### Public Drive ```mermaid graph TD A[Drive Entity] --> B[Drive Metadata JSON] A --> C[Drive Tags] C --> D[ArFS: 0.15] C --> E[Entity-Type: drive] C --> F[Drive-Id: uuid] C --> G[Drive-Privacy: public] C --> H[Unix-Time: timestamp] B --> I[name: string] B --> J[rootFolderId: uuid] B --> K[isHidden: boolean] ``` ### Private Drive ```mermaid graph TD A[Drive Entity] --> B[Encrypted Drive Metadata JSON] A --> C[Drive Tags] C --> D[ArFS: 0.15] C --> E[Entity-Type: drive] C --> F[Drive-Id: uuid] C --> G[Drive-Privacy: private] C --> H[Drive-Auth-Mode: password] C --> I[Signature-Type: 1] C --> J[Cipher: AES256-GCM] C --> K[Cipher-IV: base64] C --> L[Content-Type: application/octet-stream] C --> M[Unix-Time: timestamp] B --> N[Encrypted JSON with name, rootFolderId, isHidden] ``` ## Next Steps Now that you understand the different ArFS entity types, explore how they work together: - [Data Model](/build/advanced/arfs/data-model) - Learn how entities relate to each other - [Privacy & Encryption](/build/advanced/arfs/privacy) - Understand how private entities work - [Creating Drives](/build/advanced/arfs/creating-drives) - Start building with ArFS # ArFS Protocol (/build/advanced/arfs) Arweave File System, or "ArFS" is a data modeling, storage, and retrieval protocol designed to emulate common file system operations and to provide aspects of mutability to your data hierarchy on [Arweave](/learn/what-is-arweave)'s otherwise permanent, immutable data storage blockweave. Due to Arweave's permanent, immutable and public nature traditional file system operations such as permissions, file/folder renaming and moving, and file updates cannot be done by simply updating the onchain data model. ArFS works around this by implementing a privacy and encryption pattern and defining an append-only transaction data model using tags within [Arweave Transaction headers](https://docs.arweave.org/developers/server/http-api#transaction-format). ## Key Features ### File Structure ArFS organizes files and folders using a hierarchical structure. Files are stored as individual transactions on the Arweave blockchain, while folders are metadata that reference these file transactions. ### Metadata Each file and folder has associated metadata, such as the name, type, size, and modification timestamp. ArFS uses Arweave's tagging system to store this metadata in a standardized format, which allows for easy querying and organization. ### File Permissions ArFS supports public and private file permissions. Public files can be accessed by anyone on the network, while private files are encrypted using the owner's private key, ensuring only they can decrypt and access the content. ### File Versioning ArFS supports versioning of files, allowing users to store multiple versions of a file and access previous versions at any time. This is achieved by linking new file transactions to previous versions through the use of metadata tags. ### Search and Discovery ArFS enables users to search and discover files based on their metadata, such as file names, types, and tags. This is made possible by indexing the metadata stored within the Arweave blockchain. ### Interoperability ArFS is designed to be interoperable with other decentralized applications and services built on the Arweave network. Different applications and users can therefore work with the same files. ## Getting Started To start using ArFS, you'll need to familiarize yourself with the Arweave ecosystem, acquire AR tokens to cover storage costs, and choose a compatible client or library to interact with the ArFS protocol. ## ArFS Version History | Version | Date | Release Notes | | ------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | 0.10 | August 2020 | The brief, beta version that was in use during initial testing of ArDrive across Web (Dart) and legacy CLI (Typescript). | | 0.11 | September 2020 | Includes all of the major functionality supporting file systems on Arweave including new drives, folders, files, renames, moves and privacy. | | 0.12 | December 2022 | Added Snapshot entities to support quick synchronization of drive state. | | 0.13 | August 2023 | Added pins | | 0.14 | January 2024 | Added `isHidden` property to file and folder metadata to enable clients to "hide" content from end users. | | 0.15 | May 2025 | Added `Drive-Signature` entity type and `Signature-Type` metadata property on Drive entities. | ## Next Steps Ready to dive deeper into ArFS? Here's what you should explore next: - [Entity Types](/build/advanced/arfs/entity-types) - Understand the different ArFS entities and their structure - [Data Model](/build/advanced/arfs/data-model) - Learn how ArFS organizes data hierarchically - [Privacy & Encryption](/build/advanced/arfs/privacy) - Secure your data with private drives - [Creating Drives](/build/advanced/arfs/creating-drives) - Get started with your first ArFS drive - [Reading Data](/build/advanced/arfs/reading-data) - Query and retrieve your ArFS data ## Resources For more information, documentation, and community support, refer to the following resources: - [Arweave Official Website](https://www.arweave.org/) - [Arweave Developer Documentation](https://docs.arweave.org/) - [Arweave Community Discord](https://discord.gg/arweave) # Privacy & Encryption (/build/advanced/arfs/privacy) The Arweave blockweave is inherently public. But with apps that use ArFS, like ArDrive, your private data never leaves your computer without using military grade (and [quantum resistant](https://blog.boot.dev/cryptography/is-aes-256-quantum-resistant/#:~:text=Symmetric%20encryption%2C%20or%20more%20specifically,key%20sizes%20are%20large%20enough)) encryption. This privacy layer is applied at the Drive level, and users determine whether a Drive is public or private when they first create it. Private drives must follow the ArFS privacy model. With ArDrive specifically, every file within a Private Drive is symmetrically encrypted using [AES-256-GCM](https://iopscience.iop.org/article/10.1088/1742-6596/1019/1/012008/pdf) (for small files and metadata transactions) or [AES-256-CTR](https://xilinx.github.io/Vitis_Libraries/security/2020.1/guide_L1/internals/ctr.html) (for large files, over 100MiB). Every Private drive has a top-level "Drive Key" which uses a combination of the user's Arweave wallet signature, a user defined drive password, and a unique drive identifier ([uuidv4](https://en.wikipedia.org/wiki/Universally_unique_identifier)). Each file has its own "File Key" derived from the "Drive Key". This allows for single files to be shared without exposing access to the other files within the Drive. Once a file is encrypted and stored on Arweave, it is locked forever and can only be decrypted using its file key. **NOTE**: Usable encryption standards are not limited to AES-256-GCM or AES-256-CTR. Any Encryption method may be used so long as it is clearly indicated in the `cipher` tag. ## Deriving Keys Private drives have a global drive key, `D`, and multiple file keys `F`, for encryption. This enables a drive to have as many uniquely encrypted files as needed. One key is used for all versions of a single file (since new file versions use the same File-Id) `D` is used for encrypting both Drive and Folder metadata, while `F` is used for encrypting File metadata and the actual stored data. Having these different keys, `D` and `F`, allows a user to share specific files without revealing the contents of their entire drive. `D` is derived using HKDF-SHA256 with an [unsalted]() RSA-PSS signature of the drive's id and a user provided password. `F` is also derived using HKDF-SHA256 with the drive key and the file's id. ```mermaid graph TD A[User Password] --> B[Drive Key Derivation] C[Wallet Signature] --> B D[Drive ID] --> B B --> E[Drive Key D] E --> F[File Key Derivation] G[File ID] --> F F --> H[File Key F] E --> I[Encrypt Drive Metadata] E --> J[Encrypt Folder Metadata] H --> K[Encrypt File Metadata] H --> L[Encrypt File Data] style A fill:#e1f5fe style C fill:#e1f5fe style D fill:#e1f5fe style G fill:#e1f5fe style E fill:#c8e6c9 style H fill:#c8e6c9 ``` Other wallets (like [ArConnect](https://www.arconnect.io/)) integrate with this Key Derivation protocol just exposing an API to collect a signature from a given Arweave Wallet in order to get the SHA-256 signature needed for the [HKDF](https://en.wikipedia.org/wiki/HKDF) to derive the Drive Key. An example implementation, using Dart, is available [here](https://github.com/ardriveapp/ardrive-web/blob/187b3fb30808bda452123c2b18931c898df6a3fb/docs/private_drive_kdf_reference.dart), with a Typescript implementation [here](https://github.com/ardriveapp/ardrive-core-js/blob/f19da30efd30a4370be53c9b07834eae764f8535/src/utils/crypto.ts). ## Private Drives Drives can store either public or private data. This is indicated by the `Drive-Privacy` tag in the Drive entity metadata. ``` Drive-Privacy: "" ``` If a Drive entity is private, an additional tag `Drive-Auth-Mode` must also be used to indicate how the Drive Key is derived. ArDrive clients currently use a secure password along with the Arweave Wallet private key signature to derive the global Drive Key. ``` Drive-Auth-Mode?: 'password' ``` On every encrypted Drive Entity, a `Cipher` tag must be specified, along with the public parameters for decrypting the data. This is done by specifying the parameter with a `Cipher-*` tag. eg. `Cipher-IV`. If the parameter is byte data, it must be encoded as Base64 in the tag. ArDrive clients currently use AES256-GCM for all symmetric encryption, which requires a Cipher Initialization Vector consisting of 12 random bytes. ``` Cipher?: "AES256-GCM" Cipher-IV?: "" ``` Additionally, all encrypted transactions must have the `Content-Type` tag `application/octet-stream` as opposed to `application/json` Private Drive Entities and their corresponding Root Folder Entities will both use these keys and ciphers generated to symmetrically encrypt the JSON files that are included in the transaction. This ensures that only the Drive Owner (and whomever the keys have been shared with) can open the drive, discover the root folder, and continue to load the rest of the children in the drive. ## Private Files When a file is uploaded to a private drive, it by default also becomes private and uses the same drive keys used for its parent drive. Each unique file in a drive will get its own set of file keys based off of that file's unique `FileId`. If a single file gets a new version, its `File-Id` will be reused, so the same File Key is used for all versions in that file's history. These file keys can be shared by the drive's owner as needed. Private File entities have both its metadata and data transactions encrypted using the same File Key, ensuring all facets of the data is truly private. As such, both the file's metadata and data transactions must both have a unique `Cipher-IV` and `Cipher` tag: ``` Cipher?: "AES256-GCM" Cipher-IV?: "" ``` Just like drives, private files must have the `Content-Type` tag set as `application/octet-stream` in both its metadata and data transactions: ``` Content-Type: "application/octet-stream" ``` ## Encryption Process Here's how the encryption process works for private drives: ```mermaid sequenceDiagram participant User participant Client participant Wallet participant Arweave User->>Client: Create private drive Client->>Wallet: Request signature Wallet->>Client: Return signature Client->>Client: Derive drive key Client->>Client: Encrypt drive metadata Client->>Arweave: Upload encrypted drive User->>Client: Upload file to private drive Client->>Client: Derive file key Client->>Client: Encrypt file metadata Client->>Client: Encrypt file data Client->>Arweave: Upload encrypted metadata Client->>Arweave: Upload encrypted data ``` ## Security Best Practices When working with private drives, follow these security guidelines: ### Password Management - Use strong, unique passwords for each drive - Consider using a password manager - Never share passwords in plain text ### Key Storage - Never store drive keys in plain text - Use secure key derivation functions - Implement proper key rotation if needed ### Access Control - Share file keys only with authorized users - Implement proper access logging - Regularly audit drive access ### Data Handling - Encrypt data before transmission - Use secure communication channels - Implement proper error handling ## Drive Signature (ArFS v0.15) ArFS v0.15 introduces a new `Drive-Signature` entity type to help bridge signature derivation schemes across ArFS versions. This is particularly important for maintaining access to private drives created with older wallet signing methods. The drive signature entity stores an encrypted version of the pre-v0.15 wallet signature that's necessary for deriving the drive key. This allows continued access to historical drive contents while using modern wallet signing APIs. ```mermaid graph TD A[Legacy Wallet Signature] --> B[Encrypt with v0.15 scheme] B --> C[Drive-Signature Entity] C --> D[Store on Arweave] D --> E[Retrieve when needed] E --> F[Decrypt signature] F --> G[Use for drive key derivation] style A fill:#ffecb3 style C fill:#c8e6c9 style G fill:#e1f5fe ``` ## Next Steps Ready to implement privacy in your ArFS applications? - [Creating Private Drives](/build/advanced/arfs/creating-drives) - Learn how to create secure drives - [Upgrading Private Drives](/build/advanced/arfs/upgrading-drives) - Update legacy drives to v0.15 - [Reading Data](/build/advanced/arfs/reading-data) - Query and decrypt your private data # Reading Data (/build/advanced/arfs/reading-data) Clients can perform read operations to create a timeline of entity write transactions which can then be replayed to construct the Drive state. This is done by querying an Arweave GraphQL index for the user's respective transactions. [Arweave GraphQL Guide](https://gql-guide.vercel.app/) can provide more information on how to use Arweave GraphQL. If no GraphQL index is available, drive state can only be generated by downloading and inspecting all transactions made by the user's wallet. This timeline of transactions should be grouped by the block number of each transaction. At every step of the timeline, the client can check if the entity was written by an authorized user. This also conveniently enables the client to surface a trusted entity version history to the user. To determine the owner of a Drive, clients must check for who created the first Drive Entity transaction using that `Drive-Id`. Until a trusted permissions or ACL system is put in place, any transaction in a drive created by any wallet other than the one who created the first Drive Entity transaction could be considered spam. The `Unix-Time` defined on each transaction should be reserved for tie-breaking same entity updates in the same block and should not be trusted as the source of truth for entity write ordering. This is unimportant for single owner drives but is crucial for multi-owner drives with updateable permissions (currently undefined in this spec) as a malicious user could fake the `Unix-Time` to modify the drive timeline for other users. - Drives that have been updated many times can have a long entity timeline which can be a performance bottleneck. To avoid this, clients can cache the drive state locally and sync updates to the file system by only querying for entities in blocks higher than the last time they checked. - Not checking for Drive Ownership could result in seeing incorrect drive state and GraphQL queries. ## Folder/File Paths ArweaveFS does not store folder or file paths along with entities as these paths will need to be updated whenever the parent folder name changes which can require many updates for deeply nested file systems. Instead, folder/file paths are left for the client to generate from the folder/file names. ## Folder View Queries Clients that want to provide users with a quick view of a single folder can simply query for an entity timeline for a particular folder by its id. Clients with multi-owner permissions will additionally have to query for the folder's parent drive entity for permission based filtering of the timeline. ## Basic Query Patterns ### Query All Drive Entities ```graphql query { transactions( tags: [ { name: "ArFS", values: ["0.15"] } { name: "Entity-Type", values: ["drive"] } { name: "Drive-Id", values: ["your-drive-id"] } ] ) { edges { node { id block { height timestamp } tags { name value } } } } } ``` ### Query Folder Contents ```graphql query ($parentFolderId: String!) { transactions( tags: [ { name: "ArFS", values: ["0.15"] } { name: "Parent-Folder-Id", values: [$parentFolderId] } ] ) { edges { node { id block { height timestamp } tags { name value } } } } } ``` ### Query File Entities ```graphql query ($fileId: String!) { transactions( tags: [ { name: "ArFS", values: ["0.15"] } { name: "Entity-Type", values: ["file"] } { name: "File-Id", values: [$fileId] } ] ) { edges { node { id block { height timestamp } tags { name value } } } } } ``` ## Building Drive State The process of building drive state involves several steps: ```mermaid graph TD A[Query Drive Entities] --> B[Sort by Block Height] B --> C[Process Files First] C --> D[Process Folders] D --> E[Process Drive Metadata] E --> F[Build Hierarchy Tree] F --> G[Resolve Conflicts] G --> H[Return Complete State] style A fill:#e3f2fd style H fill:#c8e6c9 ``` ### Step-by-Step Process 1. **Query for all entities** associated with a specific `Drive-Id` 2. **Sort by block height** to establish chronological order 3. **Process entities bottom-up** starting with files and folders 4. **Build the hierarchy** by following parent-child relationships 5. **Handle conflicts** by using the most recent entity version ### Example Implementation ```javascript async function buildDriveState(driveId) { // Query all entities for the drive const entities = await queryDriveEntities(driveId); // Sort by block height entities.sort((a, b) => a.block.height - b.block.height); // Process entities const driveState = { drive: null, folders: new Map(), files: new Map(), }; for (const entity of entities) { const entityType = getTagValue(entity.tags, "Entity-Type"); switch (entityType) { case "drive": driveState.drive = processDriveEntity(entity); break; case "folder": driveState.folders.set( getTagValue(entity.tags, "Folder-Id"), processFolderEntity(entity) ); break; case "file": driveState.files.set( getTagValue(entity.tags, "File-Id"), processFileEntity(entity) ); break; } } return driveState; } ``` ## Using Snapshots For large drives, snapshots can significantly improve performance: ```mermaid sequenceDiagram participant Client participant Gateway participant Arweave Client->>Gateway: Query for latest snapshot Gateway->>Client: Return snapshot data Client->>Client: Process snapshot data Client->>Gateway: Query for newer transactions Gateway->>Client: Return newer entities Client->>Client: Merge with snapshot data Client->>Client: Return complete drive state ``` ### Snapshot Query ```graphql query ($driveId: String!) { transactions( tags: [ { name: "ArFS", values: ["0.15"] } { name: "Entity-Type", values: ["snapshot"] } { name: "Drive-Id", values: [$driveId] } ] sort: HEIGHT_DESC first: 1 ) { edges { node { id block { height timestamp } tags { name value } } } } } ``` ## Performance Optimization ### Caching Strategies - **Local caching** - Store frequently accessed data locally - **Incremental updates** - Only fetch new transactions since last sync - **Snapshot usage** - Use snapshots for large drives - **Batch queries** - Combine multiple queries when possible ### Query Optimization - **Use specific tags** - Narrow down queries with relevant tags - **Limit results** - Use pagination for large result sets - **Filter by date** - Query specific time ranges - **Index utilization** - Write queries that GraphQL indexes can answer ## Error Handling ### Common Issues - **Network timeouts** - Implement retry logic - **Invalid data** - Validate entity structure - **Missing entities** - Handle incomplete data gracefully - **Decryption errors** - Proper error handling for private data ### Best Practices - **Validate ownership** - Check drive ownership before processing - **Handle conflicts** - Resolve entity version conflicts - **Graceful degradation** - Provide fallbacks for missing data - **User feedback** - Inform users of sync status ## Security Considerations ### Data Validation - **Verify signatures** - Check transaction signatures - **Validate ownership** - Ensure drive ownership - **Check timestamps** - Validate entity timestamps - **Sanitize data** - Clean user-provided data ### Privacy Protection - **Decrypt carefully** - Handle private data securely - **Key management** - Protect encryption keys - **Access control** - Implement proper permissions - **Audit logging** - Track data access ## Next Steps Now that you understand how to read ArFS data, explore these related topics: - [Privacy & Encryption](/build/advanced/arfs/privacy) - Secure your data with private drives - [Upgrading Private Drives](/build/advanced/arfs/upgrading-drives) - Update legacy drives to v0.15 - [Creating Drives](/build/advanced/arfs/creating-drives) - Start building with ArFS # Upgrading Private Drives (/build/advanced/arfs/upgrading-drives) ## Overview Private drives rely on a combination of user-set password and a wallet signature for encryption and decryption. [Wander](https://www.wander.app/), formerly ArConnect, is a popular Arweave wallet that is deprecating its `signature()` method in favor of `signDataItem()` or `signMessage()`. In order to preserve access to private drive contents that were secured via drive keys created via 'signature()', ArFS v0.15 introduces a new drive key derivation scheme that both utilizes the modern signing APIs and bridges historical drive keys for usage with it. Because private drive entities exist on chain and their encryption cannot be altered, an upgrade is required to allow continued access to "V1" private drives. This upgrade essentially takes a signature from the drive owner wallet, encrypts it using the required signature structure for V2 private drives, and places it on Arweave as a new "Drive-Signature" entity. This allows the signature to be fetched and decrypted using the latest methods before using it to decrypt the private drive in the V1 format. The below instructions for upgrading a private drive will work during the deprecation period for the `signature()` method from Wanter. Once this period is over, and `signature()` loses all support, additional steps will be required to obtain the correct signature format to decrypt V1 private drives in order to upgrade them. There is, at this time, no set date for when the deprecation period will end. ## The Upgrade Process The upgrade process involves creating a new `Drive-Signature` entity that contains an encrypted version of the legacy signature needed to decrypt the private drive. ```mermaid sequenceDiagram participant User participant Client participant Wallet participant Arweave User->>Client: Initiate drive upgrade Client->>Wallet: Request legacy signature Wallet->>Client: Return signature Client->>Client: Encrypt signature with v0.15 scheme Client->>Arweave: Upload Drive-Signature entity Client->>Client: Update drive with Signature-Type tag Client->>Arweave: Upload updated drive entity Client->>User: Upgrade complete ``` ### Drive-Signature Entity The `Drive-Signature` entity stores the encrypted legacy signature: ```json ArFS: "0.15", Entity-Type: "drive-signature", Signature-Format: "1", Cipher?: "AES256-GCM", Cipher-IV: "" {data: } ``` ### Updated Drive Entity The drive entity is updated with a new `Signature-Type` tag: ```json ArFS: "0.15", Cipher?: "AES256-GCM", Cipher-IV?: "", Content-Type: "", Drive-Id: "", Drive-Privacy: "", Drive-Auth-Mode?: "password", Entity-Type: "drive", Signature-Type?: "1", Unix-Time: "" Metadata JSON { "name": "", "rootFolderId": "", "isHidden": false } ``` ## Using ArDrive The upgrade process has been made simple by using the [ArDrive app](https://app.ardrive.io/). ### Step 1: Log into ArDrive If the connected wallet has V1 private drives that need to be updated, a banner will appear at the top of the screen. ![ArDrive Upgrade Banner](https://turbo-gateway.com/kJzzrYY4KIHLTC9VOECfzvVAjaO1_FOkejWgsRNbLx4) ### Step 2: Click "Update Now!" This will open a modal listing the drives that need to be updated, and linking to more information about the upgrade process. ![ArDrive Upgrade Modal](https://turbo-gateway.com/Qa-qeKkr1flXl1-fdLapjKNmme0lKA43YQbjmMvnF-U) ### Step 3: Click "Update" The process of upgrading the private drives will begin, and involve signing messages depending on how many drives are being upgraded. When the process is complete, a new modal will appear listing the drives that have been successfully updated. ![ArDrive Upgrade Complete](https://turbo-gateway.com/qEwT3oFZbFDpmRQpw9j1c_okN5pIkFqRuoSwBzhc3HQ) ## Manual Upgrade Process If you need to upgrade drives programmatically, here's the process: ### 1. Identify V1 Drives Query for drives that don't have the `Signature-Type` tag: ```graphql query { transactions( tags: [ { name: "ArFS", values: ["0.15"] } { name: "Entity-Type", values: ["drive"] } { name: "Drive-Privacy", values: ["private"] } ] ) { edges { node { id tags { name value } } } } } ``` ### 2. Create Drive-Signature Entity ```javascript async function createDriveSignature(driveId, legacySignature) { // Encrypt the legacy signature const encryptedSignature = await encryptSignature(legacySignature); // Create the drive signature entity const driveSignature = { data: encryptedSignature, tags: [ { name: "ArFS", value: "0.15" }, { name: "Entity-Type", value: "drive-signature" }, { name: "Signature-Format", value: "1" }, { name: "Cipher", value: "AES256-GCM" }, { name: "Cipher-IV", value: cipherIV }, ], }; // Upload to Arweave return await uploadTransaction(driveSignature); } ``` ### 3. Update Drive Entity ```javascript async function updateDriveEntity(driveId) { // Get existing drive entity const driveEntity = await getDriveEntity(driveId); // Add Signature-Type tag const updatedTags = [ ...driveEntity.tags, { name: "Signature-Type", value: "1" }, ]; // Create updated drive entity const updatedDrive = { data: driveEntity.data, tags: updatedTags, }; // Upload to Arweave return await uploadTransaction(updatedDrive); } ``` ## Verification After upgrading, verify the process was successful: ### Check Drive-Signature Entity ```graphql query ($driveId: String!) { transactions( tags: [ { name: "ArFS", values: ["0.15"] } { name: "Entity-Type", values: ["drive-signature"] } { name: "Drive-Id", values: [$driveId] } ] ) { edges { node { id block { height timestamp } tags { name value } } } } } ``` ### Check Updated Drive Entity ```graphql query ($driveId: String!) { transactions( tags: [ { name: "ArFS", values: ["0.15"] } { name: "Entity-Type", values: ["drive"] } { name: "Drive-Id", values: [$driveId] } { name: "Signature-Type", values: ["1"] } ] ) { edges { node { id block { height timestamp } tags { name value } } } } } ``` ## Troubleshooting ### Common Issues - **Signature not found** - Ensure the wallet supports the required signing methods - **Encryption errors** - Verify the encryption parameters are correct - **Upload failures** - Check network connectivity and retry - **Permission denied** - Ensure you own the drive being upgraded ### Error Handling ```javascript async function upgradeDrive(driveId) { try { // Get legacy signature const legacySignature = await getLegacySignature(driveId); // Create drive signature entity await createDriveSignature(driveId, legacySignature); // Update drive entity await updateDriveEntity(driveId); console.log("Drive upgraded successfully"); } catch (error) { console.error("Upgrade failed:", error); // Handle error appropriately } } ``` ## Best Practices ### Before Upgrading - **Backup your data** - Ensure you have access to your drive contents - **Test with one drive** - Start with a single drive to verify the process - **Check wallet compatibility** - Ensure your wallet supports required methods - **Verify ownership** - Confirm you own the drives being upgraded ### During Upgrading - **Monitor progress** - Keep track of upgrade status - **Handle errors gracefully** - Implement proper error handling - **Batch operations** - Upgrade multiple drives efficiently - **User feedback** - Provide clear status updates ### After Upgrading - **Verify functionality** - Test drive access and operations - **Update clients** - Ensure all clients support v0.15 - **Monitor performance** - Check for any performance issues - **Document changes** - Keep track of upgraded drives ## Migration Timeline ```mermaid gantt title ArFS v0.15 Migration Timeline dateFormat YYYY-MM-DD section Phase 1 Legacy Support :active, legacy, 2024-01-01, 2024-06-30 section Phase 2 Migration Period :migration, 2024-07-01, 2024-12-31 section Phase 3 Legacy Deprecation :deprecation, 2025-01-01, 2025-06-30 section Phase 4 Full v0.15 :full, 2025-07-01, 2025-12-31 ``` ## Next Steps After upgrading your drives, explore these related topics: - [Privacy & Encryption](/build/advanced/arfs/privacy) - Understand the new encryption scheme - [Reading Data](/build/advanced/arfs/reading-data) - Query your upgraded drives - [Creating Drives](/build/advanced/arfs/creating-drives) - Create new v0.15 drives # Downloading Indexes with BitTorrent (/build/advanced/bittorrent-indexes) Gateways that publish [shared indexes](/learn/gateways/index-sharing) offer each band as a torrent, so you don't need a gateway to take a copy. Any BitTorrent client can download one, and peers don't charge for what they send. This guide downloads a band from turbo-gateway.com with qBittorrent and with aria2 on the command line, then checks it against the publisher's signature. To read single IDs rather than whole bands, see [Reading Shared Indexes](/build/advanced/index-publications). The torrent only delivers the bytes. What makes them trustworthy is the signed publication. Always finish with the [check](#check-the-download). ## What You Need - **Disk.** About 21 GB for the five root-TX bands turbo-gateway.com publishes, or a few hundred MB to try the tip band. aria2 needs up to 64 MB more per band for pad files while it downloads (see [below](#download-with-aria2)). - **Network.** Nothing to open to download: your client connects out to the peers. To seed well, forward or open your client's listening port, so that peers can connect to you as well. Some networks and hosting providers block or restrict BitTorrent traffic. ## Find the Torrents A publisher lists its bands in one signed JSON document at `/ar-io/indexes`. Each band that is seeded has a `torrent` entry, and its `magnet` link is part of what the publisher signed: ```bash curl -s https://turbo-gateway.com/ar-io/indexes \ | jq -r '.indexes[] | .name as $index | .bands[] | "\($index) \(.id)\n\(.torrent.magnet)\n"' ``` ```text root-tx-index b0-h2010500-tip-20261010 magnet:?xt=urn:btih:2c35385ba96632bcfe39566d8f336e8c9b462f18&xt=urn:btmh:12201710ca02d31b9b4c7305c4506a67fdaaec634cb7200c2d6cce70725ed13686b1&dn=bfa9f0269c7b03f8&tr=http%3A%2F%2F167.235.37.218%3A6969%2Fannounce root-tx-index b1-h1950000-tip-20260929 magnet:?xt=urn:btih:57f33f3cdda02de27eb42f84e62b3b67dccc758c&xt=urn:btmh:122027a8171c4e95fe7eeb86c4a1efe9c2190861f7f7e183cdc877549f87a2abc07a&dn=0a27456b192621a7&tr=http%3A%2F%2F167.235.37.218%3A6969%2Fannounce ... ``` turbo-gateway.com publishes two indexes. This guide downloads the **root-TX bands** (`root-tx-index`): five bands, about 21 GB in all. The band id shows the block heights each covers: `b0-h2010500-tip` runs from height 2,010,500 to the chain tip, and `b4-h0-1349999` covers the start of the chain. The tip band is the smallest and is rebuilt often. The older bands are 2 to 8 GB and rarely change. Start with `b0` to try it out. The list also shows the 24 **L1 bands** (`parquet-l1`), about 15 GB, which come with gateway Release 85, not released yet. They download the same way. The `.cdb` partitions described below belong to root-TX bands only; an L1 band holds Parquet files and a `band.json`. See [Query a Parquet Dataset](/build/advanced/index-publications#query-a-parquet-dataset). The magnets on this page were current when it was written. Bands are rebuilt and replaced, so take yours from the publication. Prefer the `magnet` link. A `.torrent` file, served at the band's `torrentUrl`, is not signed. If you use one, check that the client shows the same info hash as the band's signed `infohashV1`. ## Download with qBittorrent ### Add the Magnet Link In qBittorrent, choose **File → Add Torrent Link**, paste the band's magnet link, and click **Download**. Pick a save folder when asked. The client first fetches the torrent's metadata from peers and checks it against the info hash in the link. Then it downloads the files. ### Know What You Are Seeing - **Name.** The torrent is named with 16 hex characters, such as `bfa9f0269c7b03f8` for `b0`, not with the band id. The name is derived from the band's files, so two gateways publishing the same bytes share one swarm. It is also the name of the folder the files are saved in. - **Info hash.** The **General** tab shows *Info Hash v1* and *Info Hash v2*. These should match the band's `infohashV1` and `infohashV2`. The torrents are hybrid v1 and v2, so any current client can join. - **Size.** qBittorrent can show up to 64 MB more than the band: BEP 47 pad files, which line each file up with a piece boundary and are never written. The folder holds only the band's real files: for a root-TX band, `manifest.json` and 256 partitions, `00.cdb` to `ff.cdb`. - **Trackers.** The **Trackers** tab lists the publisher's tracker as *Working*. It only answers for the publisher's own bands. Peers are also found through the DHT and peer exchange. ### Let It Finish Speed depends on how many gateways are seeding a band when you download it. The tip band takes seconds; a 2 GB band takes minutes. ## Download with aria2 [aria2](https://aria2.github.io/) downloads a magnet link from the command line and stops once the download finishes: ```bash aria2c --seed-time=0 --bt-save-metadata=false \ 'magnet:?xt=urn:btih:2c35385ba96632bcfe39566d8f336e8c9b462f18&xt=urn:btmh:12201710ca02d31b9b4c7305c4506a67fdaaec634cb7200c2d6cce70725ed13686b1&dn=bfa9f0269c7b03f8&tr=http%3A%2F%2F167.235.37.218%3A6969%2Fannounce' ``` ```text Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= ece690|OK | 0B/s|[MEMORY][METADATA]bfa9f0269c7b03f8 710e97|OK | 12MiB/s|/data/bfa9f0269c7b03f8/00.cdb (513more) ``` Unlike qBittorrent, aria2 (1.37) writes the pad files, up to 64 MB of zeros, into a hidden `.pad/` folder inside the download. Delete it once the download is done: ```bash rm -r bfa9f0269c7b03f8/.pad ``` To seed after downloading, which helps the next person, drop `--seed-time=0`. ## Check the Download The client checked every piece against the info hash. That proves you got the torrent the link names. It doesn't prove the files are the publisher's index. For that, check the publication's signature against the key the gateway registry holds for the publisher, then check each file's size and SHA-256 against the publication. ### Verify the Publication Save the publication, then check its signature and sequence as [Verify It](/build/advanced/index-publications#verify-it) shows, on this saved copy: ```bash curl -s https://turbo-gateway.com/ar-io/indexes > publication.json ``` ### Check the Band's Files This script finds the band by its torrent name, the folder your client downloaded into. Then it checks each file's size and SHA-256 against the publication you verified. It needs Node.js 20 or later and nothing else: ```javascript title="verify-band.mjs" // Check a band downloaded over BitTorrent against a verified publication. // Usage: node verify-band.mjs const [, , publicationFile, bandDir] = process.argv; const doc = JSON.parse(await readFile(publicationFile, "utf8")); async function sha256File(file) { const hash = crypto.createHash("sha256"); for await (const chunk of createReadStream(file)) hash.update(chunk); return hash.digest("hex"); } // The band whose files are in bandDir: matched by torrent name, which is // the directory name a torrent client downloads into. const bands = doc.indexes.flatMap((index) => index.bands); const torrentName = (b) => new URLSearchParams(b.torrent?.magnet.split("?")[1]).get("dn"); const band = bands.find((b) => torrentName(b) === path.basename(bandDir)); if (band === undefined) throw new Error(`no band in the publication for ${path.basename(bandDir)}`); console.log(`band ${band.id}, infohash v1 ${band.torrent.infohashV1}`); // Every file: size and SHA-256 as signed. let bad = 0; for (const file of band.files) { const p = path.join(bandDir, file.name); const size = await stat(p).then((s) => s.size, () => undefined); if (size !== file.size || (await sha256File(p)) !== file.sha256) { bad += 1; console.log(`MISMATCH ${file.name}`); } } console.log(bad === 0 ? `all ${band.files.length} files match: ok` : `${bad} files do not match`); process.exit(bad === 0 ? 0 : 1); ``` ```bash node verify-band.mjs publication.json ./bfa9f0269c7b03f8 ``` ```text band b0-h2010500-tip-20261010, infohash v1 2c35385ba96632bcfe39566d8f336e8c9b462f18 all 257 files match: ok ``` A changed or missing file is reported and the script exits with status 1: ```text MISMATCH 00.cdb MISMATCH manifest.json 2 files do not match ``` The check is against the publication as it is now, so check a band soon after you download it. Once the publisher replaces a band, it drops out of the publication. ## What to Do with It - **Keep seeding.** Leave the torrent running in your client. Every seeder makes the next download faster, for gateways as well as for people. - **Look up IDs.** A root-TX band is a partitioned CDB64 index from data item ID to root transaction. See [Look Up One ID](/build/advanced/index-publications#look-up-one-id) to read it. - **Let a gateway do it.** A gateway can subscribe to a publisher, download and check every band by itself, keep them current, and seed them: ```bash ./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --torrent --restart ``` See [Index Sharing](/build/run-a-gateway/manage/index-sharing#subscribe-to-a-publisher) for the full setup. # EthAReum Protocol (/build/advanced/ethareum) The **EthAReum protocol** enables the generation of private keys for an Arweave wallet using a signature from an Ethereum or Solana wallet. This allows users to create an Arweave wallet directly through popular wallet providers like MetaMask, so one wallet extension manages keys on both chains. Generated private keys provide a fully functional Arweave wallet, equipped to perform all standard operations, including holding AR tokens and Turbo Credits, and uploading data to the Arweave network. ## How It Works EthAReum uses a deterministic key derivation process that combines: - **Ethereum/Solana wallet signature** - Provides the cryptographic foundation - **User-generated password** - Adds additional entropy and security - **Standardized derivation algorithm** - Ensures reproducible results The protocol generates a unique Arweave wallet that is cryptographically linked to your Ethereum or Solana wallet but remains completely independent. ## Browser Compatibility **Recommended Browser**: For optimal performance, use **Chrome** when working with EthAReum and MetaMask. While EthAReum functions correctly in most browsers, there are ongoing efforts to resolve some edge case compatibility issues in other environments. ## Password Security The EthAReum protocol incorporates a user-generated password in the wallet derivation process. This password provides an extra layer of security by contributing additional entropy to the wallet's derivation and serves as a critical verification step for wallet access. **Permanent Password**: The password used during the derivation of private keys is **permanent and cannot be changed or recovered** by any administrator. ArDrive is a decentralized platform with no account administration. It is crucial to keep this password secure. ### Password Requirements - Must be set during initial wallet creation - Used for all subsequent logins - Required for encrypting private uploads - Cannot be recovered if forgotten ## Wallet Addresses The public address of the generated Arweave wallet is derived from its public key and will be **different** from the public address of the Ethereum or Solana wallet used to generate it. ### Viewing Your Address The exact steps to obtain your generated wallet's public address depend on the dApp interface: - **ArDrive**: Click the user profile icon in the top right when logged in - **Other dApps**: Check the wallet settings or profile section ## Key Management ### Keyfiles vs Seed Phrases The Arweave ecosystem primarily uses **keyfiles** rather than seed phrases for wallet access: - **Keyfile**: JSON file containing a Json Web Key (JWK) that acts as private keys - **Seed Phrase**: Supported but not universally implemented across all dApps ### Accessing Your Keys Both keyfile and seed phrase are available for download in most dApps: - **ArDrive**: Click the user profile icon in the top right when logged in - **Other dApps**: Check wallet settings or ## Security Considerations ### One-Way Control EthAReum generates Arweave wallet private keys using a signature from your Ethereum/Solana wallet, ensuring that control only extends in one direction: - ✅ **EthAReum can generate** Arweave wallets from Ethereum/Solana signatures - ❌ **EthAReum cannot access** your Ethereum/Solana wallet or assets - ✅ **Your Ethereum/Solana assets remain** completely secure and independent ### Signature Security **Beware of Malicious dApps**: Some malicious dApps or websites may disguise high-risk authorization transactions as simple signature requests. Always ensure that you only provide signatures to reputable and trusted dApps like ArDrive. ### Best Practices 1. **Verify dApp authenticity** before providing signatures 2. **Use strong, unique passwords** for wallet derivation 3. **Backup your keyfile** in a secure location 4. **Never share your password** or keyfile with anyone 5. **Test with small amounts** before committing to large transactions ## Implementation Examples ### Basic Wallet Generation ```javascript // Example: Generate Arweave wallet from Ethereum signature async function generateArweaveWallet(ethereumSignature, password) { // This is a conceptual example - actual implementation // would use the EthAReum protocol specification const derivedKey = await deriveKeyFromSignature( ethereumSignature, password, "arweave" // derivation context ); return { address: getAddressFromKey(derivedKey), keyfile: createKeyfile(derivedKey), seedPhrase: generateSeedPhrase(derivedKey), }; } ``` ### Integration with MetaMask ```javascript // Example: Request signature from MetaMask async function requestEthereumSignature() { const accounts = await ethereum.request({ method: "eth_requestAccounts", }); const message = "Sign this message to generate your Arweave wallet"; const signature = await ethereum.request({ method: "personal_sign", params: [message, accounts[0]], }); return signature; } ``` ## Use Cases ### Cross-Chain dApp Development - **Unified wallet experience** across Ethereum and Arweave - **Simplified onboarding** for users familiar with Ethereum - **Reduced friction** in multi-chain applications ### Data Storage Solutions - **Decentralized file storage** using existing Ethereum wallets - **NFT metadata storage** on Arweave with Ethereum wallet access - **Cross-chain data management** for DeFi applications ### Developer Benefits - **Familiar wallet interfaces** for users - **Reduced development complexity** for multi-chain apps - **Enhanced user experience** with single wallet management ## Next Steps Learn about structured data storage on Arweave using your generated wallet. Upload data efficiently using Turbo Credits with your EthAReum wallet. Learn how to find and access data stored with your generated wallet. # Reading Shared Indexes (/build/advanced/index-publications) Gateways that take part in [Index Sharing](/learn/gateways/index-sharing) publish signed indexes that locate data items inside their bundles. Gateways use them automatically, but nothing about them is gateway-only: any client can fetch a publication, verify it, and look up an ID itself. This page shows how. To have your own gateway subscribe or publish, see [Index Sharing](/build/run-a-gateway/manage/index-sharing). ## The Publication A publishing gateway serves one JSON publication at `/ar-io/indexes`: ```json { "version": 1, "publisher": "34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5", "sequence": 12, "issuedAt": "2026-10-01T12:00:00.000Z", "expiresAt": "2026-10-02T12:00:00.000Z", "indexes": [ { "name": "root-tx-index", "kind": "cdb64-root-tx", "bands": [ { "id": "b1-h1950000-tip-20260917", "heightRange": [1950000, null], "files": [ { "name": "manifest.json", "size": 41233, "sha256": "…" }, { "name": "00.cdb", "size": 7012345, "sha256": "…" } ] } ] } ], "signature": { "alg": "ed25519", "keyId": "34LYv…", "sig": "…" } } ``` `publisher` is the gateway's wallet, `signature.keyId` its observer address, and `signature.sig` a base64 Ed25519 signature. The publication may carry fields not shown here, and later versions may add more. Keep them: the signature covers them. ## Verify It ### Get the Publisher's Registered Observer Address Look up the publisher's wallet in the gateway registry: with the [ar.io SDK](/sdks/ar-io-sdk), or from any gateway's `/ar-io/peers`. The publication must be signed by that gateway's `observerAddress`. A signature from any other key proves only that somebody signed something. ### Check the Signature The signed message is the prefix `ar-io-index-publication/v1` and a newline, followed by the [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785) canonical JSON of the publication with `signature` removed. This works in Node.js 20+ and current browsers, with one dependency (`npm install json-canonicalize`): ```javascript const B58 = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"; function base58Decode(s) { let n = 0n; for (const c of s) n = n * 58n + BigInt(B58.indexOf(c)); const bytes = []; while (n > 0n) { bytes.unshift(Number(n % 256n)); n /= 256n; } for (const c of s) { if (c !== "1") break; bytes.unshift(0); } return new Uint8Array(bytes); } async function verifyPublication(doc, observerAddress) { const { signature, ...unsigned } = doc; if (signature?.alg !== "ed25519") throw new Error("unknown algorithm"); if (signature.keyId !== observerAddress) throw new Error("not the registered key"); const key = await crypto.subtle.importKey( "raw", base58Decode(signature.keyId), { name: "Ed25519" }, false, ["verify"], ); const message = new TextEncoder().encode( "ar-io-index-publication/v1\n" + canonicalize(unsigned), ); const sig = Uint8Array.from(atob(signature.sig), (c) => c.charCodeAt(0)); return crypto.subtle.verify("Ed25519", key, sig, message); } const doc = await fetch("https://turbo-gateway.com/ar-io/indexes").then((r) => r.json()); console.log(await verifyPublication(doc, "")); ``` Use a real RFC 8785 library. A key-sorted `JSON.stringify` gives different bytes for some numbers and keys, and the signature will not verify. ### Check It Is Current Remember the highest `sequence` you have accepted from each publisher, and refuse a lower one: a cache or mirror could serve an older publication. If `expiresAt` has passed, the publisher has stopped signing; its bands are still valid, but nothing newer is coming. ## Fetch a Band File Fetch files by their SHA-256. The address can't change meaning, so it is safe to cache and can come from any server that has the file: ```javascript async function fetchFile(gateway, file) { const res = await fetch(`${gateway}/ar-io/indexes/blob/${file.sha256}`); if (!res.ok) throw new Error(`HTTP ${res.status}`); const bytes = new Uint8Array(await res.arrayBuffer()); const digest = await crypto.subtle.digest("SHA-256", bytes); const hex = [...new Uint8Array(digest)].map((b) => b.toString(16).padStart(2, "0")).join(""); if (bytes.length !== file.size || hex !== file.sha256) throw new Error("file does not match"); return bytes; } ``` Range requests are supported, which is how a large file resumes. Peers seeding a band over BitTorrent are not metered; see [Bands as torrents](#bands-as-torrents). A `503` with `Retry-After` means the publisher is part way through replacing a band: fetch the publication again after the delay. ## Bands as Torrents A publisher that runs a torrent engine adds an optional `torrent` entry to each band it seeds: ```json "torrent": { "infohashV1": "", "infohashV2": "", "magnet": "magnet:?xt=urn:btih:…", "torrentUrl": "/ar-io/indexes/torrents/.torrent" } ``` | Field | Meaning | | --- | --- | | `infohashV1` | The v1 infohash, 40 hex characters | | `infohashV2` | The v2 infohash, 64 hex characters. Torrents are hybrid v1 + v2 | | `magnet` | A magnet link for the torrent | | `torrentUrl` | Where the `.torrent` file is served. On an ar.io gateway it is addressed by the v1 infohash, so a band rebuilt under the same id gets a new URL | The entry is absent when the publisher runs no engine; the HTTP routes always work. The torrent route can answer `404` until the publisher has built a band's torrent, which is normal. **The torrent name** is derived from content, not the band id: the first 16 hex characters of the SHA-256 over one line per file, `\0\0\n`, with files in bytewise name order. So publishers of the same bytes share one infohash and one swarm. It is also the `` in the WebSeed route, `/ar-io/indexes/webseed//`, which serves band files to torrent clients and is metered like the blob route. **Only the info dictionary is signed.** The infohashes cover the torrent's info dictionary (file names, sizes and piece hashes) and nothing else. Trackers and WebSeeds in a `.torrent` file are outside it and unsigned. The gateway's own subscriber checks a `.torrent` against the signed infohashes. It checks that the file list is exactly the band's signed files and sizes (plus BEP 47 pad files), drops every WebSeed it names, and hashes every downloaded file against its signed SHA-256, as over HTTP. A client of its own should do the same. ### Pull a Whole Index with Any BitTorrent Client To mirror a publisher's full index, for a pipeline or an agent, use the `magnet` link or the `.torrent` file with any BitTorrent client. Peers aren't metered, so this is the cheapest way to take everything. Before using a `.torrent`, check its infohash matches the signed `infohashV1`, since the file itself is unsigned. After the download, check every file's size and SHA-256 against the signed publication, exactly as for an HTTP fetch: the infohash protects the pieces, but only the publication says these are the right files. Seed it afterwards if you can; that is what keeps the swarm fast. [Downloading Indexes with BitTorrent](/build/advanced/bittorrent-indexes) walks through it with qBittorrent and aria2. ## Look Up One ID A `cdb64-root-tx` band is a [partitioned CDB64 index](/build/run-a-gateway/manage/cdb64#partitioned-indexes). To find the root transaction of one data item: 1. Fetch and check the band's `manifest.json`. 2. Pick the partition for the ID's first byte: the manifest lists partitions by two-character hex prefix. 3. Fetch and check that one partition, up to about 30 MB. 4. Look the 32-byte ID up in it. The value is MessagePack, holding the root transaction ID and, when known, the item's byte offsets. Bands may overlap. A publisher should never have two bands disagree about an item, since an item has one location, so search them newest first and stop at the first match. The [reference client](https://github.com/ar-io/ar-io-node/blob/develop/docs/index-publication.md#reference-client) does all of this in about 130 lines of Python, with the CDB64 reader and MessagePack decoder written out. ## Query a Parquet Dataset **Release Requirement**: `parquet-l1` bands come with gateway Release 85, which is not released yet. Some gateways, such as turbo-gateway.com, already publish them from pre-release builds. A `cdb64-root-tx` band answers one question quickly: where is this ID. A `parquet-l1` band answers a different kind: how many, how large, which, over the whole chain. It holds Arweave's base layer as [Apache Parquet](https://parquet.apache.org/), partitioned into bands by block height, so any engine that reads Parquet can query it. No band files to download, no import into a database, and no API to learn. The examples use DuckDB, which installs its own `httpfs` and `json` extensions on first use. ### See What a Gateway Publishes Not every gateway publishes every dataset. `parquet-l1` is opt-in, so check first: ```bash curl -s https://turbo-gateway.com/ar-io/indexes \ | jq '.indexes[] | {name, kind, bands: (.bands | length)}' ``` ```json { "name": "root-tx-index", "kind": "cdb64-root-tx", "bands": 5 } { "name": "parquet-l1", "kind": "parquet-l1", "bands": 24 } ``` `name` is the dataset you want; `kind` tells you how to read it. They are not the same thing, and for `root-tx-index` they differ. ### The Tables Each band directory holds one Parquet file per table, plus a `band.json` with the band's heights, row counts and a schema version: | File | One row per | Key columns | | --- | --- | --- | | `blocks.parquet` | block | `height`, `indep_hash`, `previous_block`, `block_size`, `weave_size`, `tx_root`, `hash_list_merkle` | | `transactions.parquet` | transaction | `id`, `height`, `data_size`, `data_root`, `owner_address`, `target`, `quantity`, `reward`, `content_type`, `format`, `offset` | | `tags.parquet` | tag | `id`, `height`, `tag_index`, `tag_name`, `tag_value` | | `block_transactions.parquet` | transaction in a block | `height`, `id`, `block_transaction_index` | | `wallets.parquet` | wallet | `address`, `public_modulus` | IDs, addresses, hashes and tag names and values are stored as raw bytes (`BLOB`), not text. Base64url is a presentation format, so convert when you want to read one. ### Lookup Files **Release Requirement**: lookup files (layout `l1-3`) come with Release 85, which is not released yet. From layout `l1-3` (the `schema` in a band's publication `metadata`, and in its `band.json`), a band also holds three lookup files. Parquet has no index, so finding one transaction otherwise means scanning every band's `id` column. A lookup file is derived from the band's tables and sorted by a key, in row groups of 16,384 rows. Parquet keeps each row group's min and max, so a reader holding a key reads the footer, then the one or two row groups that can hold it. | File | One row per | Columns | | --- | --- | --- | | `lookup_tx_id.parquet` | transaction | `id8`, `height` | | `lookup_wallet.parquet` | transaction an address signed (`role` 0) or received (`role` 1) | `addr8`, `role`, `height`, `data_size` | | `lookup_tag.parquet` | distinct tag name and value | `name8`, `val8`, `name`, `value`, `txs`, `first_height`, `last_height` | The keys are unsigned 64-bit integers made one of two ways, which any client can reproduce: - **`prefix64`**: the first 8 bytes, big-endian, zero-padded on the right if shorter. For IDs and addresses. In DuckDB: `('0x' || rpad(left(hex(x), 16), 16, '0'))::UBIGINT`. - **`sha256_64`**: `prefix64` of the value's SHA-256. For tag names and values. In DuckDB: `('0x' || left(sha256(x), 16))::UBIGINT`. | Encoding | Input | Output | | --- | --- | --- | | `prefix64` | the 32 bytes of ID `O048e9pT5nX1CPrMjGC1y1dWdtd3AChFX27hoRsVIdA` | `0x3b4e3c7bda53e675` | | `prefix64` | the single byte `0xab` | `0xab00000000000000` | | `sha256_64` | `App-Name` | `0xbf6cc2a967f23a82` | | `sha256_64` | `ArDrive-App` | `0xa2c30101e8045f65` | A key is a pointer: two values can share one, so always finish in the table the key points into, as the examples below do. `data_size` and `txs` are carried so a wallet's bytes stored and a tag's count need no table read. A lookup file is listed and signed in the publication like any band file. In a gateway checkout, `./tools/ar-io-node index-l1-verify --bands-dir` (Release 85) checks that it holds exactly what its band's tables give. `parquet-l1` is the **base layer only**: Arweave transactions, not the data items bundled inside them. For items inside bundles, use [`root-tx-index`](#look-up-one-id) or the gateway's GraphQL. ### Query It in Place, Downloading Nothing The byte routes serve HTTP range requests, and Parquet is designed to be read that way: the reader fetches the footer, decides which row groups it needs, and fetches only those. So a query against a dataset on a gateway moves a small fraction of it. Only a publisher serves the byte routes. A gateway that subscribes to a dataset answers `404` for them, so point queries at the publisher named in the publication. This is a complete script. [DuckDB](https://duckdb.org/) is the only dependency, and it reads the publication itself to find the bands: ```sql INSTALL httpfs; LOAD httpfs; INSTALL json; LOAD json; SET VARIABLE gateway = 'https://turbo-gateway.com'; SET VARIABLE txs = ( SELECT list(getvariable('gateway') || '/ar-io/indexes/parquet-l1/' || b.id || '/transactions.parquet') FROM (SELECT unnest(ix.bands) AS b FROM (SELECT unnest(indexes) AS ix FROM read_json(getvariable('gateway') || '/ar-io/indexes')) WHERE ix.name = 'parquet-l1') ); SELECT count(*) AS transactions FROM read_parquet(getvariable('txs')) WHERE height BETWEEN 1500000 AND 1500099; ``` Swap the last statement for whatever you want to ask. The largest transactions ever posted to the base layer, with their IDs converted back to base64url: ```sql SELECT rtrim(replace(replace(to_base64(id), '+', '-'), '/', '_'), '=') AS tx_id, height, round(data_size / power(2, 30), 1) AS gib FROM read_parquet(getvariable('txs')) ORDER BY data_size DESC LIMIT 3; ``` ``` ┌─────────────────────────────────────────────┬─────────┬───────┐ │ tx_id │ height │ gib │ ├─────────────────────────────────────────────┼─────────┼───────┤ │ oWRzBr3KHhULAL-s5ULeXac1mb_WQOX5uFBRea16iRI │ 1992471 │ 621.9 │ │ 7vg2832WFsisEcBr1oBQ8ldc4EGOkjQdwW46hDvJsOs │ 1989512 │ 159.6 │ │ SFUs16XzaWFhWvWOR55hb-5X081r2LeWWAHq4HWUt00 │ 1372669 │ 33.7 │ └─────────────────────────────────────────────┴─────────┴───────┘ ``` Tags and content types work the same way. Both columns are bytes, so cast them: ```sql SELECT CAST(tag_name AS VARCHAR) AS tag, count(*) AS n FROM read_parquet(getvariable('gateway') || '/ar-io/indexes/parquet-l1//tags.parquet') GROUP BY 1 ORDER BY n DESC LIMIT 5; ``` ### Find One Transaction **Release Requirement**: this example needs `l1-3` bands, which come with Release 85, not released yet. Point lookups are what a scan is bad at: finding one ID means reading every band's `id` column. The [lookup files](#lookup-files) turn it into two small reads, first the height from `lookup_tx_id`, then the rows at that height. This is a complete script: ```sql INSTALL httpfs; LOAD httpfs; INSTALL json; LOAD json; SET VARIABLE gateway = 'https://vilenarios.com'; SET VARIABLE tx = 'dwXPQC8J1u2M8nr5uP-rDgi5JVvpQ5MrjMNRjleUyzk'; -- Every band's file prefix and layout, from the publication. CREATE TEMP TABLE bands AS SELECT getvariable('gateway') || '/ar-io/indexes/parquet-l1/' || b.id || '/' AS dir, b.metadata.schema AS schema FROM (SELECT unnest(ix.bands) AS b FROM (SELECT unnest(indexes) AS ix FROM read_json(getvariable('gateway') || '/ar-io/indexes')) WHERE ix.name = 'parquet-l1'); SET VARIABLE lookups = (SELECT list(dir || 'lookup_tx_id.parquet') FROM bands WHERE schema = 'l1-3'); SET VARIABLE txs = (SELECT list(dir || 'transactions.parquet') FROM bands); -- The ID as bytes, and its key: the first 8 bytes as an unsigned integer. SET VARIABLE id = from_base64(rpad(replace(replace(getvariable('tx'), '-', '+'), '_', '/'), 44, '=')); SET VARIABLE id8 = ('0x' || left(hex(getvariable('id')), 16))::UBIGINT; -- 1. The lookup files give the height (a key can match more than one). SET VARIABLE heights = ( SELECT list(DISTINCT height) FROM read_parquet(getvariable('lookups')) WHERE id8 = getvariable('id8') ); -- 2. The transaction, from the rows at those heights. SELECT height, data_size, CAST(content_type AS VARCHAR) AS content_type FROM read_parquet(getvariable('txs')) WHERE height BETWEEN list_min(getvariable('heights')) AND list_max(getvariable('heights')) AND id = getvariable('id'); ``` ``` ┌─────────┬───────────┬──────────────┐ │ height │ data_size │ content_type │ ├─────────┼───────────┼──────────────┤ │ 1150000 │ 36739084 │ NULL │ └─────────┴───────────┴──────────────┘ ``` Measured against a gateway publishing the whole chain at `l1-3` (24 bands, heights 0 to 2,017,885): | Finding one transaction | Bytes fetched | Time | | --- | --- | --- | | With the lookup files (the script above) | 12.9 MB | 0.7 s | | Scanning `id` across every band | 2.6 GB | 26 s | The last step matches the full ID, not the key, so two IDs that share a key can't be confused. The `height` filter is what keeps it cheap: each band's footer rules out every row group but the one or two holding that height. A band published before layout `l1-3` has no lookup files, so a transaction in it is found only by a scan, as in the examples above. ### What Actually Crosses the Network The cost depends on the query, not on the size of the dataset. A `count(*)` over the whole dataset reads about 4 MB: one footer per band, then one row group. An aggregate over a whole column, such as `sum(data_size)`, reads that column. Filtering on `height` is the cheapest filter, since bands are partitioned by it and row groups are ordered within a band. This depends on range requests working end to end. A reverse proxy with a cache zone in front of a gateway can answer a range request with the whole file instead, which turns a 4 MB query into hundreds of megabytes. It happens when the gateway meters its byte routes, because the responses are then `private`, nothing can be cached, and the proxy has no stored object to answer a range from. If a gateway behaves that way, its operator needs [the uncached location block](/build/run-a-gateway/manage/nginx-caching#index-sharing). Check with `curl -s -o /dev/null -w '%{http_code} %{size_download}' -r 0-99 `, which should print `206 100`. ### Verify Before You Trust It Querying over HTTP means trusting the server for the bytes it streams you. If the answer matters, [verify the publication](#verify-it) first, then fetch the band files you need [by digest](#fetch-a-band-file) and query them locally. Two things make this cheaper than it sounds: - Every band file has a signed SHA-256 in the publication, so a local copy is checkable. - `Repr-Digest` is co-signable, so even a `206` carries a signature over the digest of the whole file it came from. A client can confirm which file it is reading bytes out of without fetching all of it. A `parquet-l1` dataset is also checkable against Arweave itself, independently of the publisher: block linkage, `hash_list_merkle`, `tx_root` and weave accounting can all be recomputed from the rows. In a gateway checkout, `./tools/ar-io-node index-l1-verify --bands-dir` (Release 85) does that over a local copy. ### Metering The publication and `.torrent` files are free. The byte routes are metered like data, so an analytical client is a paying or allowlisted client, with a free allowance first. An accidental full-file download can spend that allowance in one request and the next range request returns `402`; that is metering, not a fault. For sustained or heavy work, take the whole dataset [over BitTorrent](#pull-a-whole-index-with-any-bittorrent-client), which is not metered, and query it locally: ```sql SELECT count(*) FROM read_parquet('parquet-l1/*/transactions.parquet') WHERE height BETWEEN 1000000 AND 1000100; ``` ### This or GraphQL? Both read a gateway's index, and they are good at different things. | Use | When | | --- | --- | | **GraphQL** | Anything that includes data items inside bundles; anything near the chain tip; filtering on several fields at once; a browser or a small client with no query engine | | **`parquet-l1`** | Counting, summing, grouping or joining over the whole base layer; bulk GraphQL cannot aggregate: there is no `count` or `sum`, and a large result has to be paged. On an `l1-3` band, a point lookup is two small reads through the [lookup files](#lookup-files); on a band of an older layout, it is a scan of every band's `id` column. A published band set also lags the tip by its publishing cadence. # Advanced (/build/advanced) ## Overview Explore advanced topics and specialized guides for building on Arweave and ar.io. These resources are designed for developers and operators who need deeper technical knowledge and advanced configuration options. ## Advanced Topics **Advanced ArFS documentation** for structured data storage **Verify a gateway's signed index publication** and look up a data item's root transaction from any client **Understanding wallet address normalization** across different networks **Security mechanisms** in ar.io gateways **Key topics:** - Same-origin policy **Generate Arweave wallets** from Ethereum or Solana wallets ## Ready to Go Advanced? **New to Arweave?** Start with our [Getting Started guide](/build) to understand the basics. **Building dApps?** Check out [ArFS Protocol](/build/advanced/arfs) for structured data storage solutions. # Normalized Addresses (/build/advanced/normalized-addresses) ## Overview Different blockchains use different formats for the [public keys](/glossary) of wallets, and the [native addresses](/glossary) for those wallets. In most cases, when a system in the Arweave ecosystem needs to display the wallet address of a wallet from a different blockchain, for instance in the `Owner.address` value of a data item signed by an ETH or Solana wallet, that address will be normalized into the format recognized by Arweave. Specifically, a 43 character base64url representation of the sha256 hash of the public key. This is done to prevent potential errors by systems in the Arweave ecosystem that expect these values to be a certain size and conform to a specific format. Essentially, normalized addresses are a way to represent public keys and wallet addresses from other blockchains in a way that is familiar to systems in the Arweave ecosystem. A tool for easily obtaining a normalized addresses from public keys can be found at [ar://normalize-my-key](https://normalize-my-key.ar.io/) ## At A Glance | | Arweave | ETH/POL | Solana | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------- | | **Native Address** | 9ODOd-\_ZT9oWoRMVmmD4G5f9Z6MjvYxO3Nen-T5OXvU | 0x084af408C8E492aC52dc0Ec76514A7deF8D5F03f | Cd5yb4mvbuQyyJgAkriFZbWQivh2zM68KGZX8Ksn1L85 | | **base64url Encoded Public Key** | 0jkGWDFYI3DHEWaXhZitjTg67T-enQwXs50lTDrMhy2qb619_91drv_50J5PwrOYJiMmYhiEA5ojMvrrAFY-Dm1bJbJfVBU1kIsPho2tFcXnbSOa2_1bovAys0ckJU07wkbmIUpzp3trdxYReB4jayMMOXWw9B8xS0v81zFmK3IbCtL9N6WNTMONOSMATHFQrGqtDhDUqKyIsQZCBPFvfGykRWaLWzbtAUrApprqG9hfExQzppNsw0gsftNSHZ1emC5tC2fuib6FhQw9TE2ge9tUjEZNALcVZvopTtTX0H2gEfnRJ48UNeV3SKggjXcoPVeivmqXuPBGncXWWq1pHR-Xs4zSLA5Mgcw_tQJc4FIER0i7hUlZXoc991ZHyOvAC-GlHWzQwvrlY11oD38pB47NkHN2WVPtUCAtyYQe5TE6Xznd9kPgqqvVUkV0s0suh5vINGoiPEnMjyhYEN7eOmJRIJ_A87IJesbdPRV4ZzBsqPbd02RG3ZuVpc3gI1xKvwH1WS05XI8eWK-BbvB3oxB7WjaQTWcfBWhMEULiwx-SucuyAzPAw3i6Wjtq61TcL9SdWhmOf9_yo-Np052tj7MQ66nmgdOH_MEKYjAdFypxTsRQoSLbv28HEcSjwx8u3pY0q0gKMK_5X2XKJrp2i2GB_fVgbcpH9YsgrYxh1Q8 | 2W5VMzNKYwr51QsiYBHUS5h5wxZf_uBgG7C6xiHgBHwwLUty5LHKFFBDlAxTCTAhglcmys2_HQoOj_LnCkA3 | rK8XXxd8JqsZFPXVOwkSWS5Gh1SJzftfCOLpLk4i1FY | | **Normalized Address** | 9ODOd-\_ZT9oWoRMVmmD4G5f9Z6MjvYxO3Nen-T5OXvU | 5JtuS4yOFtUX2Rg3UU7AgBaUqh4s8wyyNTZk9UrzI-Q | K8kpPM1RID8ZM2sjF5mYy0rP4gXSRDbrwPUd9Qths64 | ## Public Keys and Addresses Crypto wallets consist of two separate components. The public keys, which are public knowledge and can be seen by anyone, and the private keys, which only the owner of a wallet should have access to. Crypto wallet addresses are derived from the public key. {" "} It is important to note that all crypto wallet public and private keys are binary data. The values provided below for Arweave and Ethereum/Polygon public keys are base64url and hex encoded representations of that binary data respectively. ### Arweave The public key for an Arweave wallet is the `n` field of the JWK json file. 0jkGWDFYI3DHEWaXhZitjTg67T-enQwXs50lTDrMhy2qb619_91drv_50J5PwrOYJiMmYhiEA5ojMvrrAFY-Dm1bJbJfVBU1kIsPho2tFcXnbSOa2_1bovAys0ckJU07wkbmIUpzp3trdxYReB4jayMMOXWw9B8xS0v81zFmK3IbCtL9N6WNTMONOSMATHFQrGqtDhDUqKyIsQZCBPFvfGykRWaLWzbtAUrApprqG9hfExQzppNsw0gsftNSHZ1emC5tC2fuib6FhQw9TE2ge9tUjEZNALcVZvopTtTX0H2gEfnRJ48UNeV3SKggjXcoPVeivmqXuPBGncXWWq1pHR-Xs4zSLA5Mgcw_tQJc4FIER0i7hUlZXoc991ZHyOvAC-GlHWzQwvrlY11oD38pB47NkHN2WVPtUCAtyYQe5TE6Xznd9kPgqqvVUkV0s0suh5vINGoiPEnMjyhYEN7eOmJRIJ_A87IJesbdPRV4ZzBsqPbd02RG3ZuVpc3gI1xKvwH1WS05XI8eWK-BbvB3oxB7WjaQTWcfBWhMEULiwx-SucuyAzPAw3i6Wjtq61TcL9SdWhmOf9_yo-Np052tj7MQ66nmgdOH_MEKYjAdFypxTsRQoSLbv28HEcSjwx8u3pY0q0gKMK_5X2XKJrp2i2GB_fVgbcpH9YsgrYxh1Q8 The public wallet address for that wallet is `9ODOd-_ZT9oWoRMVmmD4G5f9Z6MjvYxO3Nen-T5OXvU`, this is obtained by decoding the public key from base64url to normalize padding, sha256 hashing the result, and then base64url encoding that. ### Ethereum/Polygon The public key for an EVM wallet (Ethereum, Polygon/Matic) is derived from its private key, using the [Elliptic Curve Digital Signature Algorithm](https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm), or ECDSA. `0xb5d96e5533334a630af9d50b226011d44b9879c3165ffee0601bb0bac621e0047c302d4b72e4b1ca145043940c53093021825726cacdbf1d0a0e8ff2e70a4037` The public wallet address is `0x084af408C8E492aC52dc0Ec76514A7deF8D5F03f`, this is obtained by removing the first byte from the public key, Keccak-256 hashing the remainder, taking the the last 20 bytes (40 hexadecimal characters) and prepending `0x` to it. ### Solana A Solana wallet is an array of 64 bytes. The first 32 bytes are the private key, and the last 32 bytes are the public key. Below is the public key portion of a Solana wallet: `[172, 175, 23, 95, 23, 124, 38, 171, 25, 20, 245, 213, 59, 9, 18, 89, 46, 70, 135, 84, 137, 205, 251, 95, 8, 226, 233, 46, 78, 34, 212, 86]` The public wallet address for this wallet is `Cd5yb4mvbuQyyJgAkriFZbWQivh2zM68KGZX8Ksn1L85`, this is derived by base58 encoding the public key bytes. ## Normalizing Addresses As shown in the above examples, the format of public keys, and the resulting derived wallet addresses, vary widely between blockchains. Arweave manages this by applying the same derivation methods that Arweave uses for its own wallets to the public keys from other chains. ### Ethereum/Polygon The leading `0x` and uncompressed flag `04` (if present) is removed from the public key of an EVM wallet, and then the remainder is base64url encoded to obtain the Arweave normalized public key. Continuing with the same public key in the above example, the normalized public key would be: `2W5VMzNKYwr51QsiYBHUS5h5wxZf_uBgG7C6xiHgBHwwLUty5LHKFFBDlAxTCTAhglcmys2_HQoOj_LnCkA3` This value is what is used as the GraphQL tag `owner` value for data items being uploaded to Arweave using an EVM wallet. The normalized address is then derived from this value by sha256 hashing it, and then base64url encoding the result: `5JtuS4yOFtUX2Rg3UU7AgBaUqh4s8wyyNTZk9UrzI-Q` ### Solana The normalized public key for Solana wallets are derived similarly. The 32 byte public key is base64url encoded: `rK8XXxd8JqsZFPXVOwkSWS5Gh1SJzftfCOLpLk4i1FY` Again, this value is used for the GraphQl tag `owner` when uploading data. It can then be sha256 hashed, and base64url encoded again to derive the normalized address: `K8kpPM1RID8ZM2sjF5mYy0rP4gXSRDbrwPUd9Qths64` ## Discovering Public Keys from On-Chain Transactions In many cases, you may only have a wallet's native address and need to derive its normalized address. Since normalized addresses are derived from public keys, and public keys are not always readily available, you can recover them from onchain transaction signatures. The [ar://normalize-my-key](https://normalize-my-key.ar.io/) tool implements this discovery process, allowing you to look up an address or transaction hash/signature to automatically recover the public key and derive the normalized address. ### EVM Chains (Ethereum, Polygon, Base) EVM chains use the Elliptic Curve Digital Signature Algorithm (ECDSA) for transaction signing. A unique property of ECDSA is that the public key can be mathematically recovered from a valid signature and the signed message hash. The recovery process works as follows: 1. **Fetch a transaction** where the target address is the sender (the `from` field) 2. **Reconstruct the unsigned transaction** by extracting the transaction fields (nonce, to, value, gas, data, chainId, etc.) and determining the transaction type (legacy, EIP-2930, or EIP-1559) 3. **Serialize the transaction** according to its type and compute the Keccak-256 hash of the serialized data 4. **Extract the signature components** (r, s, v) from the transaction 5. **Recover the public key** using ECDSA recovery with the message hash and signature The `v` value encodes the recovery bit needed to identify which of the two possible public keys is correct: - For EIP-1559/EIP-2930 transactions: `v` is already 0 or 1 - For pre-EIP-155 legacy transactions: `v` is 27 or 28, subtract 27 to get the recovery bit - For EIP-155 transactions: `v = chainId * 2 + 35 + recoveryBit` Once the 65-byte uncompressed public key is recovered, the normalized address is derived by removing the `0x` prefix and uncompressed flag (`04`), base64url encoding the remaining 64 bytes, then sha256 hashing and base64url encoding the result. Public key recovery only works with transactions where the target address is the **sender**. If an address has only received funds but never sent any, there will be no signature from which to recover the public key. ### Solana Solana transactions include the signer's public key directly in the transaction's account keys array. The first account key in any transaction is always the fee payer/signer. The discovery process is simpler: 1. **Fetch a transaction** by its signature using the Solana RPC 2. **Extract the first account key** from the transaction's account keys (this is the signer's public key) 3. **Convert to normalized format** by base64url encoding the 32-byte public key, then sha256 hashing and base64url encoding the result Unlike EVM chains, Solana does not require cryptographic recovery since the public key is explicitly included in the transaction data. # Browser Sandboxing (/build/advanced/sandboxing) ## Overview Browser sandboxing allows data requests to a gateway node to benefit from the security advantages of using a browser's same-origin policy by redirecting the requests to a pseudo-unique subdomain of the gateway's apex domain. For example, an attempt to access `https://turbo-gateway.com/gnWKBqFXMJrrksEWrXLQRUQQQeFhv4uVxesHBcT8i6o` would redirect to `https://qj2yubvbk4yjv24syelk24wqivcbaqpbmg7yxfof5mdqlrh4rova.turbo-gateway.com/gnWKBqFXMJrrksEWrXLQRUQQQeFhv4uVxesHBcT8i6o` Two DNS records are required to link a domain to an Arweave transaction on a gateway node. For example, `www.mycustomsite.com` would need the following records to link it to `www.arweave-gateway.net`: - A DNS CNAME record pointing to an Arweave gateway: www CNAME `arweave-gateway.net`, - A DNS TXT record linking the domain with a specific transaction ID: arweavetx TXT `kTv4OkVtmc0NAsqIcnHfudKjykJeQ83qXXrxf8hrh0S` When a browser requests `www.mycustomsite.com` the user's machine will (through the usual DNS processes) resolve this to the IP address for the gateway node `arweave-gateway.net`. When the gateway receives an HTTP request with a non-default hostname, e.g. `www.mycustomsite.com` instead of `www.arweave-gateway.net`, the gateway will query the DNS records for `www.mycustomsite.com` and the 'arweavetx' TXT record will tell the node which transaction to serve. ## TLS and its Role in Browser Sandboxing Transport Layer Security (TLS) is a cryptographic protocol designed to provide communications security over a computer network. In the context of Arweave applications and browser sandboxing, TLS plays a critical role in ensuring secure data transmission and enabling the effective use of browser security features. When Arweave applications are accessed without TLS, most browsers restrict the use of native cryptographic functions. These functions, which include hashing, signing, and verification, are essential for the secure operation of Arweave permaweb apps. Without TLS, not only are these functions unavailable, but the applications also become susceptible to various security threats, notably man-in-the-middle (MITM) attacks. Although Arweave transactions are signed, making direct MITM attacks challenging, the absence of encryption can expose other vulnerabilities. For instance, attackers could intercept and alter the `/price` endpoint, potentially causing transaction failures or leading to overcharging. To address these concerns, gateway operators are responsible for generating and maintaining TLS certificates for their gateways. This can be achieved through various systems, such as ACME for Let's Encrypt. An important step in setting up a gateway is obtaining a wildcard TLS certificate for the gateway's domain. This certificate secures traffic on both the apex domain and its single-level subdomains (e.g., `gateway.com` and `subdomain.gateway.com`). The integration of TLS is crucial for the implementation of browser sandboxing. When a browser requests a transaction from a gateway, the gateway issues a 301 redirect to a subdomain of the gateway, using a Base32 pseudo-unique address derived from the transaction ID. This redirection, secured by TLS, invokes the browser's same-origin policy. As a result, the requested web page is confined within a secure sandbox environment, isolated from other domains. This isolation is vital for maintaining the integrity and security of transactions and interactions within Arweave's permaweb applications. ## Deriving Sandbox Value Ar.io nodes generate browser sandbox values deterministically. Because of this, it is possible to calculate ahead of time what that value will be for a particular transaction id. Sandbox values are a Base32 encoding of the transaction ID. Ar.io gateways use the following code snippet to accomplish the encoding: ```typescript const expectedTxSandbox = (id: string): string => { return toB32(fromB64Url(id)) } ``` Example: ```typescript const id = 'gnWKBqFXMJrrksEWrXLQRUQQQeFhv4uVxesHBcT8i6o' const expectedTxSandbox = (id): string => { return toB32(fromB64Url(id)) } console.log(expectedTxSandbox) ``` Example Output: ```console qj2yubvbk4yjv24syelk24wqivcbaqpbmg7yxfof5mdqlrh4rova ``` View the full code for generating browser sandbox values [here](https://github.com/ar-io/arweave-gateway/blob/719f43f8d6135adf44c87701e95f58105638710a/src/gateway/middleware/sandbox.ts#L69). # AI Agents & LLMs (/build/agents) These docs are built to be consumed by **AI coding agents**, not just read by humans. Everything below is a plain-text file you can fetch, paste, or point an agent at — no scraping required. **Fastest path:** point your agent at `https://docs.ar.io/llms.txt`. It's a short index that links everything else, following the [llmstxt.org](https://llmstxt.org) convention agents already look for. ## Machine-readable entry points | File | What it is | Reach for it when | |---|---|---| | `/llms.txt` | A short **index** of the documentation with links and quick facts | Discovery — the entry point | | `/SKILL.md` | A **procedural agent skill**: SDK patterns, code recipes, URL conventions, constants | Generating working code | | `/llms-full.txt` | The **entire docs corpus** as one plain-text file (~1.2 MB) | Deep context, or RAG ingestion | | `/sdks//llm.txt` | **Per-SDK reference** text (see table below) | Working inside a single SDK | ### `llm.txt` vs `SKILL.md` — what's the difference? They solve different problems and are meant to be used together: - **`llms.txt` / `llms-full.txt` / `llm.txt` are *reference*.** They're documentation flattened to plain text so a model has the facts in context. Passive — they describe *what exists*. - **`SKILL.md` is *procedural*.** It follows the [Agent Skills](https://code.claude.com/docs/en/skills) convention: task-oriented instructions, known-good code recipes, and constants that teach an agent *how to actually build* on ar.io — which SDK call to make, the correct ArNS URL shape, what to avoid. A useful rule of thumb: load **`SKILL.md`** when the agent needs to *write code*, and **`llms-full.txt`** when it needs to *answer questions* about the wider platform. ## Per-SDK reference text Each SDK ships its own flattened reference, which is much cheaper to load than the full corpus: | SDK | Plain text | |---|---| | ar.io SDK | `/sdks/ar-io-sdk/llm.txt` | | Turbo SDK | `/sdks/turbo-sdk/llm.txt` | | Wayfinder | `/sdks/wayfinder/llm.txt` | | ArDrive Core JS | `/sdks/ardrive-core-js/llm.txt` | | CLIs (ArDrive CLI, ARIO Deploy) | `/sdks/(clis)/llm.txt` | ## Give your agent somewhere safe to run Agents learn fastest by *doing*, but you don't want an agent spending real money or writing permanent data while it figures things out. The **[Testnet Sandbox](/build/testnet)** is built for exactly this: - Runs the **full ar.io stack** — upload, payment, ArNS, and gateway — on Solana **devnet**. - Funded by a **faucet**, so there's no real value at risk. - **Nothing is permanent**: data is purged after ~3 days and never reaches mainnet Arweave. **One step needs a human.** The ARIO faucet is GitHub-gated and its OAuth consent can't be completed headlessly. Have a person claim once to the wallet your agent will use — after that, uploading, funding, and buying names are all scriptable. See [Agents and CI](/build/testnet/funds-and-faucet#agents-and-ci). ## Skills and integrations Several ar.io repositories ship agent tooling: - **ARIO Deploy** — a Claude Code skill for deploying apps to the permaweb and updating ArNS. See [what the skill does](/sdks/ario-deploy/what-the-skill-does) and [add it to your project](/sdks/ario-deploy/add-the-skill-to-your-project) for setup. - **`ar-io-gateway-operator`** — a conformant [Agent Skill](https://code.claude.com/docs/en/skills) for operating an ar.io node: health, indexing lag, ArNS resolution, data retrieval, and common pitfalls. Ships in the [ar-io-node](https://github.com/ar-io/ar-io-node) repo under `.claude/skills/ar-io-gateway-operator/`. - **`ario-testnet-faucet`** — the faucet claim API and flow, for agents working against the [Testnet Sandbox](/build/testnet). Ships in the `ar-io-faucet` repo. `/SKILL.md` above is itself a conformant Agent Skill — it carries `name` and `description` frontmatter, so you can drop it straight into a project as `.claude/skills/ario/SKILL.md`. ## Tools built into these docs Every page carries agent-friendly affordances: - **Open in AI** — send the current page straight to ChatGPT, Claude, Scira, or T3 Chat. - **Copy as Markdown** — grab the raw source of any page for pasting into a prompt. - **Ask Arie** — an in-docs AI assistant that answers from this documentation with citations. ## Pointing an agent at ar.io ### Load the index Fetch `https://docs.ar.io/llms.txt` so the agent knows what's available and where. ### Load the skill for code generation Fetch `https://docs.ar.io/SKILL.md` for known-good patterns, constants, and URL conventions. ### Narrow to an SDK Pull the relevant `llm.txt` (table above) instead of the full corpus — smaller context, better answers. ### Run against the sandbox Build and test on the [Testnet Sandbox](/build/testnet) before touching mainnet. ## Related # Bundler (/build/extensions/bundler) ## Overview A Turbo ANS-104 data item bundler can be run alongside an ar.io gateway. This allows gateways the ability to accept data items to be submitted to the Arweave blockweave. The bundler service can be easily run inside Docker in the same way that the gateway is. It utilizes a separate docker compose file for configuration and deployment, which also allows for the use of a separate file for environmental variables specific to the bundler service. Additionally, the separation allows operators to spin their bundler service up or down at any time without affecting their core gateway service. Despite the use of separate docker compose files, the bundler service shares a docker network with the ar.io gateway, and so is able to directly interact with the gateway service and data. For more information on ANS-104 Bundles, see the [ANS-104 Bundles](/learn/ans-104-bundles) page. ## Getting Started **NOTE**: The bundler service relies on GraphQL indexing of recently bundled and uploaded data to manage its pipeline operations. The ar.io gateway should have its indexes synced up to Arweave's current block height before starting the bundler's service stack. ### Configure Environmental Variables Environmental variables must be provided for the bundler to function and integrate properly with an existing ar.io gateway. The gateway repository provides a `.env.bundler.example` file that can be renamed to `.env.bundler` and used as a starting point. It contains the following: ```bash BUNDLER_ARWEAVE_WALLET='Stringified JWK wallet. e.g: '{ "n": "...", ... }' BUNDLER_ARWEAVE_ADDRESS='Address for above wallet' APP_NAME='ar.io bundler service' # Use localstack s3 bucket for shared data source between ar.io gateway and bundler AWS_S3_BUCKET=ar.io AWS_S3_PREFIX='data' AWS_ACCESS_KEY_ID='test' AWS_SECRET_ACCESS_KEY='test' AWS_REGION='us-east-1' AWS_ENDPOINT='http://localstack:4566' ``` - `BUNDLER_ARWEAVE_WALLET` must be the entire jwk of an Arweave wallet's keyfile, stringified. All uploads of bundled data items to Arweave will be signed and paid for by this wallet, so it must maintain a balance of AR tokens sufficient to handle the uploads. - `BUNDLER_ARWEAVE_ADDRESS` must be the [normalized public address](/glossary) for the provided Arweave wallet. - `APP_NAME` is a GraphQL tag that will be added to uploaded bundles. The remaining lines in the `.env.bundler.example` file control settings that allow the bundler service to share data with the ar.io gateway. Data sharing of contiguous data between a bundler and a gateway allows the gateway to serve optimistically cached data without waiting for it to fully settle on chain. ### Configure Optimistic Indexing By default, the bundler will only accept data items uploaded by data item signers whose [normalized wallet addresses](/glossary) are in the `ALLOW_LISTED_ADDRESSES` list. This is an additional environmental variable that can be added to your `.env.bundler` file, and must be a comma separated list of normalized public wallet addresses for wallets that should be allowed to bundle and upload data through your gateway. ```bash ALLOW_LISTED_ADDRESSES=, ``` The following permissioning configurations schemes are also possible: | Scheme | ALLOW_LISTED_ADDRESSES | SKIP_BALANCE_CHECKS | ALLOW_LISTED_SIGNATURE_TYPES | PAYMENT_SERVICE_BASE_URL | | -------------------------- | ------------------------------------------- | ------------------- | ---------------------------- | ------------------------ | | **Allow Specific Wallets** | Comma-separated normalized wallet addresses | false | EMPTY or supplied | EMPTY | | **Allow Specific chains** | EMPTY or supplied | false | arbundles sigtype int | EMPTY | | **Allow All** | n/a | true | n/a | n/a | | **Allow None** | EMPTY | false | EMPTY | EMPTY | | **Allow Payers** | EMPTY or supplied | false | EMPTY or supplied | Your payment service url | ### Set Up Indexing Bundlers submit data to the Arweave network as an [ANS-104 data item bundle](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md). This means it is several transactions wrapped into one. A gateway will need to unbundle these transactions in order to index them. A gateway should include the following ANS-104 filters in order to unbundle and index transactions from a particular bundler: ```bash ANS104_INDEX_FILTER={ "always": true } ANS104_UNBUNDLE_FILTER={ "attributes": { "owner_address": "$BUNDLER_ARWEAVE_ADDRESS" } } ``` `$BUNDLER_ARWEAVE_ADDRESS` should be replaced with the [normalized public wallet address](/glossary) associated with the bundler. **NOTE**: The above filters must be placed in the `.env` file for the core gateway service, not the bundler. Gateways handle data item indexing asynchronously. This means they establish a queue of items to index, and work on processing the queue in the background while the gateway continues with its normal operations. If a gateway has broad indexing filters, there can be some latency in indexing data items from the bundler while the gateway works through its queue. ### Configure Optimistic Indexing Gateway operators control access to their [optimistic data item indexing](/glossary) API via an admin key that must be supplied by all bundling clients in order for their requests to be accepted. This key should be made available in the environment configuration files for BOTH the core gateway, and the bundler, and should be provided as `AR_IO_ADMIN_KEY`: ```bash AR_IO_ADMIN_KEY="Admin password" ``` **NOTE**: If a gateway is started without providing the admin key, a random string will be generated to protect the gateway's admin endpoints. This can be reset by restarting the gateway with the admin key provided in the `.env` file. ## Starting and Stopping the Bundler ### Starting The bundler service is designed to run in conjunction with an ar.io gateway, and so relies on the `ar-io-network` network created in Docker when the core gateway services are spun up. It is possible to spin up the bundler while the core services are down, but the network must exist in Docker. To start the bundler, specify the env and docker-compose files being used in a `docker compose up` command: ```bash docker compose --env-file ./.env.bundler --file docker-compose.bundler.yaml up -d ``` The `-d` flag runs the command in "detached" mode, so it will run in the background without requiring the terminal to remain active. ### Stopping To spin the bundler service down, specify the docker-compose file in a `docker compose down` command: ```bash docker compose --file docker-compose.bundler.yaml down ``` ### Logs While the bundler service is running in detached mode, logs can be checked by specifying the docker-compose file in a `docker compose logs` command: ```bash docker compose --file docker-compose.bundler.yaml logs -f --tail=0 ``` - `-f` runs the command in "follow" mode, so the terminal will continue to watch and display new logs. - `--tail=` defines the number of logs to display that existed prior to running the command. `0` displays only new logs. ## Useful Docker Commands Monitor and manage your bundler service with these commands: ```bash # View all running services docker ps # Start bundler service in background docker compose --env-file ./.env.bundler --file docker-compose.bundler.yaml up -d # Stop bundler service docker compose --file docker-compose.bundler.yaml down # Pull latest bundler images docker compose --file docker-compose.bundler.yaml pull # Follow bundler logs docker compose --file docker-compose.bundler.yaml logs -f --tail=10 # Check bundler service status docker compose --file docker-compose.bundler.yaml ps # Restart bundler service docker compose --file docker-compose.bundler.yaml restart ``` ## Next Steps Now that you have a bundler set up to accept data uploads, continue building your gateway infrastructure: } title="Set Up Monitoring" description="Deploy Grafana to visualize your gateway's performance metrics" href="/build/extensions/grafana" /> } title="Add ClickHouse" description="Improve query performance with ClickHouse and Parquet integration" href="/build/extensions/clickhouse" /> } title="Buy an ArNS Name" description="Get a human-readable name for your gateway and start serving the permanent web" href="/learn/arns/name-registration" /> # ClickHouse & Parquet (/build/extensions/clickhouse) ClickHouse is an optional analytical database that runs alongside your ar.io gateway to accelerate GraphQL queries on large datasets. It stores stable historical data in a compressed columnar format while SQLite continues handling real-time writes and recent data. ClickHouse requires version **24.8 or later**. Earlier versions have known issues with projections on ReplacingMergeTree tables. The default Docker image uses ClickHouse 26.3. ## How It Works The gateway uses a speed/batch layer architecture: - **SQLite (speed layer)** — Handles all real-time writes, recent/unstable data, and non-GraphQL operations (data retrieval, chunk fetches, admin APIs) - **ClickHouse (batch layer)** — Stores stable historical data exported as Parquet files, optimized for analytical GraphQL queries |writes| B[SQLite] B -->|stable data export| C[Parquet Files] C -->|batch import| D[ClickHouse] E[GraphQL Query] -->|height > boundary| B E -->|height ≤ boundary| D `} /> ### Query Routing When ClickHouse is enabled, GraphQL queries are split by block height: - **ClickHouse** handles blocks up to `(max_imported_height - 10)` — the bulk of historical data - **SQLite** handles blocks above that boundary — recent, potentially unstable data Results from both backends are merged and deduplicated before being returned to the client. If either backend is unavailable, a circuit breaker degrades gracefully rather than failing the entire query. ### When to Enable ClickHouse Enable ClickHouse if your gateway: - Serves heavy GraphQL traffic on historical data - Indexes millions of data items (large unbundling filters) - Needs faster bootstrapping than a full SQLite reindex (import pre-built Parquet snapshots) - Would benefit from 3-5x storage compression on historical data Keep it disabled for small deployments, gateways that only serve data retrieval (not GraphQL), or single-node low-resource environments. ## Quick Start ### Configure Environment Add ClickHouse settings to your `.env` file: ```bash CLICKHOUSE_URL=http://clickhouse:8123 CLICKHOUSE_PASSWORD=your-password CLICKHOUSE_SQLITE_MIN_HEIGHT_ENABLED=true ADMIN_API_KEY=your-admin-key ``` If using a non-default username: ```bash CLICKHOUSE_USER=your-username ``` ### Configure Unbundling Filters ClickHouse stores data items exported from SQLite. The more you unbundle, the more data flows into ClickHouse. At minimum, configure filters that match the pre-built snapshot you plan to import. The ArDrive snapshot includes all ArDrive-uploaded data items. To match this: ```bash ANS104_UNBUNDLE_FILTER='{ "and": [ { "not": { "or": [ { "tags": [ { "name": "Bundler-App-Name", "value": "Warp" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "Redstone" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "AO" } ] }, { "attributes": { "owner_address": "-OXcT1sVRSA5eGwt2k6Yuz8-3e3g9WJi5uSE99CWqsBs" } }, { "attributes": { "owner_address": "ZE0N-8P9gXkhtK-07PQu9d8me5tGDxa_i4Mee5RzVYg" } }, { "attributes": { "owner_address": "6DTqSgzXVErOuLhaP0fmAjqF4yzXkvth58asTxP3pNw" } } ] } }, { "tags": [ { "name": "App-Name", "valueStartsWith": "ArDrive" } ] } ] }' ANS104_INDEX_FILTER='{ "tags": [ { "name": "App-Name", "value": "ArDrive-App" } ] }' ``` See the [Filters guide](/build/run-a-gateway/manage/filters) for building custom filter configurations. ### Start ClickHouse ClickHouse runs as a Docker profile alongside your gateway: ```bash docker compose --profile clickhouse up -d ``` This starts both the ClickHouse server and the auto-import daemon. ### Import a Parquet Snapshot Download and import a pre-built snapshot to bootstrap your ClickHouse instance: ```bash # Download the ArDrive snapshot (~3.5GB, current to April 23, 2025) curl -L https://turbo-gateway.com/JVmsuD2EmFkhitzWN71oi9woADE4WUfvrbBYgremCBM -o ardrive-parquet.tar.gz # Extract tar -xzf ardrive-parquet.tar.gz # Move into the data directory mkdir -p data/parquet mv 2025-04-23-ardrive-ans104-parquet/* data/parquet/ # Run the import script ./scripts/clickhouse-import ``` The import takes 10-20 minutes depending on hardware. ### Verify Check the transaction count: ```bash docker compose exec clickhouse clickhouse-client \ --password your-password \ -q 'SELECT COUNT(DISTINCT id) FROM transactions' ``` Expected result: `32712311` (for the ArDrive snapshot). Test a GraphQL query: ```bash curl -g -X POST \ -H "Content-Type: application/json" \ -d '{"query":"query { transactions(ids: [\"YSNwoYB01EFIzbs6HmkGUjjxHW3xuqh-rckYhi0av4A\"]) { edges { node { block { height } bundledIn { id } } } } }"}' \ http://localhost:3000/graphql ``` ## Auto-Import The `clickhouse-auto-import` container runs a continuous loop that exports new stable data from SQLite to Parquet and imports it into ClickHouse. This keeps ClickHouse up to date without manual intervention. Each cycle: 1. Reloads TTL rules from `config/clickhouse-ttl-rules.yaml` 2. Advances the 3. Exports new Parquet partitions from SQLite 4. Imports Parquet into ClickHouse staging tables 5. Migrates staging to the final `transactions` table (applying TTL rules) ### Auto-Import Configuration | Variable | Default | Description | |----------|---------|-------------| | `CLICKHOUSE_AUTO_IMPORT_SLEEP_INTERVAL` | `3600` | Seconds between import cycles | | `CLICKHOUSE_AUTO_IMPORT_HEIGHT_INTERVAL` | `1000` | Blocks per Parquet partition | | `CLICKHOUSE_AUTO_IMPORT_MAX_ROWS_PER_FILE` | - | Maximum rows per Parquet file | The default 1-hour interval is conservative. For gateways with high unbundling throughput, reduce to 600-900 seconds to keep ClickHouse more current. ## TTL Rules (Data Retention) TTL rules control how long data items remain in ClickHouse before automatic expiration. Use them to manage storage costs by expiring ephemeral or low-value data while retaining important content indefinitely. ### Configuration Copy the example template and edit: ```bash cp config/clickhouse-ttl-rules.example.yaml config/clickhouse-ttl-rules.yaml ``` Example configuration: ```yaml # Keep L1 transactions forever, expire unmatched data items after 30 days default_ttl_seconds: 2592000 l1_never_expires: true rules: # Keep ArDrive content forever - tag_name: App-Name tag_value: ArDrive match: prefix never_expire: true # Expire ephemeral chat messages after 1 day - tag_name: App-Name tag_value: ephemeral-chat ttl_seconds: 86400 # Expire test uploads after 1 hour - tag_name: App-Name tag_value: test- match: prefix ttl_seconds: 3600 # Expire specific owner's data after 7 days - field: owner_address value: abcDEF0123xyz ttl_seconds: 604800 ``` ### Rule Precedence When multiple rules could apply, the first matching branch wins: 1. `l1_never_expires: true` AND row is L1 transaction → kept forever 2. Any matching rule with `never_expire: true` → kept forever 3. One or more matching TTL rules → shortest `ttl_seconds` wins 4. `default_ttl_seconds` set → applied to all unmatched rows 5. Otherwise → kept indefinitely (no expiry) ### Match Modes - **`exact`** (default) — O(1) dictionary lookup. Use for specific tag values. - **`prefix`** — Matches the beginning of the value. Use for `Content-Type` with parameters (e.g., `image/gif` catches `image/gif; charset=...`) or app name families. TTL rules apply only to rows imported after the rules are loaded. Previously imported rows keep their existing expiration. To apply new rules retroactively, re-import affected Parquet partitions. ### Normalization - Tag names are lower-cased and trimmed - Tag values are trimmed but case-preserving - Owner values use base64url format (as displayed on Arweave explorers) ## Deployment Topologies ### Single Gateway (Default) One gateway runs ClickHouse locally. The auto-import daemon handles continuous data ```bash docker compose --profile clickhouse up -d ``` ### Shared ClickHouse Cluster Multiple gateways share a replicated ClickHouse cluster. Only **one** gateway runs `clickhouse-auto-import` as the writer; others query the same cluster as readers. ```bash # Writer gateway CLICKHOUSE_URL=http://clickhouse-writer:8123 # runs clickhouse-auto-import # Reader gateways CLICKHOUSE_URL=http://clickhouse-reader:8123 # do NOT run clickhouse-auto-import ``` Reader gateways still need `START_WRITERS=true` because SQLite indexing is required for data retrieval, chunk fetches, and other non-GraphQL operations. Only GraphQL queries are routed to ClickHouse. ### Independent ClickHouse per Gateway Each gateway runs its own ClickHouse instance, bootstrapped from a shared Parquet snapshot (downloaded from Arweave, S3, or BitTorrent). No shared state between gateways. This is the simplest topology for operators who want ClickHouse performance without cluster management. ## Configuration Reference ### Connection | Variable | Default | Description | |----------|---------|-------------| | `CLICKHOUSE_URL` | - | ClickHouse HTTP endpoint (e.g., `http://clickhouse:8123`) | | `CLICKHOUSE_USER` | `default` | ClickHouse username | | `CLICKHOUSE_PASSWORD` | - | ClickHouse password | ### Query Routing | Variable | Default | Description | |----------|---------|-------------| | `CLICKHOUSE_SQLITE_MIN_HEIGHT_ENABLED` | `false` | Enable height-based query split between ClickHouse and SQLite | | `CLICKHOUSE_SQLITE_MIN_HEIGHT_BUFFER` | `10` | Number of blocks below ClickHouse max height still served by SQLite | | `CLICKHOUSE_MAX_HEIGHT_CACHE_TTL_SECONDS` | `60` | Cache duration for ClickHouse max-height lookup | | `CLICKHOUSE_QUERY_TIMEOUT_SECONDS` | `3` | Query timeout (both client and server side) | | `CLICKHOUSE_GQL_MAX_ROWS_TO_READ` | `10000000` | Safety guardrail: max rows scanned per query | | `CLICKHOUSE_GQL_DEDUPE_HEADROOM` | `4` | Pagination headroom multiplier for deduplication | ### Circuit Breaker The circuit breaker protects against cascading failures when the SQLite leg is unhealthy. When tripped, queries degrade to ClickHouse-only results with a partial result warning. | Variable | Default | Description | |----------|---------|-------------| | `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_TIMEOUT_MS` | `5000` | SQLite query timeout before counting as error | | `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_ERROR_THRESHOLD_PERCENTAGE` | `80` | Error rate percentage that trips the breaker | | `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_ROLLING_COUNT_TIMEOUT_MS` | `60000` | Rolling window for error rate calculation | | `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_RESET_TIMEOUT_MS` | `30000` | Wait time before attempting half-open trial | ## Tuning ### Short Pages in GraphQL Pagination If GraphQL queries return fewer results than expected per page, unmerged duplicates in ClickHouse may be consuming pagination slots. Increase the deduplication headroom: ```bash CLICKHOUSE_GQL_DEDUPE_HEADROOM=8 # default is 4 ``` This makes the inner query fetch more rows to compensate for duplicates that get collapsed. ### Query Timeouts For queries over very large datasets or complex tag filters, increase the timeout: ```bash CLICKHOUSE_QUERY_TIMEOUT_SECONDS=10 # default is 3 ``` ### Max Rows Safety Guardrail If queries hit the `max_rows_to_read` limit (returns an error), you can raise it — but do so with caution as it removes the protection against runaway full-table scans: ```bash CLICKHOUSE_GQL_MAX_ROWS_TO_READ=50000000 # default is 10M ``` ## Upgrading ### Schema Evolution Schema changes are applied automatically on each import cycle via idempotent `ALTER TABLE` statements. No manual migration steps are needed for routine upgrades. ### Projection Rebuild (One-Time) Gateways upgrading from pre-24.8 ClickHouse deployments may need to rebuild the owner projection: ```sql ALTER TABLE transactions MATERIALIZE PROJECTION owner_projection; ``` This runs in the background and may take time on large tables. ### Rollback If you need to roll back a ClickHouse upgrade: 1. Drop the `transactions` table 2. Re-import from your Parquet files using `./scripts/clickhouse-import` Parquet files are the source of truth — ClickHouse is always rebuildable from them. ## Troubleshooting ### Import Failures Failed imports leave files in `data/etl/staging/` for inspection. Check the auto-import container logs: ```bash docker compose --profile clickhouse logs clickhouse-auto-import -f ``` Fix the underlying issue (disk space, permissions, ClickHouse connectivity) and restart the container — the next cycle will retry. ### Disk Usage Growing If ClickHouse disk usage is higher than expected: - Check your TTL rules are configured and expiring data as intended - Verify background merges are running (rows are only deleted on merge): ```bash docker compose exec clickhouse clickhouse-client \ --password your-password \ -q 'SELECT partition, count() FROM system.parts WHERE table = '\''transactions'\'' GROUP BY partition' ``` - Force a merge on a specific partition if needed (heavyweight operation): ```bash OPTIMIZE TABLE transactions PARTITION 14 FINAL ``` ### Checking Import Status View the current max imported height: ```bash docker compose exec clickhouse clickhouse-client \ --password your-password \ -q 'SELECT max(height) FROM transactions' ``` Compare this to the gateway's current height (`/height` endpoint) — the difference indicates how far behind ClickHouse is. ## Docker Commands ```bash # Start gateway with ClickHouse docker compose --profile clickhouse up -d # Stop gateway with ClickHouse docker compose --profile clickhouse down # View ClickHouse logs docker compose --profile clickhouse logs clickhouse -f # View auto-import logs docker compose --profile clickhouse logs clickhouse-auto-import -f # Restart auto-import after config change docker compose --profile clickhouse restart clickhouse-auto-import # Pull latest images docker compose --profile clickhouse pull ``` ## Related } title="Filters" description="Configure unbundling and indexing filters that control what data flows into ClickHouse" href="/build/run-a-gateway/manage/filters" /> } title="Grafana Monitoring" description="Visualize gateway and ClickHouse performance metrics" href="/build/extensions/grafana" /> } title="Environment Variables" description="Full configuration reference for all gateway services" href="/build/run-a-gateway/manage/environment-variables" /> } title="Gateway Architecture" description="Understand the data layer and worker system architecture" href="/learn/gateways/architecture" /> # Grafana (/build/extensions/grafana) ## Overview Ar.io gateways track extensive performance and operational metrics using [Prometheus](https://prometheus.io/). A [Grafana](https://grafana.com/) sidecar can be deployed to visualize these metrics, providing an easy way to monitor gateway health and performance. The Grafana sidecar is deployed as a separate Docker container that uses the same network as the gateway, making it simple to integrate with your existing setup. ![Grafana Dashboard](/content/grafana.png) ## Quick Start ### Deploy Grafana Deploy the Grafana sidecar using the provided Docker Compose file: ```bash docker compose -f docker-compose.grafana.yaml up -d ``` This command assumes you're running from the root directory of the gateway. If running from a different directory, adjust the path to the docker-compose file accordingly. ### Verify Deployment Check that Grafana is running properly: ```bash docker compose -f docker-compose.grafana.yaml logs -f --tail=25 ``` Press `Ctrl+C` to exit the logs. Look for any error messages or permission issues. ### Access Grafana Navigate to `http://localhost:1024` in your browser to access Grafana. **Default credentials:** - Username: `admin` - Password: `admin` Updated credentials may be lost if the Grafana sidecar is restarted. Be sure to log into Grafana immediately after every startup to ensure Grafana cannot be accessed with the default credentials. ## Exposing Dashboard Publicly To expose your Grafana dashboard externally through your domain, you'll need to configure nginx as a reverse proxy. This requires DNS setup and SSL certificates as covered in the [gateway installation guide](/build/run-a-gateway/quick-start). This setup assumes you've already configured DNS, SSL certificates, and nginx as described in the [Installation & Setup guide](/build/run-a-gateway/quick-start). ### Deploy Grafana Sidecar First, ensure your Grafana container is running: ```bash docker compose -f docker-compose.grafana.yaml up -d ``` Verify it's accessible locally at `http://localhost:1024`. ### Update Nginx Configuration Edit your existing nginx configuration file (`/etc/nginx/sites-available/default`) to add the Grafana location block: ```nginx # Add this block inside your existing HTTPS server block (port 443) location /grafana/ { proxy_pass http://localhost:1024/grafana/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } ``` Your complete nginx configuration should look like this: ```nginx # Force redirects from HTTP to HTTPS server { listen 80; listen [::]:80; server_name .com *..com; location / { return 301 https://$host$request_uri; } } # Forward traffic to your node and provide SSL certificates server { listen 443 ssl; listen [::]:443 ssl; server_name .com *..com; ssl_certificate /etc/letsencrypt/live/.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/.com/privkey.pem; location / { proxy_pass http://localhost:3000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; # Forward ar.io headers if present in the request proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; } # Grafana dashboard access location /grafana/ { proxy_pass http://localhost:1024/grafana/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } ``` ### Test and Reload Nginx Validate your nginx configuration: ```bash sudo nginx -t ``` If the configuration is valid, reload nginx: ```bash sudo systemctl reload nginx ``` ### Access Your Dashboard Navigate to `https://.com/grafana/` in your browser to access your Grafana dashboard externally. **Default credentials:** - Username: `admin` - Password: `admin` ## Troubleshooting ### Fix Permission Issues ### Method 1: Modify Directory Permissions The simplest solution is to modify the permissions of the Grafana data directory: ```bash sudo chmod -R 777 ./data/grafana ``` This command assumes you're running from the root directory of the gateway. Adjust the path if running from a different directory. ### Method 2: Change Grafana User Alternatively, modify the `docker-compose.grafana.yaml` file to use root user: ```yaml grafana: image: grafana/grafana:latest user: root ports: - "3000:3000" ``` ### Verify Fix Restart Grafana and check logs: ```bash docker compose -f docker-compose.grafana.yaml restart docker compose -f docker-compose.grafana.yaml logs -f ``` ### Resolve Connection Problems ### Check Container Status Verify Grafana is running: ```bash docker compose -f docker-compose.grafana.yaml ps ``` ### Check Port Availability Ensure the port isn't already in use: ```bash netstat -tulpn | grep :1024 # or lsof -i :1024 ``` ### Review Logs Check for specific error messages: ```bash docker compose -f docker-compose.grafana.yaml logs --tail=50 ``` ### Fix Configuration Issues ### Validate Nginx Configuration Test your Nginx configuration: ```bash sudo nginx -t ``` ### Check Proxy Settings Ensure proxy headers are correctly configured: ```nginx location /grafana/ { proxy_pass http://localhost:1024/grafana/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } ``` ### Restart Services Restart both Nginx and Grafana: ```bash sudo systemctl restart nginx docker compose -f docker-compose.grafana.yaml restart ``` ## Security Considerations Always change the default admin credentials immediately after first login. Default credentials are publicly known and pose a security risk. ### Best Practices 1. **Change Default Password** - Use a strong, unique password 2. **Enable HTTPS** - Use SSL certificates for external access 3. **Restrict Access** - Use firewall rules to limit access 4. **Regular Updates** - Keep Grafana updated to latest version 5. **Backup Configuration** - Export and backup dashboard configurations ## Documentation & Support - **Grafana Documentation** - [Official Grafana docs](https://grafana.com/docs/) - **Prometheus Metrics** - [Understanding gateway metrics](https://prometheus.io/docs/concepts/metric_types/) - **Community Support** - Join the [ar.io Discord](https://discord.com/invite/HGG52EtTc2) for help ## Useful Docker Commands Monitor and manage your Grafana sidecar with these commands: ```bash # View all running services docker ps # Start Grafana sidecar docker compose -f docker-compose.grafana.yaml up -d # Stop Grafana sidecar docker compose -f docker-compose.grafana.yaml down # Pull latest Grafana images docker compose -f docker-compose.grafana.yaml pull # Follow Grafana logs docker compose -f docker-compose.grafana.yaml logs -f --tail=25 # Check Grafana container status docker compose -f docker-compose.grafana.yaml ps # Restart Grafana sidecar docker compose -f docker-compose.grafana.yaml restart # View Grafana logs without following docker compose -f docker-compose.grafana.yaml logs --tail=50 # Start Grafana in foreground (for debugging) docker compose -f docker-compose.grafana.yaml up # Check port availability netstat -tulpn | grep :1024 ``` ## Next Steps Now that you have monitoring set up, continue building your gateway infrastructure: } title="Optimize Performance" description="Learn advanced gateway optimization techniques for better performance" href="/build/run-a-gateway/manage/filters" /> } title="Add ClickHouse" description="Improve query performance with ClickHouse and Parquet integration" href="/build/extensions/clickhouse" /> } title="Deploy Bundler" description="Accept data uploads directly through your gateway" href="/build/extensions/bundler" /> # Extensions & Sidecars (/build/extensions) ## What are Extensions? Extensions are additional scripts and tools you can run alongside your gateway to expand its capabilities or enhance the operator experience. The full list of community extensions can be found at [gateways.ar.io/#/extensions](https://gateways.ar.io/#/extensions). ## What are Sidecars? Sidecars are dockerized services that add additional functionality, APIs, and services to ar.io gateways. They run as separate containers alongside your gateway, providing specialized capabilities. ## Getting Started with Team-Supported Sidecars The following sidecars are developed and maintained by the ar.io team, designed to run alongside your gateway as separate containers. }> Visualize gateway metrics with comprehensive dashboards and performance monitoring. } > Improve query performance for large datasets using columnar storage and analytical optimization. } > Accept and process ANS-104 data item uploads with multiple payment methods and access control. **Ready to enhance your gateway?** Click any sidecar above to get started with detailed setup guides. ## Explore More } title="Monitor your gateway with Grafana" description="Set up comprehensive monitoring and analytics for your gateway infrastructure" href="/build/extensions/grafana" /> } title="Performance Optimization" description="Optimize your gateway for large datasets and high-performance queries" href="/build/run-a-gateway/manage/filters" /> } title="Gateway Operations" description="Learn advanced gateway management, troubleshooting, and configuration" href="/build/run-a-gateway/manage" /> } title="Developer SDKs" description="Integrate ar.io services into your applications with our SDKs" href="/sdks" /> # Application Distribution with ArNS + Manifests (/build/guides/application-distribution) **Overview:** This guide shows how to distribute software permanently using Arweave manifests and ArNS. You can publish binaries once and serve them from stable, human-readable URLs that keep working as releases change. Files are stored immutably on Arweave, old versions remain available, and links never break. **What you get:** - Permanent storage for all releases - Stable URLs via ArNS (for example, `releases_yourapp.ar.io`) - Simple versioned paths for latest and pinned releases - Automated CI/CD with GitHub Actions and Turbo - No centralized registries or CDNs The case study that follows walks through a real implementation for the Harlequin CLI and shows how to apply this pattern to your own software. ## Architecture Overview ### System Components B[GoReleaser] B --> C[Multi-Platform Binaries] C --> D[Turbo SDK Upload] D --> E[Arweave Storage] E --> F[Manifest Creation] F --> G[ArNS Routing] G --> H[Global Distribution] I[Install Script] --> J[Platform Detection] J --> K[Binary Download] K --> L[Gzip Decompression] L --> M[Installation]`} /> ### Core Technologies 1. **Arweave**: Permanent data storage blockchain 2. **ArNS**: Decentralized naming service for content routing 3. **Turbo SDK**: Efficient data upload and payment handling 4. **GoReleaser**: Multi-platform binary compilation 5. **GitHub Actions**: Automated CI/CD pipeline ## Implementation Deep Dive ### 1. Binary Build Pipeline The system uses GoReleaser for creating optimized, multi-platform binaries: ```yaml # .goreleaser.yaml builds: - id: harlequin main: ./main.go binary: harlequin goos: [linux, darwin, windows] goarch: [amd64, arm64] flags: [-trimpath] ldflags: - -s -w # Strip debug info for smaller binaries - -X main.version={{.Version}} ``` **Platform Matrix:** - Linux: AMD64, ARM64 - macOS: AMD64 (Intel), ARM64 (Apple Silicon) - Windows: AMD64, ARM64 ### 2. Arweave Storage Strategy #### Compression Optimization All binaries are compressed with gzip before upload, achieving significant storage savings: ```typescript // Compress binary before upload const binaryData = readFileSync(binary); const compressedData = gzipSync(binaryData); const compressionRatio = ((1 - compressedData.length / binaryData.length) * 100).toFixed(1); // Upload with proper content headers const upload = await turboClient.upload({ data: compressedData, dataItemOpts: { tags: [ {name: "Content-Type", value: "application/gzip"}, {name: "Content-Encoding", value: "gzip"}, {name: "Original-Content-Type", value: "application/octet-stream"}, {name: "Original-Size", value: binaryData.length.toString()}, {name: "Compressed-Size", value: compressedData.length.toString()} ] } }); ``` **Storage Efficiency Results:** - Average compression ratio: ~70% - 10MB binary → ~3MB storage cost - Significant cost savings at scale #### Data Item Tagging Each upload includes comprehensive metadata for discoverability and management: ```typescript const dataItemOptions = { tags: [ {name: 'Type', value: 'release'}, {name: 'App-Name', value: 'Harlequin-CLI'}, {name: 'App-Version', value: version}, {name: 'Content-Type', value: 'application/gzip'}, {name: 'Platform', value: platform}, {name: 'Architecture', value: arch} ] } ``` ### 3. Manifest-Based Routing The system creates an Arweave manifest that provides structured routing for all binaries and metadata: ```typescript const manifest: ArweaveManifest = { manifest: 'arweave/paths', version: '0.1.0', index: { path: 'install_cli.sh' // Default route }, paths: { // Version-specific binaries 'releases/1.2.3/linux/amd64': { id: 'arweave_tx_id_1' }, 'releases/1.2.3/darwin/arm64': { id: 'arweave_tx_id_2' }, // Latest symlinks for convenience 'releases/latest/linux/amd64': { id: 'arweave_tx_id_1' }, 'releases/latest/darwin/arm64': { id: 'arweave_tx_id_2' }, // API endpoints 'releases': { id: 'releases_json_tx_id' }, 'install_cli.sh': { id: 'install_script_tx_id' } } }; ``` ### 4. ArNS URL Structure Combining ArNS with manifests creates a permanent, human-readable API for your releases. The ArNS undername points to the manifest, and the manifest paths define the URL structure: ```typescript // Update ArNS record to point to new manifest await ant.setRecord({ undername: 'install_cli', // Subdomain transactionId: manifestId, // New manifest TX ID ttlSeconds: 60 // Cache TTL }); ``` This creates a complete URL API: ``` https://install_cli_harlequin.ar.io/ ├── install_cli.sh # Installation script (default) ├── releases # JSON API with version metadata └── releases/ ├── 1.2.3/ # Version-specific binaries │ ├── linux/amd64 │ ├── darwin/arm64 │ └── windows/amd64 └── latest/ # Latest version aliases ├── linux/amd64 └── darwin/arm64 ``` **Key Benefits:** - `install_cli_harlequin.ar.io` provides a permanent, friendly URL - Manifest paths create a logical REST-like API structure - Updating the ArNS record points to new releases while preserving the URL ### 5. Intelligent Installation Script The installation script provides a sophisticated user experience with platform detection, version management, and error handling: #### Platform Detection ```bash # Detect OS and architecture OS="$(uname -s)" ARCH="$(uname -m)" case $OS in Darwin) PLATFORM="darwin" ;; Linux) PLATFORM="linux" ;; CYGWIN*|MINGW*|MSYS*) PLATFORM="windows" ;; esac case $ARCH in x86_64) ARCH="amd64" ;; aarch64|arm64) ARCH="arm64" ;; armv7*) ARCH="armv7" ;; esac ``` #### Version Management ```bash # Interactive version selection get_available_versions() { curl -fsSL "${BASE_URL}/releases" -o /tmp/releases.json if has_jq; then jq -r '.[].tag_name' /tmp/releases.json | sed 's/^v//' else parse_version "$TEMP_FILE" ``` ## CI/CD Integration ### GitHub Actions Workflow The deployment is fully automated through GitHub Actions: ```yaml name: Nx Release on: push: branches: [main, develop] paths: ['cli/**'] jobs: release: runs-on: ubuntu-latest steps: - name: Setup Go & GoReleaser uses: goreleaser/goreleaser-action@v6.4.0 - name: Release (Main - Stable) if: github.ref == 'refs/heads/main' run: npx nx release patch --yes env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ARWEAVE_WALLET_JWK: ${{ secrets.ARWEAVE_WALLET_JWK }} - name: Test Installation run: | curl -fsSL https://install_cli_harlequin.daemongate.io | \ DRYRUN=true VERSION=0.1.0 sh ``` ### Nx Integration The system leverages Nx for orchestrating the complex release pipeline: ```json { "nx-release-publish": { "executor": "nx:run-commands", "options": { "command": "goreleaser release --clean && cd scripts && yarn deploy" }, "configurations": { "dry-run": { "command": "goreleaser release --skip=publish && yarn deploy:dryrun" } } } } ``` **Pipeline Stages:** 1. **Build**: GoReleaser creates multi-platform binaries 2. **Upload**: Turbo SDK uploads compressed binaries to Arweave 3. **Manifest**: Creates routing manifest with all binary paths 4. **ArNS**: Updates domain to point to new manifest 5. **Verification**: Tests installation script functionality ## Security & Reliability ### Decentralization Benefits - **No Single Point of Failure**: Distributed across the entire ar.io network - **Censorship Resistance**: No central authority can block access - **Geographic Redundancy**: Data replicated globally ### Access Control ```typescript // Wallet-based deployment authorization const signer = new ArweaveSigner(wallet); const turboUploader = TurboFactory.authenticated({signer}); // Only authorized wallet can update ArNS records await ant.setRecord({ undername: 'install_cli', transactionId: manifestId }); ``` ## Conclusion This implementation demonstrates that Arweave manifests combined with ArNS provide a powerful alternative to traditional application distribution: - **Automated Release Pipeline**: CI/CD integration enables seamless multi-platform builds and deployment - **Decentralized and Permanent Storage**: Applications are stored immutably across the Arweave network - **Zero Reliance on Centralized Registries**: No dependency on npm, GitHub Releases, or traditional CDNs - **Permanent Friendly Names**: ArNS provides human-readable URLs with unbreakable links to your releases ## Next Steps } > Learn the fundamentals of Arweave manifest structure and creation. } > Register and manage ArNS names for your distribution endpoints. } > Explore advanced Turbo SDK features for optimized uploads. } > Understand security considerations for permanent data. # ArNS Marketplace (/build/guides/arns-marketplace) **ArNS tokens** can be traded and sold on standard Solana NFT marketplaces. ANTs (Ar.io Name Tokens) are Metaplex Core NFTs, making them tradeable on **Tensor**, **Magic Eden**, and other compatible platforms. ## Trading ANTs on NFT Marketplaces ANTs are standard Metaplex Core NFTs on Solana. This means they are natively supported by the Solana NFT ecosystem: - **[Tensor](https://tensor.trade)** — Solana's leading NFT marketplace with advanced trading features - **[Magic Eden](https://magiceden.io)** — Popular cross-chain NFT marketplace with Solana support - **Direct transfers** — ANTs can be sent directly between Solana wallets ### Lazy Reconciliation When an ANT is sold on a marketplace (outside the ar.io protocol), the existing controllers are not immediately cleared. Instead, **lazy reconciliation** occurs on the next write operation — controllers are cleared automatically, ensuring the new owner gets clean control of the name. ## What Are ANTs? **Ar.io Name Tokens (ANTs)** are: - **Metaplex Core NFTs** — Standard Solana NFTs with onchain state in PDAs - **Transferable assets** — Can be bought, sold, and traded on any compatible marketplace - **Digital assets** — Represent ownership and control of ArNS domains - **DNS-like routing** — Each ANT stores records pointing to Arweave transaction IDs ## How Trading Works ### 1. Token Ownership **When you own an ANT:** - You control the domain name and its records - You can update where it points (Arweave TX IDs or IPFS CIDs) - You can transfer ownership via direct transfer or marketplace sale - You can list it on Tensor, Magic Eden, or other Solana NFT marketplaces - You can add up to 10 controllers who can manage records on your behalf ### 2. Marketplace Trading **Trading mechanisms:** - **Marketplace listings** — List your ANT on Tensor or Magic Eden with a fixed price or auction - **Direct transfers** — Send the NFT directly to another Solana wallet - **Collection offers** — Accept collection-wide offers from buyers - **Instant sales** — Sell into existing bids on marketplaces ### 3. Name Characteristics **What makes ANTs desirable:** - **Domain length** - Shorter names are more memorable - **Memorability** - Easy-to-remember names are more useful - **Brand potential** - Names that could become recognizable - **Uniqueness** - Creative and distinctive names - **Content attached** - Domains with established content ## Potential Trading Examples ### Popular Domain Types **Short names:** - `ar://ai` - Single letter domains - `ar://web3` - Industry keywords - `ar://nft` - Popular terms **Brandable names:** - `ar://crypto` - Industry terms - `ar://decentralized` - Descriptive names - `ar://permanent` - Arweave-related terms ### Potential Use Cases **Personal branding** - Use memorable names for your identity **Project organization** - Create names for different projects **Content management** - Organize content under specific names **Community building** - Create recognizable names for communities ## Getting Started ### 1. Acquire ANTs **Ways to get ANTs:** - **Register new domains** - Create your own primary names - **Buy from others** - Purchase existing domains - **Participate in auctions** - Bid on available names - **Trade with others** - Exchange domains you own ### 2. Choose Names **Consider these factors:** - **Domain length** - Shorter names are more memorable - **Memorability** - Easy to remember and type - **Brand potential** - Could become recognizable - **Current content** - What's already attached to the domain - **Personal preference** - What fits your needs and style ### 3. Trade Safely **Best practices:** - **Verify ownership** - Confirm the seller owns the domain - **Check domain status** - Ensure it's not expired or locked - **Use escrow services** - Protect both buyer and seller - **Document transfers** - Keep records of all transactions ## Benefits - **Transferable ownership** - Move domains between wallets - **Creative expression** - Own and manage creative domain names - **Community participation** - Engage with the ArNS ecosystem - **Content organization** - Structure your permanent web presence - **Identity management** - Use names for personal or project identity ## Ready to Trade? } > Learn about ArNS Primary Names for domain creation. } > Learn how to deploy permanent apps and websites. } > Explore the ArNS documentation for advanced features. # Storing DePIN Data on Arweave Using Turbo (/build/guides/depin) DePIN networks require **scalable and cost-effective storage solutions** they can trust. With vast amounts of data generated by decentralized physical infrastructure networks, traditional onchain storage is prohibitively expensive, yet networks need reliable, long-term access to their device data. Arweave via ar.io provides **chain-agnostic, permanent and immutable storage** for a one-time fee, ensuring networks can access any device data previously stored and verify it has not been tampered with. ## Getting Started with DePIN Data Storage ### Prepare Your Data Structure Organize your DePIN device data in a consistent format. Here's an example for environmental sensor data: ```json { "device_id": "airmon-007", "timestamp": "2025-09-22T14:31:05Z", "location": { "lat": 51.5098, "lon": -0.118 }, "pm25": 16, "co2_ppm": 412, "noise_dB": 41.2 } ``` **Best Practices:** - Use consistent field names across all devices - Include timestamps in ISO format - Add device identifiers for tracking - Consider data compression for large datasets ### Tag Your Data for Discovery Proper tagging is essential for [finding your data](/build/access/find-data) later. Consider these tags for DePIN data: ```json { "name": "App-Name", "value": "AirQuality-DePIN-v1.0" }, { "name": "Device-ID", "value": "airmon-007" }, { "name": "Device-Type", "value": "Environmental-Sensor" }, { "name": "Network-Name", "value": "AirQuality-Network" }, { "name": "Data-Category", "value": "Air-Quality" }, { "name": "Location", "value": "London-UK" }, { "name": "Device-Timestamp", "value": "2025-09-22T14:31:05Z" } ``` **Tagging Strategy:** - Use consistent naming conventions - Include geographic identifiers - Add device type classifications - Include data categories for filtering For more detailed information on tagging, see our [Tagging documentation](/build/upload/tagging) ### Upload to Arweave Select the best method for your DePIN network's needs: ```typescript // Initialize with your wallet const turbo = await TurboFactory.authenticated({ privateKey: jwk, // Your Arweave wallet token: 'arweave' }) // Upload device data const result = await turbo.upload({ data: JSON.stringify(deviceData), dataItemOpts: { tags: [ { name: "Content-Type", value: "application/json" }, { name: "App-Name", value: "AirQuality-DePIN-v1.0" }, { name: "Device-ID", value: "airmon-007" }, { name: "Device-Type", value: "Environmental-Sensor" }, { name: "Network-Name", value: "AirQuality-Network" }, { name: "Data-Category", value: "Air-Quality" }, { name: "Location", value: "London-UK" }, { name: "Device-Timestamp", value: "2025-09-22T14:31:05Z" } ] } }) ``` ```bash # Install Turbo CLI npm install -g @ardrive/turbo-sdk # Upload a single file turbo upload-file --file-path sensor-data.json \ --tag "Content-Type:application/json" \ --tag "App-Name:AirQuality-DePIN-v1.0" \ --tag "Device-Type:Environmental-Sensor" \ --tag "Network-Name:AirQuality-Network" \ --tag "Data-Category:Air-Quality" \ --tag "Location:London-UK" \ --tag "Device-Timestamp:2025-09-22T14:31:05Z" # Upload entire folder turbo upload-folder --folder-path ./sensor-data \ --tag "App-Name:AirQuality-DePIN-v1.0" \ --tag "Network-Name:AirQuality-Network" \ --tag "Data-Category:Air-Quality" \ --index-file index.json ``` For more advanced uploading options, see our [Advanced Uploading with Turbo](/build/upload/advanced-uploading-with-turbo) guide, or the [Turbo SDK documentation](/sdks/turbo-sdk) directly. ## Querying Your DePIN Data ### Find Your Data Use GraphQL to search for your DePIN data by tags and criteria: ```graphql # Find all data for a specific device, most recent results first query { transactions( tags: [ { name: "App-Name", values: ["AirQuality-DePIN-v1.0"] } { name: "Device-ID", values: ["airmon-007"] } ] first: 100 sort: HEIGHT_DESC ) { edges { node { id tags { name value } data { size } } } } } ``` ```graphql # Find data by location query { transactions( tags: [ { name: "App-Name", values: ["AirQuality-DePIN-v1.0"] } { name: "Location", values: ["London-UK"] } ] first: 50 ) { edges { node { id tags { name value } } } } } ``` For more advanced querying options, see our [Find Your Data](/build/access/find-data) documentation. ### Access and Use Your Data Once you have transaction IDs from your queries, choose how to fetch and process the data: **Direct data fetching:** ```javascript // Example: Process air quality data async function processAirQualityData(transactionIds) { const results = [] for (const txId of transactionIds) { const response = await fetch(`https://turbo-gateway.com/${txId}`) const data = await response.json() // Process the data const processed = { Device_ID: data.device_id, timestamp: data.timestamp, location: data.location, pm25: data.pm25, co2_ppm: data.co2_ppm, noise_dB: data.noise_dB } results.push(processed) } return results } ``` For more information on fetching data, see our [Fetch Data](/build/access/fetch-data) documentation. **Verified data with optimized routing:** ```javascript import { createWayfinderClient, PreferredWithFallbackRoutingStrategy, FastestPingRoutingStrategy, HashVerificationStrategy } from "@ar.io/wayfinder-core"; const wayfinder = createWayfinderClient({ ario: ARIO.mainnet(), routingStrategy: new PreferredWithFallbackRoutingStrategy({ preferredGateway: 'https://your-gateway.com', fallbackStrategy: new FastestPingRoutingStrategy({ timeoutMs: 500 }), }), verificationStrategy: new HashVerificationStrategy({ trustedGateways: ['https://turbo-gateway.com'], }), telemetrySettings: { enabled: true, clientName: 'AirQuality-DePIN-v1.0', }, }); // Fetch and verify data using ar:// protocol async function processVerifiedAirQualityData(transactionIds) { const results = [] for (const txId of transactionIds) { const response = await wayfinder.request(`ar://${txId}`) const data = await response.json() // Process the verified data const processed = { Device_ID: data.device_id, timestamp: data.timestamp, location: data.location, pm25: data.pm25, co2_ppm: data.co2_ppm, noise_dB: data.noise_dB, noise_level: data.noise_dB, verified: true // Data is cryptographically verified } results.push(processed) } return results } ``` Learn more about data verification with [Wayfinder](/build/access/wayfinder). ## Next Steps In production, teams have several options to take this further to provide significantly more value to the network and its users including: } > Pay in different Tokens and organise device data files with folders or manifests. }> Operate a gateway optimised to index and serve your device data fast. }> Create mutable data structures for permanent device data and decentralised apps. These approaches can make your DePIN data even more resilient and useful. See more detailed guides about this below and or join our discord to find out more. ## Need Help? If you're interested in exploring these advanced features for your DePIN network, join our [Discord community](https://discord.com/invite/HGG52EtTc2) or reach out to our team. # Encrypted Data Storage with Nillion Blind Computing (/build/guides/encrypted-data-nillion) Store **encrypted data permanently** on Arweave while controlling access through **Nillion's nilDB private storage**. This approach ensures your data remains encrypted on Arweave, decryption keys are securely stored in **nilDB** (Private Storage), and only authorized users can access and decrypt the data. Arweave provides **permanent, immutable storage** for encrypted data, while Nillion's **nilDB** stores decryption keys encrypted and split across multiple nodes. Users can encrypt files using their private keys stored in nilDB, upload the encrypted data to Arweave, and later retrieve and decrypt the data using the same private key—all while maintaining complete control over access permissions. ## Prerequisites Before starting, you'll need to complete Nillion's setup process: ### Create Builder and Get Tokens **Create a Test Builder:** - Visit [Nillion Subscription Portal](https://subscription.nillion.com) - Create a testnet public/private key pair for network access - Use two distinct keys: one for network access, one for subscription payments **Get Testnet NIL Tokens:** - Visit [NIL Faucet](https://faucet.testnet.nillion.com/) - Fund your account with testnet NIL tokens ### Subscribe to nilDB Service **Activate nilDB Subscription:** - Use your subscription wallet to pay for nilDB service - Save your private key in hex format for authentication - You'll need this key to access Nillion's Private Storage services ### Set Up Development Environment **Create a New Project:** ```bash mkdir nillion-arweave-demo cd nillion-arweave-demo npm init -y ``` **Install Required Dependencies:** ```bash npm install @nillion/secretvaults @nillion/nuc @nillion/blindfold @ardrive/turbo-sdk arweave dotenv @noble/curves consola npm install --save-dev typescript @types/node ts-node @tsconfig/node20 ``` **Set Up TypeScript:** ```bash npx tsc --init ``` **Update tsconfig.json:** ```json { "extends": "@tsconfig/node20/tsconfig.json", "compilerOptions": { "outDir": "dist/src", "baseUrl": ".", "module": "esnext", "moduleResolution": "bundler", "allowUnusedLabels": false, "allowImportingTsExtensions": true, "noEmit": true, "allowUnreachableCode": false, "noFallthroughCasesInSwitch": true, "noUncheckedIndexedAccess": true, "noUnusedLocals": true, "noUnusedParameters": true, "forceConsistentCasingInFileNames": true, "declaration": true, "sourceMap": true, "resolveJsonModule": true }, "include": ["src"], "exclude": ["dist", "bundle", "node_modules"] } ``` **Configure Environment Variables:** Create a `.env` file with your configuration: ``` NIL_BUILDER_PRIVATE_KEY=your_hex_private_key_here NIL_BUILDER_COLLECTION_ID=your_collection_id_here # If it exists, one will be created otherwise NILCHAIN_URL=http://rpc.testnet.nilchain-rpc-proxy.nilogy.xyz NILAUTH_URL=https://nilauth.sandbox.app-cluster.sandbox.nilogy.xyz NILDB_NODES=https://nildb-stg-n1.nillion.network,https://nildb-stg-n2.nillion.network,https://nildb-stg-n3.nillion.network ``` ## Understanding the Implementation Let's build the application step by step, explaining what each part does: ### Set Up Project Structure and Imports First, create the main application file `src/index.ts` and understand the imports: ```typescript import "dotenv/config"; ``` **What these imports do:** - **Nillion SDKs**: `@nillion/secretvaults` and `@nillion/nuc` for private storage and access control - **Arweave**: `arweave` and `@ardrive/turbo-sdk` for permanent data storage - **Crypto**: Node.js built-in crypto functions for AES-256-GCM encryption - **Utilities**: Helper functions for key conversion and UUID generation ### Define Configuration and Types Set up the application configuration with proper TypeScript types: ```typescript NILCHAIN_URL: string; NILAUTH_URL: string; NILDB_NODES: string[]; NIL_PAYER_PRIVATE_KEY: string; NIL_BUILDER_PRIVATE_KEY: string; NIL_BUILDER_COLLECTION_ID: string; }; const config: AppConfig = { NILCHAIN_URL: process.env.NILCHAIN_URL || "http://rpc.testnet.nilchain-rpc-proxy.nilogy.xyz", NILAUTH_URL: process.env.NILAUTH_URL || "https://nilauth.sandbox.app-cluster.sandbox.nilogy.xyz", NILDB_NODES: process.env.NILDB_NODES ? process.env.NILDB_NODES.split(",") : [ "https://nildb-stg-n1.nillion.network", "https://nildb-stg-n2.nillion.network", "https://nildb-stg-n3.nillion.network", ], NIL_BUILDER_PRIVATE_KEY: process.env.NIL_BUILDER_PRIVATE_KEY, NIL_BUILDER_COLLECTION_ID: process.env.NIL_BUILDER_COLLECTION_ID, }; ``` **What this does:** - **Type safety**: Defines the structure of our configuration object - **Environment variables**: Loads settings from `.env` file with fallback defaults - **Network endpoints**: Points to Nillion testnet infrastructure - **Multiple nodes**: Uses 3 nilDB nodes for redundancy and security ### Create Encryption and Decryption Functions Implement AES-256-GCM encryption using the user's private key: ```typescript // Utility functions const downloadFile = async (url, location) => { try { const response = await fetch(url); const data = await response.text(); await fs.promises.writeFile(location, data); console.log(`💾 Downloaded file saved to: ${location}`); } catch (error) { console.error("❌ Download file failed:", error); } }; const encryptContent = (content, encryptionKey) => { // Convert hex private key to buffer const privateKeyBuffer = Buffer.from(encryptionKey, "hex"); // Derive a 32-byte encryption key from your private key using SHA-256 const derivedKey = createHash("sha256").update(privateKeyBuffer).digest(); // Validate derived key length (should be 32 bytes for AES-256) if (derivedKey.length !== 32) { throw new Error("Derived encryption key must be exactly 32 bytes for AES-256"); } // Generate secure encryption parameters const iv = randomBytes(16); // Encrypt using AES-256-GCM const cipher = createCipheriv("aes-256-gcm", derivedKey, iv); let encrypted = cipher.update(content); encrypted = Buffer.concat([encrypted, cipher.final()]); // Get the authentication tag (16 bytes for GCM) const authTag = cipher.getAuthTag(); // Combine IV + authTag + encrypted data into a single buffer return Buffer.concat([iv, authTag, encrypted]); }; ``` **What this does:** - **Key derivation**: Uses SHA-256 to create a 32-byte encryption key from the user's private key - **AES-256-GCM**: Provides both confidentiality and authenticity - **Random IV**: Each encryption uses a unique initialization vector - **Auth tag**: Prevents tampering with the encrypted data - **Combined format**: Stores IV, auth tag, and encrypted data together ### Create Decryption Function Implement the corresponding decryption function: ```typescript const decryptContent = (encryptedData, encryptionKey, outputLocation) => { // Convert hex private key to buffer const privateKeyBuffer = Buffer.from(encryptionKey, "hex"); // Derive the same 32-byte encryption key using SHA-256 const derivedKey = createHash("sha256").update(privateKeyBuffer).digest(); // Validate derived key length if (derivedKey.length !== 32) { throw new Error("Derived encryption key must be exactly 32 bytes for AES-256"); } // Extract IV, auth tag, and encrypted content if (encryptedData.length ### Create Token Generation Function Implement delegation token creation for secure access control: ```typescript const generateToken = async (parentToken, command, audience, tokenExpirySeconds, privateKey, body) => { const token = NucTokenBuilder.extending(parentToken) .command(command) .audience(audience) .expiresAt(Math.floor(Date.now() / 1000) + tokenExpirySeconds); if (body) token.body(body); return token.build(privateKey); }; ``` **What this does:** - **Delegation tokens**: Creates time-limited access tokens for users - **Command-based**: Specifies exactly what operations the token allows - **Audience-specific**: Only works for the intended user - **Time-limited**: Automatically expires after the specified duration - **Cryptographically signed**: Uses the builder's private key for authenticity ### Create Utility Files Create supporting utility files for Arweave operations and logging: **Create `src/arweave.ts`:** ```typescript const ARWEAVE_HOST = "turbo-gateway.com"; try { const arweave = Arweave.init({ host: ARWEAVE_HOST, port: 1984, protocol: 'http', timeout: 20000, logging: true, }); // Generate a new wallet const wallet = await arweave.wallets.generate(); // Get the wallet address const address = await arweave.wallets.jwkToAddress(wallet); return { wallet, address }; } catch (error) { logger.error('Error creating wallet:', error); } } try { const signer = new ArweaveSigner(wallet); const turbo = TurboFactory.authenticated({ signer }); const result = await turbo.upload({ data, dataItemOpts: { tags: [ { name: "Content-Type", value: "text/plain" }, { name: "Title", value: "My demo file" }, ], }, }); return result; } catch (error) { logger.error('Error uploading securely encrypted file:', error); } } ``` **Create `src/logger.ts`:** ```typescript ``` **What these do:** - **Arweave utilities**: Handle wallet creation and file uploads using Turbo - **Consola logging**: Provides a beautiful, consistent logging experience with colors and formatting - **Error handling**: Gracefully handle and log errors ### Implement the Main Workflow Now let's implement the main application logic that ties everything together: ```typescript async function main() { try { let collectionId; const dataId = randomUUID(); // Builder Client const builderKeypair = Keypair.from(config.NIL_BUILDER_PRIVATE_KEY); const builder = await SecretVaultBuilderClient.from({ keypair: builderKeypair, urls: { chain: config.NILCHAIN_URL, auth: config.NILAUTH_URL, dbs: config.NILDB_NODES, }, blindfold: { operation: "store" }, }); await builder.refreshRootToken(); // Check if builder is set up correctly const existingProfile = await builder.readProfile(); ``` **What this does:** - **Builder initialization**: Creates a builder client with your private key - **Network connection**: Connects to Nillion testnet infrastructure - **Token refresh**: Gets a fresh authentication token - **Profile validation**: Ensures the builder is properly set up ### Set Up Data Collection Create or validate the collection for storing user private keys: ```typescript // Create the Owned Collection if it doesn't exist if (!config.NIL_BUILDER_COLLECTION_ID) { collectionId = randomUUID(); const schema = { "$schema": "http://json-schema.org/draft-07/schema#", type: "array", items: { type: "object", properties: { _id: { "type": "string", "format": "uuid" }, private_key: { "type": "object", "properties": { "%share": { "type": "string" } }, required: ["%share"] } }, required: ["_id", "private_key"] } }; const collection = { _id: collectionId, type: "owned", name: "Nillion / Arweave Demo App User Profiles", schema, }; const collectionResult = await builder.createCollection(collection); console.log(`✅ Created Owned Collection with ID: ${collectionId}`); console.log(`Make sure to update your .env file with this NIL_BUILDER_COLLECTION_ID to proceed.`); process.exit(0); } else { if (existingProfile.data.collections.indexOf(config.NIL_BUILDER_COLLECTION_ID) === -1) { throw new Error(`Builder does not have collection ${config.NIL_BUILDER_COLLECTION_ID} registered. Please check your .env configuration.`); } else { console.log(`✅ Builder is set up correctly with DID: ${builder.did}`); console.log(`✅ Using Owned Collection ID: ${config.NIL_BUILDER_COLLECTION_ID}`); collectionId = config.NIL_BUILDER_COLLECTION_ID; } } ``` **What this does:** - **Collection creation**: Creates a new collection if none exists - **Schema definition**: Defines the structure for storing user private keys - **Validation**: Ensures the builder has access to the specified collection - **Environment setup**: Prompts you to update your `.env` file with the collection ID ### Create User and Store Private Key Generate a new user and securely store their private key in nilDB: ```typescript // Create user and store its private key in nilDB const secretKey = await SecretKey.generate( { nodes: config.NILDB_NODES.map(url => ({ url })) }, { store: true } ); const userKeypair = Keypair.from(bytesToHex(secretKey.material)); const userDid = userKeypair.toDid().toString(); const user = await SecretVaultUserClient.from({ baseUrls: config.NILDB_NODES, keypair: userKeypair, blindfold: { operation: "store" } }); // Grant write access to the user const delegationToken = await generateToken( builder.rootToken, new Command(["nil", "db", "data", "create"]), userKeypair.toDid(), 3600, // 1 hour builder.keypair.privateKey() ); console.log(`🗝️ Delegation token created`); // User creates profile with Private Key await user.createData(delegationToken, { owner: userDid, acl: { grantee: builder.did.toString(), read: false, write: false, execute: true, }, collection: collectionId, data: [ { _id: dataId, private_key: { "%allot": userKeypair.privateKey() }, } ], }); console.log(`✅ User profile created: ${userDid}`); ``` **What this does:** - **Secret key generation**: Creates a cryptographically secure private key - **Keypair creation**: Converts the secret key to a usable keypair - **User client**: Initializes a user client for nilDB operations - **Delegation token**: Grants the user permission to create data - **ACL setup**: Defines access control - builder can execute queries but not read/write - **Data storage**: Stores the user's private key in the nilDB collection ### Encrypt and Upload File Now let's encrypt a file and upload it to Arweave: ```typescript const wallet = await createWallet(); console.log(`💼 Arweave wallet created: ${wallet?.address}`); // Retrieve the user private key from nilDB const retrievedUserKey = await user.readData({ collection: collectionId, document: dataId, }); // Encrypt file contents const fileData = fs.readFileSync("test/demo.txt"); const encrypted = encryptContent(fileData, retrievedUserKey.data.private_key); const upload = await uploadFile(encrypted, wallet?.wallet); console.log(`✅ File uploaded to Arweave with txId: ${upload.id}`); // Download the file and decrypt it const downloadFileName = `./test/encrypted_demo_${Date.now()}.txt`; console.log(`🕒 Downloading file from Arweave`); await downloadFile(`https://turbo-gateway.com/${upload.id}`, downloadFileName); const decrypted = decryptContent(encrypted, retrievedUserKey.data.private_key, `./test/decrypted_demo_${Date.now()}.txt`); ``` **What this does:** - **Arweave wallet**: Creates a new wallet for uploading files - **Key retrieval**: Gets the user's private key from nilDB - **File encryption**: Encrypts the file using the user's private key - **Arweave upload**: Uploads the encrypted file to permanent storage - **File download**: Downloads the file from Arweave - **Decryption**: Decrypts the file using the same private key ### Complete the Application Finish the main function with error handling: ```typescript } catch (error) { console.log(error); console.error("⚠️ Error setting up builder:", JSON.stringify(error, null, 2)); } } main().catch(console.error); ``` **What this does:** - **Error handling**: Catches and logs any errors that occur during execution - **Application startup**: Runs the main function when the script is executed ## Create Test File Before running the demo, create a test file to encrypt: ```bash mkdir test echo "Hello, this is a test file for Nillion + Arweave encryption demo!" > test/demo.txt ``` ## Understanding the Complete Workflow The main application performs the following steps: 1. **Initialize Builder**: Sets up the Nillion Builder client with your private key 2. **Create/Verify Collection**: Creates a new collection or uses an existing one for storing private keys 3. **Generate User**: Creates a new user with a private key stored securely in nilDB 4. **Create Wallet**: Generates an Arweave wallet for uploading data 5. **Encrypt Data**: Encrypts the test file using the user's private key 6. **Upload to Arweave**: Uploads the encrypted data to Arweave using Turbo 7. **Download & Decrypt**: Downloads the encrypted file and decrypts it using the stored private key **Key Security Features:** - Private keys are stored securely in nilDB with proper access control - Data is encrypted using AES-256-GCM with the user's private key - Only authorized users can access the decryption keys - Encrypted data is permanently stored on Arweave ## Running the Complete Demo Add a start script to your `package.json`: ```json { "scripts": { "start": "ts-node src/index.ts", "build": "tsc", "dev": "ts-node --watch src/index.ts" } } ``` Then run the complete workflow: ```bash # Run the complete workflow npm start ``` This will execute the entire process: 1. Initialize the Builder and create a collection 2. Generate a user and store their private key in nilDB 3. Encrypt a sample file using the private key 4. Upload the encrypted file to Arweave 5. Download the encrypted file from Arweave 6. Decrypt the file using the stored private key ## nilCC To securely compute the logic, we could take advantage of [nilCC](https://docs.nillion.com/build/compute/overview), Nillion's Confidential Computing product, that allows you to run application logic inside a TEE. This way, all the interaction could happen inside the confidential environment with no risk of sensitive information leakage. nilCC workloads can be easily triggered via its REST API: ```bash curl --location '{endpoint}/api/v1/workloads/create' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header 'x-api-key: xxx' \ --data '{ "name": "private-stamement-workload", "artifactsVersion": "0.1.2", "dockerCompose": "services:\n private-statements:\n image: my/workload-logic:v0.2\n environment:\n - DELEGATION_TOKENS=${DELEGATION_TOKENS}\n - COLLECTION_ID=${COLLECTION_ID}\n - DOCUMENT_ID=${DOCUMENT_ID}\n - NODE_ENV=production\n build: .\n ports:\n - \"8080:8080\"", "envVars": { "DOCUMENT_ID": "b05917e6-996c-4e90-a49f-b62fa891da1b", "COLLECTION_ID": "ce9b1d1c-8006-4053-a0c8-f46ad711fc26", "DELEGATION_TOKENS": "W3sidXJsIjoiaHR0cHM6Ly9uaWxkYi1zdGctbjEubmlsbGlvbi5uZXR3b3JrIiwidG9rZW4iOiJleUpoYkdjaU9pSkZVekkxTmtzaWZRLmV5SnBjM01pT2lKa2FXUTZibWxzT2pBeU5qTmlZMkpsTjJVeU5UZGhNamhrTmpjMk5EWTVNemc0WlRnd05EWmxNelEwTW1JeU5UVm1Zakk0TWpZME9EbGhOalE0TW1ZMk9ESmhNRFpsWWpreU1TSXNJbUYxWkNJNkltUnBaRHB1YVd3Nk1ESmxNemcwTm1NME5UVmtZbU5sWldZNVpXWm1PR0U0TkRFeU4yTXpZbVV4WWprM01UbGhZekExTkRFMVpXWmlaamN5Tnprd1pqTXhabUU1Wmpnd01qZGhJaXdpYzNWaUlqb2laR2xrT201cGJEb3dNall6WW1OaVpUZGxNalUzWVRJNFpEWTNOalEyT1RNNE9HVTRNRFEyWlRNME5ESmlNalUxWm1JeU9ESTJORGc1WVRZME9ESm1Oamd5WVRBMlpXSTVNakVpTENKbGVIQWlPakUzTlRrME56Y3hPRGtzSW1OdFpDSTZJaTl1YVd3dlpHSXZkWE5sY25NdmNtVmhaQ0lzSW1GeVozTWlPbnQ5TENKdWIyNWpaU0k2SWpRMVltUTFPVGxsTmpSbU4yVXdaVEZoWWpnNFpqVXdPVEV5TURRME5UWTFJbjAuQ05lMXlISFlUT3cyZW5lYnh4Nm56VWJaQnpJTzdKVmhYZTBMQVo4aTI3TmJzR0JHMHdBbVVGV2VKbG9oc0FxNHBXREVacGQyamtQaXNGMFZCakNHM0EiLCJwdWJsaWNLZXkiOiIwMmUzODQ2YzQ1NWRiY2VlZjllZmY4YTg0MTI3YzNiZTFiOTcxOWFjMDU0MTVlZmJmNzI3OTBmMzFmYTlmODAyN2EifSx7InVybCI6Imh0dHBzOi8vbmlsZGItc3RnLW4yLm5pbGxpb24ubmV0d29yayIsInRva2VuIjoiZXlKaGJHY2lPaUpGVXpJMU5rc2lmUS5leUpwYzNNaU9pSmthV1E2Ym1sc09qQXlOak5pWTJKbE4yVXlOVGRoTWpoa05qYzJORFk1TXpnNFpUZ3dORFpsTXpRME1tSXlOVFZtWWpJNE1qWTBPRGxoTmpRNE1tWTJPREpoTURabFlqa3lNU0lzSW1GMVpDSTZJbVJwWkRwdWFXdzZNREkxTnpreVpUazJZVFk0WXpCaU4yVm1OemM1TkRrMk1ETXlOMlJqTlRjd056QTBZelprWkRVMk5XTm1NbU5oWTJZeU1EWmlaR00zTW1RMk1USXpaamt3SWl3aWMzVmlJam9pWkdsa09tNXBiRG93TWpZelltTmlaVGRsTWpVM1lUSTRaRFkzTmpRMk9UTTRPR1U0TURRMlpUTTBOREppTWpVMVptSXlPREkyTkRnNVlUWTBPREptTmpneVlUQTJaV0k1TWpFaUxDSmxlSEFpT2pFM05UazBOemN4T0Rrc0ltTnRaQ0k2SWk5dWFXd3ZaR0l2ZFhObGNuTXZjbVZoWkNJc0ltRnlaM01pT250OUxDSnViMjVqWlNJNklqVmhNMlV4WW1aaU9ETm1OVGN5WkdVeE5tRTBOV0U1TlRsaFpETXpPREZrSW4wLjE5Nk9uVWZYT3ZnNDVnbHNUaThkZ09OWkc2R3E2NXMxVlBFa01La1huVEZCUlFmamV1R1JzTkVHUnJYci1obmp1Z1BIeWlpWmJiT0JzMU9ndkYzS053IiwicHVibGljS2V5IjoiMDI1NzkyZTk2YTY4YzBiN2VmNzc5NDk2MDMyN2RjNTcwNzA0YzZkZDU2NWNmMmNhY2YyMDZiZGM3MmQ2MTIzZjkwIn0seyJ1cmwiOiJodHRwczovL25pbGRiLXN0Zy1uMy5uaWxsaW9uLm5ldHdvcmsiLCJ0b2tlbiI6ImV5SmhiR2NpT2lKRlV6STFOa3NpZlEuZXlKcGMzTWlPaUprYVdRNmJtbHNPakF5TmpOaVkySmxOMlV5TlRkaE1qaGtOamMyTkRZNU16ZzRaVGd3TkRabE16UTBNbUl5TlRWbVlqSTRNalkwT0RsaE5qUTRNbVkyT0RKaE1EWmxZamt5TVNJc0ltRjFaQ0k2SW1ScFpEcHVhV3c2TURNd05EQXdNVFU1TW1NelpESmhOR0ZtTkdaa01EUTVaamMxWVRVMk1qTmxNVEE1TXpsaU16ZGpNemhqWXpZMFl6STJORGd3TVdFMU5UWTNZalE1TTJGaUlpd2ljM1ZpSWpvaVpHbGtPbTVwYkRvd01qWXpZbU5pWlRkbE1qVTNZVEk0WkRZM05qUTJPVE00T0dVNE1EUTJaVE0wTkRKaU1qVTFabUl5T0RJMk5EZzVZVFkwT0RKbU5qZ3lZVEEyWldJNU1qRWlMQ0psZUhBaU9qRTNOVGswTnpjeE9Ea3NJbU50WkNJNklpOXVhV3d2WkdJdmRYTmxjbk12Y21WaFpDSXNJbUZ5WjNNaU9udDlMQ0p1YjI1alpTSTZJak5rTXpKaVlXSmhaV1ZsTkRjNU5qY3dNbUUyWW1aa016RTVNV013TURaa0luMC5qYTlzQWpJRUFJb09FM2ZmcnIwV190aXlLNWZWWVhQbVdNWjIxbHlhb0d3eUZoMzQycnFhNldUSHhkT3dVSHVXdVhRR0FCOEZTQnhRWks4NXFLbGQtUSIsInB1YmxpY0tleSI6IjAzMDQwMDE1OTJjM2QyYTRhZjRmZDA0OWY3NWE1NjIzZTEwOTM5YjM3YzM4Y2M2NGMyNjQ4MDFhNTU2N2I0OTNhYiJ9XQ==" }, "publicContainerName": "my-workload", "publicContainerPort": 8080, "memory": 1024, "cpus": 1, "disk": 10, "gpus": 0, "workloadId": "88384328-3038-4a8e-8d45-bbebf6d748d4", "creditRate": 1, "status": "scheduled", "accountId": "some-account" }' ``` or the [nilCC Workload Manager](https://nilcc.nillion.com/). Full instructions can be found [here](https://docs.nillion.com/build/compute/quickstart) ## Additional Resources For reference and additional examples, you can also check out the official demo repository: **[Nillion Arweave Demo](https://github.com/elmurci/nillion-arweave-demo)** This repository contains the complete working implementation that this guide is based on. ## Next Steps } > Clone and run the complete working example from the official repository. } > Learn advanced uploading techniques for large encrypted datasets and batch processing. } > Deep dive into Nillion's nilDB for secure key storage and management. } > Learn more about Nillion's native access control and delegation tokens. ## Need Help? If you're interested in implementing encrypted data storage with Nillion, join our [Discord community](https://discord.com/invite/HGG52EtTc2) or explore the [Nillion developer resources](https://docs.nillion.com) for detailed implementation guides. # From Build to Permanent App (/build/guides/hosting-decentralised-apps/deploy-permanent-dapp) This guide connects the last mile of shipping a web app: turn a static build into a permanent Turbo upload, then publish it at a human-readable ArNS name. You only need a Solana wallet, and the same URL can follow every release even though each upload remains immutable. If you have deployed to a static host before, the shape is the same: build, upload, point a name. The difference is that the upload is paid once and cannot be taken down, and the name is yours to move. ## How Turbo and ArNS Fit Together ```text Static build folder │ ▼ Turbo uploads files + creates a manifest │ ▼ Permanent manifest transaction ID │ ▼ ArNS record points your name to that ID │ ▼ https://yourname.ar.io ``` - **Turbo** stores the build files and creates an Arweave manifest that routes paths such as `/`, `/assets/app.js`, and `/about/`. - **ArNS** gives that immutable manifest a stable name. On the next release, update the name to the new manifest ID instead of changing the URL you share. - **ar.io gateways** resolve the name and serve the manifest's files. The app is also available directly by its transaction ID, independently of ArNS. [ARIO Deploy](/build/guides/hosting-decentralised-apps/deploying-with-ario-deploy) automates the same upload, manifest, and ArNS update workflow. Continue here if you want to understand and control each SDK operation yourself. ## What You'll Build By the end of this guide, your app will be: - **Permanently stored** on Arweave (can never be deleted or modified) - **Addressable by transaction ID** so every deployed version can still be retrieved - **Accessible** at `https://yourname.ar.io` (and through other ar.io gateways) - **Owned by you** as a Metaplex Core NFT (the ArNS name token) ## Prerequisites - **Node.js 18+** - **A Solana wallet** with SOL and ARIO tokens - **A built web app** (any framework that outputs a static folder — React, Next.js, Vue, Svelte, etc.) ```bash npm install @ar.io/sdk @ardrive/turbo-sdk @solana/kit bs58 ``` Requires `@ar.io/sdk` version 3.23+ for Solana support. ## Step-by-Step ### Build Your App Generate a static build of your web application: ```bash # next.config.js must have: output: 'export' npm run build # Output: ./out/ ``` ```bash npm run build # Output: ./dist/ ``` ```bash # Build your app to a static folder npm run build # Use whatever output directory your framework creates ``` ### Set Up Your Signer Create a shared setup file that both Turbo (for uploads) and the ar.io SDK (for naming) can use: ```typescript // setup.ts // Load your Solana keypair const keypairBytes = new Uint8Array( JSON.parse(fs.readFileSync('./solana-keypair.json', 'utf-8')), ); // For ar.io SDK (ArNS names, records) // For Turbo (file uploads to Arweave) // Turbo uses its own signer format — pass the secret key as base58 privateKey: bs58.encode(keypairBytes.slice(0, 32)), token: 'solana', }); ``` **Security:** Never commit your keypair file to version control. Use a dedicated deployment wallet with only the SOL and ARIO needed for the operation. ### Upload to Arweave via Turbo Upload your build folder. Turbo bundles all files into an Arweave manifest — a single transaction ID that maps to all your app's files: ```typescript // deploy.ts const ARNS_NAME = 'my-cool-app'; // the ArNS name you want const BUILD_DIR = './dist'; // your build output folder // Step 1: Upload the build folder console.log('Uploading to Arweave...'); const uploadResult = await turbo.uploadFolder({ folderPath: BUILD_DIR, dataItemOpts: { tags: [ { name: 'App-Name', value: ARNS_NAME }, { name: 'App-Version', value: '1.0.0' }, ], }, }); const manifestTxId = uploadResult.manifestResponse.id; console.log(`Uploaded! Manifest TX: ${manifestTxId}`); console.log(`Direct access: https://turbo-gateway.com/${manifestTxId}`); ``` Files up to 105 KiB are free to upload via Turbo, within a 10 MiB lifetime allowance per wallet and another per IP range. Larger uploads are paid with SOL from your wallet — no pre-funding needed when using just-in-time payments. ### Register an ArNS Name (If You Don't Have One) Skip this step if you already own the ArNS name. ```typescript // Check if the name is available try { const existing = await ario.getArNSRecord({ name: ARNS_NAME }); console.log(`Name "${ARNS_NAME}" is already registered`); } catch { // Name is available — register it console.log(`Registering "${ARNS_NAME}"...`); // Check the cost first const cost = await ario.getTokenCost({ intent: 'Buy-Name', name: ARNS_NAME, type: 'lease', years: 1, }); console.log(`Cost: ${cost / 1_000_000} ARIO`); // Buy it (this mints an ANT as a Metaplex Core NFT) await ario.buyRecord({ name: ARNS_NAME, type: 'lease', years: 1, }); console.log(`Registered "${ARNS_NAME}"!`); } ``` ### Point Your Name to Your App Set the ANT's root (@) record to your uploaded manifest: ```typescript // Get the ANT mint address from the ArNS record const record = await ario.getArNSRecord({ name: ARNS_NAME }); // Initialize the ANT const ant = ANT.init({ signer, processId: record.processId }); // Set the root record to your manifest console.log('Setting ArNS record...'); await ant.setRecord({ undername: '@', transactionId: manifestTxId, ttlSeconds: 3600, }); console.log('Done! Your app is live at:'); console.log(` https://${ARNS_NAME}.ar.io`); console.log(` https://${ARNS_NAME}.turbo-gateway.com`); ``` ### Verify Wait a minute for gateways to pick up the new record, then verify: ```bash curl -I https://my-cool-app.ar.io # Should return 200 OK with your app's index.html ``` Verify both layers so it is easier to diagnose a problem: ```bash # Storage and manifest routing curl -I https://turbo-gateway.com/YOUR_MANIFEST_TX_ID # Name resolution curl -I https://my-cool-app.ar.io ``` If the transaction URL works but the ArNS URL does not, confirm that the root (`@`) record contains the manifest transaction ID and allow for the record's TTL. If neither URL works, check the upload result and make sure the build folder contained an `index.html` file. Your app is now permanently hosted and accessible through the ar.io gateway network. ## Updating Your App Since Arweave data is immutable, "updating" means uploading a new version and updating your ArNS record to point to it: ```typescript // Upload new version const newUpload = await turbo.uploadFolder({ folderPath: './dist', dataItemOpts: { tags: [ { name: 'App-Name', value: 'my-cool-app' }, { name: 'App-Version', value: '2.0.0' }, ], }, }); // Update the record const ant = ANT.init({ signer, processId: record.processId }); await ant.setRecord({ undername: '@', transactionId: newUpload.manifestResponse.id, ttlSeconds: 3600, }); // Old version is still on Arweave forever — instant rollback if needed ``` ## Using Undernames for Staging You can use undernames to deploy staging environments alongside production: ```typescript // Deploy staging version await ant.setRecord({ undername: 'staging', transactionId: stagingManifestTxId, ttlSeconds: 300, // short TTL for faster updates }); // Access at: https://staging_my-cool-app.ar.io // When ready, promote to production await ant.setRecord({ undername: '@', transactionId: stagingManifestTxId, ttlSeconds: 3600, }); ``` ## Full Script Here's the complete deployment script you can adapt: ```typescript const ARNS_NAME = process.argv[2] || 'my-app'; const BUILD_DIR = process.argv[3] || './dist'; async function deploy() { // Load Solana keypair const keypairBytes = new Uint8Array( JSON.parse(fs.readFileSync('./solana-keypair.json', 'utf-8')), ); // Init SDKs const signer = await createKeyPairSignerFromBytes(keypairBytes); const ario = ARIO.mainnet({ signer }); const turbo = TurboFactory.authenticated({ privateKey: bs58.encode(keypairBytes.slice(0, 32)), token: 'solana', }); // Upload console.log(`Uploading ${BUILD_DIR}...`); const upload = await turbo.uploadFolder({ folderPath: BUILD_DIR, dataItemOpts: { tags: [{ name: 'App-Name', value: ARNS_NAME }], }, }); const txId = upload.manifestResponse.id; console.log(`Uploaded: https://turbo-gateway.com/${txId}`); // Get ANT and update record const arnsRecord = await ario.getArNSRecord({ name: ARNS_NAME }); const ant = ANT.init({ signer, processId: arnsRecord.processId }); console.log('Updating ArNS record...'); await ant.setRecord({ undername: '@', transactionId: txId, ttlSeconds: 3600, }); console.log(`\nLive at: https://${ARNS_NAME}.ar.io`); } deploy().catch(console.error); ``` ```bash # Usage npx tsx deploy.ts my-cool-app ./dist ``` ## Next Steps } /> } /> } /> # Deploying with ARIO Deploy (/build/guides/hosting-decentralised-apps/deploying-with-ario-deploy) import { Terminal, Rocket, Wallet, Zap, GitBranch, BookOpen, } from 'lucide-react'; [ARIO Deploy](https://github.com/ar-io/ar-io-deploy) is the recommended CLI tool for hosting decentralised applications on ar.io. It streamlines the entire deployment process by uploading your build folder to Arweave, creating Arweave manifests, and automatically updating your ArNS records in a single command. Built on the Turbo SDK, ario-deploy offers flexible payment options including pre-funded Turbo Credits or on-demand topups using ARIO or Base-ETH tokens. It works seamlessly with both Arweave and EVM wallets, making it easy to integrate permanent hosting into your existing development workflow. Check out the [series introduction](/build/guides/hosting-decentralised-apps) to learn about permanent hosting and ArNS domains. ## Prerequisites Before starting, ensure you have: - **Node.js 18+** - Download from [nodejs.org](https://nodejs.org/) - **Upload Wallet** - An Arweave JWK or EVM private key to pay for the upload - **Solana Wallet** - A base58 Solana secret key that controls your ArNS name (for ArNS record updates) - **ArNS Name** - Register one at [arns.ar.io](https://arns.ar.io) - **Command Line Familiarity** - Basic terminal/shell knowledge ARIO Deploy uses two separate keys: - **`DEPLOY_KEY`** — pays for the Arweave upload (Arweave, Ethereum, Polygon, or Solana wallet) - **`ARNS_KEY`** — a Solana key that controls the ArNS name and signs the record update These can be different wallets. The upload key handles payment; the ArNS key handles name ownership. If you only need to upload without updating an ArNS record, you can use the `ario-deploy upload` command with just a `DEPLOY_KEY`. ## Project Setup Let's create a new web application and configure it for deployment. ARIO Deploy works with any framework that generates a static build folder. Initialize a new Next.js application: ```bash title="Terminal" npx create-next-app@latest my-permaweb-app cd my-permaweb-app ``` When prompted, select your preferences. For permanent hosting, **enable static export** by updating `next.config.js`: ```javascript title="next.config.js" /** @type {import('next').NextConfig} */ const nextConfig = { output: 'export', images: { unoptimized: true, }, trailingSlash: true, } module.exports = nextConfig ``` - `output: 'export'` generates a static site in the `out` folder - `images.unoptimized: true` prevents server-side image optimization - `trailingSlash: true` ensures URLs work correctly on static hosting Add ario-deploy as a development dependency: ```bash title="Terminal" npm install --save-dev @ar.io/deploy ``` Update your `package.json` to include deployment commands: ```json title="package.json" { "scripts": { "dev": "next dev", "build": "next build", "deploy": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out" } } ``` Replace `your-arns-name` with your actual ArNS domain name. Note the `--deploy-folder out` flag is required since Next.js exports to `./out` instead of the default `./dist`. Initialize a new Vite application with React: ```bash title="Terminal" npm create vite@latest my-permaweb-app -- --template react cd my-permaweb-app npm install ``` Update `vite.config.js` to use relative paths for Arweave: ```javascript title="vite.config.js" plugins: [react()], base: './', // Use relative paths for Arweave }) ``` The `base: './'` setting ensures all asset paths are relative, which is required for proper loading on Arweave gateways. Add ario-deploy as a development dependency: ```bash title="Terminal" npm install --save-dev @ar.io/deploy ``` Update your `package.json` to include deployment commands: ```json title="package.json" { "scripts": { "dev": "vite", "build": "vite build", "deploy": "vite build && ario-deploy deploy --arns-name your-arns-name" } } ``` Replace `your-arns-name` with your actual ArNS domain name. ## Deploying from the Command Line For this walkthrough, we'll deploy directly from the command line using inline credentials. For production apps, we recommend using GitHub Actions with secrets (covered later in this guide). Always use a dedicated wallet for deployments to minimize security risks. Never commit wallet files or keys to version control. ARIO Deploy uses [Turbo](https://console.ar.io) to upload files to Arweave. Before deploying, ensure your wallet has sufficient credits. Visit the [Console app](https://console.ar.io/account) and connect your deployment wallet to view your current balance and top-up history. A typical static site (5-10 MB) costs approximately 0.1-0.5 ARIO. Credits are applied instantly and remain in your wallet for future deployments. The `npm run deploy` command we configured earlier will build your app and deploy it to Arweave. You need both the upload key (`DEPLOY_KEY`) and the ArNS authority key (`ARNS_KEY`): For Arweave upload wallets, base64 encode your JWK file. The `ARNS_KEY` is always a base58 Solana secret key: ```bash title="Terminal" DEPLOY_KEY=$(base64 -i wallet.json) ARNS_KEY="your-base58-solana-key" npm run deploy ``` - `DEPLOY_KEY` — your Arweave JWK wallet (base64-encoded), pays for the upload - `ARNS_KEY` — your Solana secret key (base58), signs the ArNS record update For EVM wallets (Ethereum, Polygon, Base), use your raw private key for the upload. The `ARNS_KEY` is always a base58 Solana secret key: ```bash title="Terminal" DEPLOY_KEY="0x1234567890abcdef..." ARNS_KEY="your-base58-solana-key" npm run deploy ``` - `DEPLOY_KEY` — your EVM private key, pays for the upload - `ARNS_KEY` — your Solana secret key (base58), signs the ArNS record update For EVM wallets, ensure your deploy script in `package.json` includes the `--sig-type` flag (e.g., `--sig-type ethereum`). When running from a terminal, ario-deploy will interactively prompt for any missing keys or configuration. You can omit `ARNS_KEY` from the command and the CLI will prompt you for it. ARIO Deploy will: 1. **Upload files** to Arweave via Turbo 2. **Create a manifest** with SPA fallback detection 3. **Update ArNS records** to point to the new transaction Expected output: ``` Starting deployment... ✔ ARIO initialized ✔ ArNS record fetched for your-arns-name ✔ Signer created (arweave) ✔ Turbo initialized ✔ Upload planned: 12 of 48 files to upload (1.2 MiB), 36 cached, 0 duplicates ✔ Turbo credits check passed ✔ Folder uploaded: abc123def456ghi789jkl012mno345pqr678stu901v (36/48 files cached, 12 uploaded) (files cached for future uploads) ✔ ANT record updated Deployment Successful! ┌─────────────┬───────────────────────────────────────────────┐ │ Tx ID │ abc123def456ghi789jkl012mno345pqr678stu901v │ │ ArNS Name │ your-arns-name │ │ Undername │ @ │ │ ArNS URL │ https://your-arns-name.ar.io │ └─────────────┴───────────────────────────────────────────────┘ ``` ArNS updates typically propagate across the gateway network within 60 seconds (the default TTL). You may need to hard refresh your browser to see changes immediately. ## Reducing Upload Costs Arweave storage is paid per byte, once. Three features keep redeploys cheap: - **Deduplication (on by default).** ario-deploy hashes every file and keeps a local cache (`.ario-deploy/transaction-cache.json`) of what it has already uploaded. Unchanged files reuse their existing transaction instead of being uploaded again, and files that are identical to another file in the same deploy are uploaded once. The credit check prices only what will actually be uploaded. - **Incremental uploads (opt-in).** `--incremental` tags each file with its content hash, and later deploys look your wallet's past uploads up on chain. A machine with no local cache, such as a fresh CI runner, still pays only for the files that changed. Uploads take a few minutes (about 5-7 in our measurements) to become findable, so a redeploy seconds after the last one may upload some files again. - **Compression (opt-in).** `--compress gzip` compresses each file before upload and tags it with `Content-Encoding`, so gateways serve it with that header and browsers decompress it transparently. HTML, JavaScript, CSS and JSON typically shrink 5-8x; already-compressed formats (JPEG, PNG, GIF, WebP, AVIF, WOFF/WOFF2, MP3, MP4, WebM, and archives) are uploaded as-is. ```json title="package.json" { "scripts": { "deploy": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --incremental --compress gzip --compress-exclude \"llms*.txt,*.md\"" } } ``` `--compress-exclude` keeps files uncompressed that tools fetch with plain HTTP clients such as `curl`, which do not decode gzip unless asked to. Measured on a 1,229-file static docs site with both flags: a redeploy from a fresh CI runner with no local cache found 1,228 files on chain and uploaded one (2.5 MiB), where the first deploy uploaded 33 MiB. Turning compression or incremental uploads on for the first time uploads each file once more, because those uploads are tracked separately. Right after a deploy, a gateway may serve a file before it has indexed that file's tags. The gateway must still send `Content-Encoding` in that case. The ar.io and Turbo gateways do; other operators' gateways do once they run an ar-io-node release with that fix, and until then may serve the compressed bytes without the header, which renders as garbage. Deploy to a test undername first (`--undername staging`) and load a few pages before pointing your main name at it. ## On-Demand Payment Instead of pre-funding Turbo Credits, you can pay for deployments on-demand. ARIO Deploy will automatically convert tokens to credits as needed. Update your `package.json` to include an on-demand deployment script: For Arweave wallets using ARIO tokens: ```json title="package.json" { "scripts": { "dev": "next dev", "build": "next build", "deploy": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out", "deploy:on-demand": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --on-demand ario --max-token-amount 2.0" } } ``` - `--on-demand ario` enables ARIO payment mode - `--max-token-amount 2.0` sets maximum ARIO to spend (prevents unexpected costs) For EVM wallets using Base-ETH: ```json title="package.json" { "scripts": { "dev": "next dev", "build": "next build", "deploy": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --sig-type ethereum", "deploy:on-demand": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --sig-type ethereum --on-demand base-eth --max-token-amount 0.01" } } ``` - `--sig-type ethereum` required for EVM wallets - `--on-demand base-eth` enables Base Network payment - Wallet must be funded with ETH on Base Network Base-ETH on-demand payment only works with Ethereum signer types. Your wallet must have ETH on the Base Network, not Ethereum mainnet. For Arweave wallets using ARIO tokens: ```json title="package.json" { "scripts": { "dev": "vite", "build": "vite build", "deploy": "vite build && ario-deploy deploy --arns-name your-arns-name", "deploy:on-demand": "vite build && ario-deploy deploy --arns-name your-arns-name --on-demand ario --max-token-amount 2.0" } } ``` - `--on-demand ario` enables ARIO payment mode - `--max-token-amount 2.0` sets maximum ARIO to spend (prevents unexpected costs) For EVM wallets using Base-ETH: ```json title="package.json" { "scripts": { "dev": "vite", "build": "vite build", "deploy": "vite build && ario-deploy deploy --arns-name your-arns-name --sig-type ethereum", "deploy:on-demand": "vite build && ario-deploy deploy --arns-name your-arns-name --sig-type ethereum --on-demand base-eth --max-token-amount 0.01" } } ``` - `--sig-type ethereum` required for EVM wallets - `--on-demand base-eth` enables Base Network payment - Wallet must be funded with ETH on Base Network Base-ETH on-demand payment only works with Ethereum signer types. Your wallet must have ETH on the Base Network, not Ethereum mainnet. See [Base documentation](https://docs.base.org/) for getting testnet or mainnet ETH. Run the on-demand deployment command with both keys: ```bash title="Terminal" DEPLOY_KEY=$(base64 -i wallet.json) ARNS_KEY="your-base58-solana-key" npm run deploy:on-demand ``` The tool will automatically convert ARIO to Turbo Credits as needed for the deployment. ```bash title="Terminal" DEPLOY_KEY="0x1234567890abcdef..." ARNS_KEY="your-base58-solana-key" npm run deploy:on-demand ``` The tool will automatically convert Base-ETH to Turbo Credits as needed for the deployment. The on-demand approach is ideal for: - **Frequent deployments** where pre-funding isn't convenient - **CI/CD pipelines** that need reliable automated deployments - **Multi-team projects** where different wallets handle different apps ## Automating with GitHub Actions The simplest way to automate deployments is with the official [`ar-io/ar-io-deploy`](https://github.com/ar-io/ar-io-deploy) GitHub Action. It handles node setup, dedup caching, and PR preview comments automatically. The action stores the dedupe cache in the GitHub Actions cache, which GitHub evicts after 7 days without use, so an infrequent deploy can start with no cache and re-upload every file. Set `incremental: 'true'` and the action finds your wallet's past uploads on chain instead, so only the files that changed are uploaded, whatever the cache holds. In your GitHub repository, navigate to **Settings** → **Secrets and variables** → **Actions** and add: | Secret | Description | |--------|-------------| | `DEPLOY_KEY` | Upload wallet key. For Arweave: base64-encoded JWK. For EVM: raw private key with `0x` prefix. | | `ARNS_KEY` | Base58-encoded Solana secret key that controls the ArNS name. | | `ARNS_NAME` | Your ArNS domain name (e.g., `myapp`). | Create `.github/workflows/deploy.yml`: ```yaml title=".github/workflows/deploy.yml" name: Deploy to Arweave on: push: branches: [main] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '20' cache: 'npm' - name: Install and build run: | npm ci npm run build - name: Deploy to Arweave uses: ar-io/ar-io-deploy@v1 with: deploy-key: ${{ secrets.DEPLOY_KEY }} arns-key: ${{ secrets.ARNS_KEY }} arns-name: ${{ secrets.ARNS_NAME }} # Optional (see Reducing Upload Costs). incremental needs v1.2.0+ and # compress v1.1.0+; pin that version, since the floating v1 tag may lag. # incremental: 'true' # compress: gzip # compress-exclude: 'llms*.txt,*.md' ``` The action automatically detects and uploads the `./dist` folder. For Next.js projects, add `deploy-folder: ./out`. Create a separate workflow for PR preview deployments at `.github/workflows/pr-preview.yml`: ```yaml title=".github/workflows/pr-preview.yml" name: PR Preview on: pull_request: types: [opened, synchronize, reopened, closed] jobs: preview: runs-on: ubuntu-latest if: github.event.pull_request.head.repo.full_name == github.repository steps: - uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '20' cache: 'npm' - name: Install and build run: | npm ci npm run build - name: Deploy preview uses: ar-io/ar-io-deploy@v1 with: deploy-key: ${{ secrets.DEPLOY_KEY }} arns-key: ${{ secrets.ARNS_KEY }} arns-name: ${{ secrets.ARNS_NAME }} preview: 'true' github-token: ${{ secrets.GITHUB_TOKEN }} ``` When `preview` is enabled, the action: - Auto-generates an undername from the PR number (e.g., `myrepo-pr-42`) - Posts a comment on the PR with the preview URL - Cleans up the undername when the PR is closed Make a commit and push to your main branch: ```bash title="Terminal" git add . git commit -m "Set up automated deployments" git push origin main ``` Check the **Actions** tab in your GitHub repository to monitor the deployment progress. Each push triggers a new deployment. For high-traffic repositories, consider adding `paths` filters to deploy only when specific files change, or use manual workflow triggers. ## Summary You now know how to deploy permanent web applications using ario-deploy: - **Static site setup** for Next.js and React + Vite with proper configuration - **Two-key model** with separate upload (`DEPLOY_KEY`) and ArNS authority (`ARNS_KEY`) keys - **Flexible wallet options** supporting Arweave, EVM, and Solana wallets - **Payment methods** including pre-funded Turbo Credits and on-demand topups with ARIO or Base-ETH - **Lower upload costs** through automatic deduplication, optional incremental uploads, and optional gzip compression - **Command-line deployment** with inline wallet credentials for quick deployments - **GitHub Actions automation** with the official `ar-io/ar-io-deploy` action for production deploys and PR previews For more details, see the [ARIO Deploy GitHub repository](https://github.com/ar-io/ar-io-deploy). In the next guide, you'll learn how to use undernames to manage multiple environments and versions of your application. # One Click Deployments with Arlink (/build/guides/hosting-decentralised-apps/deploying-with-arlink) ![Arlink login page showing GitHub, Wander, and MetaMask authentication options](/content/arlink-homepage.png) ## Introduction In the previous guides in this series, we've used a CLI tool called `ario-deploy` to configure and host decentralised apps on ar.io. In this guide we'll be using [Arlink](https://arlink.ar.io), a visual, web-based platform for hosting decentralised apps on ar.io without needing command-line tools. It offers automated builds, GitHub integration, and seamless ArNS management through an intuitive interface. Check out the [series introduction](/build/guides/hosting-decentralised-apps) to learn how ar.io enables permanent, decentralised hosting with 100+ independent gateways. ## What You'll Learn - **Deploying with Arlink** - Using the visual web interface for deployments - **GitHub integration** - Setting up automated deployments from your repository - **ArNS setup** - Connecting your ArNS name or using free Arlink undernames - **Build monitoring** - Tracking deployment progress with live logs - **When to use Arlink** - Understanding limitations and choosing the right tool ## What is Arlink? Arlink is a web-based deployment platform that simplifies hosting applications on Arweave. It provides a visual interface for developers who prefer graphical tools over command-line interfaces, while maintaining the same permanence and decentralization benefits. **Key Features:** - **Visual Interface** - No command-line knowledge required - **Automated Builds** - Auto-detects build settings and handles the entire build process - **GitHub Integration** - Deploy directly from your repositories with continuous deployment - **Real-Time Monitoring** - Live build logs and progress tracking - **ArNS Management** - Connect existing ArNS names or use free Arlink undernames All applications that can be deployed with `ario-deploy` CLI can also be deployed with Arlink. Choose the tool that fits your workflow: - **Arlink**: Visual workflows, quick deployments, smaller projects (under 10MB) - **ario-deploy**: CI/CD pipelines, large applications, custom automation See [Limitations & Considerations](#limitations--considerations) below to help decide. ## Prerequisites Before deploying with Arlink, you'll need: - **Arweave Wallet** - Create one at [Wander](https://www.wander.app/) and add AR tokens or [Turbo credits](https://console.ar.io/topup) - **GitHub Repository** - Your application code in a GitHub repository with build scripts - **Static Build Output** - Application must build to static files (HTML, CSS, JS) - **Optional: ArNS Name** - Purchase at [arns.ar.io](https://arns.ar.io) or use free Arlink undernames Test your build locally (`npm run build`) before deploying to ensure it produces static output. ## Deployment Methods Arlink offers two main deployment approaches: - **GitHub Deploy** - Connect your repository for automated builds with continuous deployment - **Template Hub** - Start with pre-built templates or add your own at [arlink.ar.io/templates](https://arlink.ar.io/templates) This guide focuses on GitHub deployment, which is the most common approach for custom applications. ## Deploying from GitHub The deployment process consists of four main phases. For detailed step-by-step instructions, see the [Arlink Quickstart Guide](https://arlink.gitbook.io/arlink-docs/getting-started/quickstart). ### 1. Connect & Authorize Navigate to the [Arlink Dashboard](https://arlink.ar.io/) and connect your Arweave wallet (Wander). Then authorize GitHub access to enable repository connections. ![Arlink login page showing GitHub, Wander, and MetaMask authentication options](/content/arlink-login.png) Arlink only requests read access to your repositories and webhook permissions for continuous deployment. ### 2. Configure Deployment Select your GitHub repository and branch. Arlink will automatically detect: - **Package Manager** - npm, yarn, or pnpm - **Framework** - React, Next.js, Vue, Astro, etc. - **Build Command** - Usually `npm run build` - **Output Directory** - `dist/`, `build/`, `out/`, etc. ![Arlink repository selection interface showing GitHub repositories with import buttons](/content/arlink-repo-select.png) Review the auto-detected settings and adjust if needed. Ensure your output directory matches your framework: | Framework | Output Directory | Notes | |-----------|-----------------|-------| | Vite/React | `dist/` | Default configuration | | Next.js | `out/` | Requires `output: 'export'` in config | | Astro | `dist/` | Static by default | | Create React App | `build/` | Default configuration | ![Arlink deploy configuration options](/content/arlink-deploy-config.png) ### 3. Choose Domain Select how your application will be accessible: **Free Arlink Undername:** - Format: `yourname_arlink.ar.io` - No ArNS name purchase required - Available immediately **Existing ArNS Name:** - Use your purchased ArNS name (e.g., `myapp.ar.io`) - Optionally add undernames (e.g., `staging_myapp.ar.io`) - Arlink automatically updates ArNS records {/* Screenshot: Domain selection interface showing Arlink undername and existing ArNS options */} ### 4. Deploy & Monitor Click **Deploy** to start the build process. Arlink will clone your repository, install dependencies, build your application, and upload to Arweave. ![Arlink deployment build monitoring](/content/arlink-deployment-process.png) **Build Timeline:** - Small apps (~1MB): 2-3 minutes - Medium apps (1-5MB): 3-5 minutes - Large apps (5-10MB): 5-10 minutes Arlink enforces a **10MB max build output** and **10-minute build timeout**. For larger applications, use `ario-deploy` CLI instead. Once complete, your application is permanently deployed and accessible via: - Your chosen domain (e.g., `myapp_arlink.ar.io`) - Any ar.io gateway (e.g., `myapp_arlink.turbo-gateway.com`) - Direct transaction ID {/* Screenshot: Successful deployment screen showing URL, transaction ID, and deployment stats */} ## ArNS Integration Arlink offers two domain options for your deployments: ### Free Arlink Undernames Arlink provides free subdomains under the `arlink` ArNS name: - **Format**: `yourname_arlink.ar.io` - **Cost**: Free (no ArNS purchase required) - **Availability**: Instant, accessible via all ar.io gateways - **Limitation**: Must be unique across all Arlink deployments ### Existing ArNS Names Connect your owned ArNS names for custom domains: - Select your ArNS name from the dashboard dropdown - Optionally add undernames for versioning (e.g., `staging_myapp`, `v1_myapp`) - Arlink automatically updates ArNS records on deployment {/* Screenshot: ArNS name selector showing owned names and undername input field */} Undernames use underscore separators: `staging_myapp` not `staging.myapp`. See [Using Undernames for Versioning](/build/guides/hosting-decentralised-apps/using-undernames-for-versioning) for versioning strategies. **Deployment Management:** The Arlink dashboard lets you view deployment history, manage undernames, and rollback to previous deployments by updating which transaction ID your ArNS name points to. ![Arlink deployment build monitoring](/content/arlink-history.png) ## Limitations & Considerations Understanding Arlink's limitations helps you choose the right deployment tool for your project. ### Size and Time Constraints | Constraint | Limit | Impact | |------------|-------|--------| | Max Build Output | 10 MB | Applications larger than 10MB cannot be deployed | | Build Timeout | 10 minutes | Complex builds exceeding 10 minutes will fail | | Deployment Cost | Subsidized (beta) | Pricing may change after beta period | The 10MB limit applies to your **build output**, not your source code. Check your build size with: ```bash npm run build du -sh dist/ # or build/, out/, etc. ``` If your build exceeds 10MB, use `ario-deploy` CLI instead. ### Comparison: Arlink vs ario-deploy CLI Choose the right tool for your use case: } title="Use Arlink When..." description="You prefer visual interfaces over command-line tools, your build output is under 10MB, you want automated GitHub deployments, you need quick one-off deployments, you want to use free Arlink undernames, or your build completes in under 10 minutes." /> } title="Use CLI When..." description="Your build output exceeds 10MB, you need custom deployment scripts, you want CI/CD pipeline integration, you need Ethereum wallet deployment, you require Base-ETH payment options, or you want full control over deployment process." /> ### Additional Limitations **Build Environment:** - Standard Node.js environment only - No custom build tools or dependencies - Limited environment variable support - No Docker or custom runtimes **Deployment Features:** - No support for Ethereum wallet signatures - No custom payment methods (Base-ETH, etc.) - Limited automation beyond GitHub integration - No programmatic API access **ArNS Management:** - Cannot create new ArNS names through Arlink - Must purchase ArNS names separately at [arns.ar.io](https://arns.ar.io) - Limited undername configuration options If you outgrow Arlink's capabilities, all your existing deployments can be managed with `ario-deploy` CLI. See the [other guides in this series](/build/guides/hosting-decentralised-apps) for CLI deployment instructions. ## Continuous Deployment Arlink automatically sets up continuous deployment when you authorize GitHub access. ### How It Works Arlink adds webhooks to your repository to detect push events. When you push to your configured branch, Arlink automatically triggers a new build and deployment. ```bash git add . git commit -m "Update homepage content" git push origin main # Triggers automatic deployment ``` {/* Screenshot: Webhook configuration settings in Arlink dashboard */} ### Branch-Based Deployments Configure multiple branches to deploy to different undernames: | Branch | Undername | Purpose | |--------|-----------|---------| | `main` | `myapp` (root) | Production | | `develop` | `staging_myapp` | Staging | | `feature/*` | `dev_myapp` | Development | Monitor all deployments in the Arlink dashboard, which shows build status, commit hashes, build logs, and transaction IDs. For more details on continuous deployment setup, see the [Arlink Documentation](https://arlink.gitbook.io/arlink-docs). ## Summary You now know how to deploy applications using Arlink's visual interface: - **Visual web interface** for deploying without command-line tools - **GitHub integration** with automated builds and continuous deployment - **Domain options** including free Arlink undernames or existing ArNS names - **Build monitoring** with real-time logs and progress tracking - **Understanding limitations** to choose between Arlink and CLI tools for your project needs Arlink provides a quick and accessible way to deploy smaller applications. For larger builds, advanced CI/CD, or custom deployments, consider using the `ario-deploy` CLI covered in earlier guides. In our final guide we'll explore deploying using the ArDrive web UI. # Deploying with ArDrive (/build/guides/hosting-decentralised-apps/hosting-with-ardrive) ## Introduction Create **permanent dApps** using the ArDrive web interface. This guide shows you how to deploy your dApp or website to the permaweb using ArDrive's user-friendly interface. Check out the [series introduction](/build/guides/hosting-decentralised-apps) to learn how ar.io enables permanent, decentralised hosting with 100+ independent gateways. ## What You'll Learn - How to deploy dApps using ArDrive web - Creating manifests for proper file routing - Assigning friendly ArNS names - Updating your dApp with new versions ## Prerequisites **For simple apps and websites:** - Your dApp files ready for deployment - ArDrive account (free to create) **For advanced applications:** - dApp prepared with hash routing and relative file paths - Static files built (for frameworks like React) - Learn more about [preparing your dApp for deployment](/build/guides/hosting-decentralised-apps) ## Step-by-Step Deployment Go to the [ArDrive web app](https://app.ardrive.io/#/sign-in) and log in using your preferred method. If you don't have an account, follow the instructions to create one. Navigate to the drive where you want your project hosted. If you need a new drive: - Click the big red "New" button at the top left - Create a new drive - **Important:** Set the drive to **public** for others to access your dApp With your drive selected: - Click the big red "New" button again - Select "Upload Folder" - Navigate to your project's root directory (or built directory if required) - Select the entire directory to maintain your project's file structure Review the upload and associated cost. If everything looks correct, click "Confirm". **Cost Note:** Uploading to Arweave isn't free, but costs are usually quite small compared to the benefits of permanent hosting. While ArDrive displays files as a traditional file structure, they don't actually exist that way on Arweave. The manifest acts as a map to all your dApp files: - Navigate into your newly created folder by double-clicking it - Click the big red "New" button again - Select "New Manifest" in the "Advanced" section - Name the manifest and save it inside the folder you just created Once the manifest is created: - Click on it to expand its details - Go to the "Details" tab - Find the "Data TX ID" on the bottom right - Copy this unique identifier for your dApp Your dApp is now live on the permaweb forever! - Append the Data TX ID to a gateway URL: `https://turbo-gateway.com/YOUR-TX-ID` - It may take a few minutes for files to propagate through the network - Once propagated, your dApp is accessible to anyone, anywhere, at any time Make your dApp easier to access with an ArNS name: - If you own an ArNS name, you'll be prompted during manifest creation - If not, purchase one from [arns.ar.io](https://arns.ar.io) - You can also assign an ArNS name later by clicking the three dots next to any file and selecting "Assign ArNS name" ## Updating Your dApp Files uploaded to Arweave are **permanent and immutable** - they cannot be changed. However, the [Arweave File System (ArFS)](/build/advanced/arfs) protocol lets you "replace" them with new versions while keeping old ones accessible. ### How Updates Work **To update your dApp:** 1. **Make your changes** and build the static directory 2. **Upload the entire folder again** to the same location 3. **Follow the same steps** as the original upload 4. **Create a new manifest** with the same name as the old one 5. **The new manifest generates a new TX ID** for the updated dApp **Important Notes:** - The old version remains accessible to anyone with the correct TX ID - Old files won't display in ArDrive unless you view file history - Each version gets its own unique transaction ID ## Benefits of ArDrive Web Deployment - **User-friendly interface** - No command line required - **Automatic manifest creation** - Handles file routing for you - **Integrated ArNS support** - Easy domain name assignment - **Version management** - Built-in file history and updates - **Cost transparency** - See upload costs before confirming ## Next Steps } /> } /> } /> # Hosting Decentralised Apps on ar.io (/build/guides/hosting-decentralised-apps) import { BookOpen, Rocket, GitBranch, Globe, Zap, Upload, } from "lucide-react"; The goal of decentralised apps (dapps) has always been full stack. However, until now, that has not been possible due to single points of failure for hosting frontends. Ar.io solves this via: - **Permanent Storage**: Pay once and store your site forever on [Arweave](https://arweave.org/). - **Smart Domains**: a sovereign owned domain-space via [Ar.io Name System (ArNS)](/learn/arns). - **Distributed Gateway Network**: access your site as a subdomain of active [gateways](/learn/gateways) For example, this docs portal is a decentralised app: - [docs.ar.io](https://docs.ar.io) - [docs.turbo-gateway.com](https://docs.turbo-gateway.com) Same app, different gateway. **How this differs from a conventional static host.** A static host serves your site from servers it controls, for as long as your account and the service last. On ar.io you pay once per upload, the files are stored permanently, and every gateway in the network can serve them. The name you publish under is a token you own, not a record in someone else's control panel. The trade is that every deploy is immutable: you publish a new version and move the name, rather than overwriting files in place. You can repeat this by loading the `docs` subdomain for any of the active gateways in the network. ## What You'll Learn In this guide series, you'll learn how to deploy and manage permanent applications on ar.io. We'll cover the following: } /> } /> } /> } /> } /> Each guide will build on the last and by the end of series you'll be equipped to deploy and host any decentralised app. ## Next Steps } /> } /> } /> } /> # Migrating Your App to the New SDKs (/build/guides/hosting-decentralised-apps/migrating-your-app-to-new-sdks) ## Overview This guide will help developers migrate applications that interact with ar.io protocol features to the current SDKs and Solana-based workflows. For now, use this page as a placeholder while final SDK examples and migration details are reviewed. ## What This Guide Will Cover - Updating to the latest `@ar.io/sdk` version - Replacing older protocol interaction patterns with current Solana signer flows - Updating ArNS name registration and record-management code - Reviewing ANT handling for Metaplex Core NFT ownership and control - Separating Arweave data-upload wallets from Solana protocol wallets where needed - Verifying application behavior against devnet or staging examples ## Current References These pages contain related material that should inform the final migration guide: - [ar.io SDK](/sdks/ar-io-sdk) - [Registering ArNS Names Programmatically](/build/guides/working-with-arns/register-arns-programmatically) - [Setting ArNS Records Programmatically](/build/guides/working-with-arns/set-arns-records-programmatically) - [Deploy a Permanent dApp](/build/guides/hosting-decentralised-apps/deploy-permanent-dapp) ## Review Status This page needs technical review before launch. Final content should be checked against the latest `@ar.io/sdk` release, generated SDK docs, Solana signer examples, and current ArNS/ANT behavior. # Undernames for Environments and Versioning (/build/guides/hosting-decentralised-apps/using-undernames-for-versioning) In the [previous guide](/build/guides/hosting-decentralised-apps/deploying-with-ario-deploy), you deployed your application to your base ArNS name. Now you'll learn how to manage multiple versions and environments using **undernames** - subdomains under your ArNS name. An ArNS name is the stable pointer that versioned, immutable data needs: each release keeps its own permanent ID, and the name moves to the latest one. ## What You'll Learn - **Strategic versioning patterns** for managing multiple environments - **Package.json workflows** for dev → staging → production - **GitHub Actions automation** for environment-based deployments - **Instant rollbacks** using the ArNS app interface - **Best practices** for TTL configuration and testing ## Understanding Undernames Undernames let you create multiple versions under one ArNS name - like subdomains in traditional DNS, but permanent and decentralized. **Structure:** ``` your-arns-name (your ArNS name) ├─ @ (base) → your-arns-name.ar.io ├─ dev → dev_your-arns-name.ar.io ├─ staging → staging_your-arns-name.ar.io └─ v2 → v2_your-arns-name.ar.io ``` **Key benefits:** - **Unlimited undernames** with one ArNS registration - **Each undername** points to a different deployment (transaction ID) - **All versions** remain permanently accessible - **Instant rollbacks** by updating pointers in the ArNS app --- ## Common Versioning Patterns ### Pattern 1: Environment-Based Versioning Best for: Most production applications This pattern creates separate environments for your development workflow: ``` @ (production) → Current production release staging → Pre-release testing environment dev → Active development builds ``` **Add environment scripts to your project:** In your `my-permaweb-app` project from the previous guide, update `package.json` to add environment-specific deployment commands: ```json title="package.json" { "scripts": { "dev": "next dev", "build": "next build", "deploy": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out", "deploy:dev": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --undername dev --ttl-seconds 60", "deploy:staging": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --undername staging --ttl-seconds 60", "deploy:prod": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --ttl-seconds 3600" } } ``` Replace `your-arns-name` with your actual ArNS name. ```json title="package.json" { "scripts": { "dev": "vite", "build": "vite build", "deploy": "vite build && ario-deploy deploy --arns-name your-arns-name", "deploy:dev": "vite build && ario-deploy deploy --arns-name your-arns-name --undername dev --ttl-seconds 60", "deploy:staging": "vite build && ario-deploy deploy --arns-name your-arns-name --undername staging --ttl-seconds 60", "deploy:prod": "vite build && ario-deploy deploy --arns-name your-arns-name --ttl-seconds 60" } } ``` Replace `your-arns-name` with your actual ArNS name. --- ### Pattern 2: Version Archives Best for: Applications needing long-term version history This pattern extends Pattern 1 by archiving each production release to a version-specific undername for permanent access: ``` @ (production) → v2.0 (current release) staging → v2.1 (next release testing) dev → Latest development v2-0-0 → Version 2.0.0 (archived) v2-1-0 → Version 2.1.0 (archived) ``` **Add archive script:** Create a script that automatically archives based on your `package.json` version: ```javascript title="scripts/deploy-archive.js" const { execSync } = require('child_process'); const packageJson = require('../package.json'); // Get version from package.json (e.g., "2.1.0") const version = packageJson.version.replace(/\./g, '-'); // "2-1-0" console.log(`Archiving version ${version}...`); // Deploy to version-specific undername (build already done by npm script) execSync( `ario-deploy deploy --arns-name your-arns-name --deploy-folder out --undername v${version} --ttl-seconds 3600`, { stdio: 'inherit' } ); console.log(`✓ Archived at: https://v${version}_your-arns-name.ar.io`); ``` Then add to `package.json`: ```json title="package.json" { "scripts": { "deploy:dev": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --undername dev --ttl-seconds 60", "deploy:staging": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --undername staging --ttl-seconds 60", "deploy:prod": "next build && ario-deploy deploy --arns-name your-arns-name --deploy-folder out --ttl-seconds 3600", "deploy:archive": "next build && node scripts/deploy-archive.js" } } ``` ```javascript title="scripts/deploy-archive.js" const { execSync } = require('child_process'); const packageJson = require('../package.json'); // Get version from package.json (e.g., "2.1.0") const version = packageJson.version.replace(/\./g, '-'); // "2-1-0" console.log(`Archiving version ${version}...`); // Deploy to version-specific undername (build already done by npm script) execSync( `ario-deploy deploy --arns-name your-arns-name --undername v${version} --ttl-seconds 3600`, { stdio: 'inherit' } ); console.log(`✓ Archived at: https://v${version}_your-arns-name.ar.io`); ``` Then add to `package.json`: ```json title="package.json" { "scripts": { "deploy:dev": "vite build && ario-deploy deploy --arns-name your-arns-name --undername dev --ttl-seconds 60", "deploy:staging": "vite build && ario-deploy deploy --arns-name your-arns-name --undername staging --ttl-seconds 60", "deploy:prod": "vite build && ario-deploy deploy --arns-name your-arns-name --ttl-seconds 3600", "deploy:archive": "vite build && node scripts/deploy-archive.js" } } ``` Now whenever you run `deploy:archive` a version-specific immutable deployment of your app will be created and hosted permanently. For example, version 2.1.0 would be accessible forever at `https://v2-1-0_your-arns-name.ar.io`. --- ### Pattern 3: Component Architecture Best for: Monorepos with multiple deployable parts This pattern deploys different parts of your application to separate undernames: ``` @ (marketing) → Marketing website app → Main application docs → Documentation site admin → Admin panel ``` **Example package.json:** ```json title="package.json" { "scripts": { "deploy:marketing": "ario-deploy deploy --arns-name your-arns-name --deploy-folder ./marketing/dist --ttl-seconds 3600", "deploy:app": "ario-deploy deploy --arns-name your-arns-name --undername app --deploy-folder ./app/dist --ttl-seconds 60", "deploy:docs": "ario-deploy deploy --arns-name your-arns-name --undername docs --deploy-folder ./docs/dist --ttl-seconds 1800", "deploy:all": "npm run deploy:marketing && npm run deploy:app && npm run deploy:docs" } } ``` Each component can be deployed independently or all at once with `npm run deploy:all`. --- ## Automating Environment Deployments Building on the [GitHub Actions workflow](/build/guides/hosting-decentralised-apps/deploying-with-ario-deploy#automating-with-github-actions) from the previous guide, let's create environment-specific workflows that deploy based on branch activity. Deploy automatically when pushing to the `develop` branch: ```yaml title=".github/workflows/deploy-dev.yml" name: Deploy to Dev on: push: branches: [develop] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install dependencies run: npm ci - name: Deploy to Dev run: npm run deploy:dev env: DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} ``` **Usage:** Push to `develop` branch → Auto-deploys to `dev_your-arns-name.ar.io` Deploy automatically when pushing to the `staging` branch: ```yaml title=".github/workflows/deploy-staging.yml" name: Deploy to Staging on: push: branches: [staging] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install dependencies run: npm ci - name: Deploy to Staging run: npm run deploy:staging env: DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} ``` **Usage:** Push to `staging` branch → Auto-deploys to `staging_your-arns-name.ar.io` Deploy to production and create version archive on push to `main`: ```yaml title=".github/workflows/deploy-production.yml" name: Deploy to Production on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest environment: production # Configure in GitHub Settings steps: - uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '18' cache: 'npm' - name: Install dependencies run: npm ci - name: Deploy to Production run: npm run deploy:prod env: DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} - name: Archive Version run: npm run deploy:archive env: DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} ``` **Usage:** Push to `main` branch → Auto-deploys to `your-arns-name.ar.io` and archives to `v{version}_your-arns-name.ar.io` Every production deployment automatically creates a permanent archive based on your `package.json` version. For example, version 2.1.0 is archived at `v2-1-0_your-arns-name.ar.io`. **Typical git workflow:** ```bash title="Terminal" # Feature development git checkout develop git add . git commit -m "Add new feature" git push origin develop # Auto-deploys to dev # Promote to staging git checkout staging git merge develop git push origin staging # Auto-deploys to staging # Promote to production git checkout main git merge staging git push origin main # Auto-deploys to production ``` --- ## Instant Rollbacks with ArNS App One of the key benefits of undernames is instant rollbacks. Since all versions are permanently stored on Arweave, you can instantly switch between them using the [ArNS app](https://arns.ar.io). **To rollback to a previous version:** Navigate to [arns.ar.io](https://arns.ar.io) and find your ArNS name. You'll see all undernames and their transaction IDs. Find the undername with the version you want to rollback to (e.g., `v2-0-0` or `staging`) and copy its transaction ID. Click on the `@` (base) record, paste the transaction ID, and save. Production now serves that version - instant rollback! You're just updating a pointer to existing data. The old version is already on Arweave, permanent and accessible. No re-upload needed! For more details on managing undernames, see [Managing ArNS via UI](/build/guides/working-with-arns/manage-arns-ui). --- ## Summary You now know how to manage multiple environments using ArNS undernames: - **Environment patterns** for dev → staging → production workflows - **Version archives** for permanent access to all releases - **GitHub Actions automation** for branch-based deployments - **Instant rollbacks** through the ArNS app interface In our next guides we'll explore other tools you can use to host websites on ar.io without needing to configure `ario-deploy`. # Guides (/build/guides) Explore real-world applications and use cases for **Arweave** and **ar.io** infrastructure. These examples show what's possible with permanent data storage and decentralized web services. ## What You Can Build **Arweave and ar.io enable:** - **Decentralized websites** - Host permanent, censorship-resistant web content - **ArNS domains** - Create and manage decentralized domain names - **Data marketplaces** - Trade and sell digital assets and data - **Permanent applications** - Deploy apps that can't be taken down - **And much more** - The permanent web is only limited by your imagination ## Getting Started **Store encrypted data** with blind computing access control **Key topics:** - Encrypted data storage on Arweave - Nillion blind computing integration - Privacy-preserving data processing - Decentralized access control **Build permanent websites and apps** that can't be censored or taken down **Key topics:** - Step-by-step deployment guides - Multiple deployment methods (CLI, web UI) - ArNS domain integration - Production versioning strategies **Create and manage** decentralized domain names **Key topics:** - Primary name registration - Domain management - Integration with applications **Version and organize** your permanent website content **Key topics:** - Undername management - Website versioning - Component organization **Trade and sell** ArNS tokens and digital assets **Key topics:** - Ar.io Name Token (ANT) trading - Marketplace dynamics - Asset ownership **Deploy dApps easily** using the ArDrive web interface **Key topics:** - ArDrive web deployment - Manifest creation - ArNS name assignment - Version management **Distribute software applications** using Arweave manifests and ArNS routing **Key topics:** - Multi-platform application distribution - CI/CD integration with GitHub Actions - Manifest-based routing patterns - Cost optimization with compression **Move large IPFS pin sets** to permanent Arweave storage **Key topics:** - Custom Turbo-based migration scripts - CID-to-transaction mapping and provenance tags - Batch migration at scale (100k+ CIDs) - NFT metadata rewrite patterns ## Why Use Arweave? **Permanent storage** - Data stored on Arweave is permanent and cannot be deleted **Decentralized** - No single point of failure or control **Cost-effective** - Pay once, store forever **Censorship-resistant** - Content cannot be taken down by authorities ## Next Steps } /> } /> } /> # Migrate from IPFS to Arweave (/build/guides/migrate-from-ipfs-to-arweave) If you have thousands, or hundreds of thousands, of IPFS pins and want **permanent, pay-once storage**, Arweave via ar.io is a strong fit. Unlike pinning services that require ongoing fees, data uploaded to Arweave is stored permanently with a single upfront payment. This guide walks through a **custom programmatic migration** using the [Turbo SDK](/sdks/turbo-sdk). It applies to any file set pinned on IPFS, with NFT collections as a concrete example. ## Why Use a Custom Migration? IPFS migrations are usually project-specific. Your CIDs may represent raw files, directories, NFT assets, metadata JSON, application bundles, or a mix of all of them. You may also need custom tagging, retry behavior, validation rules, metadata rewrites, or contract-specific URL formats. For that reason, a custom script is usually the most flexible approach. With modern LLM-assisted development, generating and adapting a migration script for your exact CID inventory, metadata schema, and validation requirements is often faster than forcing a generic tool to fit your project. Useful docs to combine for a custom migration: - [Advanced Uploading with Turbo](/build/upload/advanced-uploading-with-turbo): authentication, uploads, and payment - [Tagging](/build/upload/tagging): metadata and discoverability - [Manifests](/build/upload/manifests): organizing collections under path-based routing - [Receipts](/build/upload/receipts): audit trail for uploads - [Storing NFTs on ar.io](/build/guides/storing-nfts): NFT asset and metadata upload patterns - ArDrive CLI also supports IPFS CID tagging for some workflows, but most large migrations benefit from a custom Turbo SDK script so you can control batching, retries, metadata rewrites, and validation. This guide connects those pieces into a migration workflow. ## When to Use a Custom Migration | Scenario | Recommended approach | | --- | --- | | A few dozen files | Manual upload via [Turbo SDK](/sdks/turbo-sdk) or [console.ar.io](https://console.ar.io/upload) | | Hundreds to millions of CIDs | Custom migration script (this guide) | | NFT collection with linked metadata | Custom script + metadata rewrite (see [NFT example](#nft-collection-example) below) | | Ongoing ingestion pipeline | Build a reusable worker that fetches from IPFS and uploads to Turbo | For full control over tagging, provenance, and reference rewriting, a custom script is usually the most flexible option. ## Migration Architecture At a high level, every migration follows the same pipeline: ```mermaid flowchart LR inventory["Inventory CIDs"] --> fetch["Fetch from IPFS gateway"] fetch --> upload["Upload via Turbo"] upload --> map["Persist CID to txId mapping"] map --> rewrite["Rewrite references"] rewrite --> index["Publish migration index"] index --> validate["Validate on ar.io"] ``` ### Inventory your CIDs Export your pin list from your pinning provider (Pinata, NFT.Storage, a self-hosted node, etc.) into a simple format: ```json [ { "cid": "bafybeig...", "path": "images/0.png", "contentType": "image/png" }, { "cid": "bafkreif...", "path": "metadata/0.json", "contentType": "application/json" } ] ``` Or use a newline-delimited file of CIDs if you have no path metadata: ``` bafybeig... bafkreif... ``` Deduplicate CIDs before starting. For file CIDs, the same CID should resolve to the same content bytes, so you only need to upload each unique file CID once. ### Fetch bytes from IPFS Retrieve each CID through a reliable IPFS gateway or your own node: ```javascript const IPFS_GATEWAY = 'https://ipfs.io/ipfs'; async function fetchFromIpfs(cid) { const response = await fetch(`${IPFS_GATEWAY}/${cid}`, { redirect: 'follow', }); if (!response.ok) { throw new Error(`Failed to fetch ${cid}: ${response.status}`); } const buffer = Buffer.from(await response.arrayBuffer()); const contentType = response.headers.get('content-type') ?? 'application/octet-stream'; return { buffer, contentType }; } ``` **Gateway reliability:** Public gateways can rate-limit or go offline. For large migrations, use your own IPFS node or a dedicated gateway from your pinning provider. Retry failed fetches with exponential backoff. ### Upload to Arweave via Turbo Authenticate with the Turbo SDK and upload each file. Tag every upload with provenance metadata so you can trace it back to its IPFS origin. ```javascript const jwk = JSON.parse(fs.readFileSync('./wallet.json', 'utf-8')); const turbo = TurboFactory.authenticated({ privateKey: jwk, token: 'arweave', }); async function uploadToArweave(cid, buffer, contentType, projectTag) { const result = await turbo.upload({ data: buffer, dataItemOpts: { tags: [ { name: 'Content-Type', value: contentType }, { name: 'App-Name', value: projectTag }, { name: 'Source-Protocol', value: 'ipfs' }, { name: 'Source-CID', value: cid }, ], }, }); return result.id; } ``` See [Tagging best practices](#tagging-for-provenance) below for recommended tags. ### Persist the CID-to-txId mapping Write results to an append-only state file after every upload. This is your migration ledger and lets you resume after failures. ```json { "cid": "bafybeig...", "txId": "Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", "contentType": "image/png", "bytes": 48291, "status": "success", "uploadedAt": "2026-05-27T12:00:00Z" } ``` Store this mapping durably, on disk at minimum, and ideally backed up. You will need it to rewrite references and to audit the migration. ### Rewrite references Update any documents, metadata, or application configs that point at `ipfs://` URIs to use Arweave references instead. For broad compatibility today, use gateway URLs in fields that external platforms must fetch immediately, and keep `ar://` references where your application, contract, or metadata consumers support them. The mapping file from the previous step drives this rewrite pass. ### Publish a migration index After the main upload pass, create a JSON index that links every original CID to its Arweave transaction ID. This gives your team and downstream users a durable lookup table for audits, support, and future migrations. ### Validate Fetch each uploaded transaction from an ar.io gateway or via [Wayfinder](/build/access/wayfinder) and compare it with what you fetched from IPFS. Retain [Turbo receipts](/build/upload/receipts) for audit purposes. ## Running Migrations at Scale For collections in the **tens or hundreds of thousands**, treat the migration as a long-running batch job. ### Cost estimation Before starting, estimate total upload cost: 1. Sum the byte size of all unique CIDs in your inventory. 2. Use the [pricing calculator](https://console.ar.io/pricing) or `turbo.getFiatEstimateForBytes()` from the SDK. 3. Purchase sufficient [Turbo Credits](/build/upload/turbo-credits) before beginning. Uploads under **105 KiB are free** and do not require a prior top-up. For large migrations this is negligible, but worth knowing for small metadata files. ### Concurrency and rate limits - Start with **low concurrency** (3–5 parallel uploads) and increase gradually while monitoring for errors. - IPFS gateways and Turbo both have rate limits. Separate fetch concurrency from upload concurrency. - Use exponential backoff on transient failures (HTTP 429, 502, network timeouts). ### Checkpointing and resume - Skip CIDs that already appear in your state file with `status: "success"`. - Write state **after each successful upload**, not in batches. If the process crashes, you lose at most one item. - Log failures separately so you can retry them in a second pass. ### Handling edge cases | Issue | What to do | | --- | --- | | Duplicate CIDs | Upload once, reuse the same txId in your mapping | | Missing/unavailable CID | Log as failed, retry later; do not block the entire run | | Unknown content type | Default to `application/octet-stream`; inspect bytes if needed | | Very large files | Stream via `turbo.uploadFile` with `fileStreamFactory` instead of buffering entirely in memory | | DagPB / directory CIDs | Resolve to individual file CIDs first; directory CIDs are not uploadable as a single blob | ## Tagging for Provenance Every migrated upload should include tags that make the data discoverable and traceable: | Tag | Purpose | | --- | --- | | `Content-Type` | Required: tells gateways how to serve the data | | `App-Name` | Identifies your project (e.g. `MyCollection-Migration-v1`) | | `Source-Protocol` | Set to `ipfs` to mark migrated content | | `Source-CID` | The original IPFS CID, useful for provenance and GraphQL queries | | `Migration-Date` | ISO timestamp of when the upload occurred | | `Collection-Name` | Optional: groups uploads from the same project | **Why tag the original CID?** Storing the source CID on each Arweave transaction lets you query your migrated data via [GraphQL](/build/access/find-data), correlate uploads back to IPFS origins, and support provenance checks without relying on an external mapping file alone. **IPFS-specific tags:** Some ArDrive CLI workflows support adding an `IPFS-Add` tag to public uploads, which may be useful where bridge-aware infrastructure recognizes that tag. For large custom migrations, the Turbo SDK pattern above is usually a better fit because you can choose your own tag names, preserve richer migration state, and adapt the script to your data model. See the full [Tagging guide](/build/upload/tagging) for tag size limits and best practices. ## Publish a Migration Index Your append-only state file is operational state. Once migration is complete, publish a clean migration index that others can use without reading your job logs. ```json { "type": "ipfs-to-arweave-migration", "version": "1.0.0", "project": "MyProject", "createdAt": "2026-05-27T12:00:00Z", "items": [ { "cid": "bafybeig...", "txId": "Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", "gatewayUrl": "https://arweave.net/Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", "arUri": "ar://Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", "contentType": "image/png", "bytes": 48291 } ] } ``` Upload the index with Turbo and tag it so it can be discovered later: ```javascript const migrationIndex = JSON.parse(fs.readFileSync('./migration-index.json', 'utf-8')); const indexUpload = await turbo.upload({ data: Buffer.from(JSON.stringify(migrationIndex)), dataItemOpts: { tags: [ { name: 'Content-Type', value: 'application/json' }, { name: 'App-Name', value: PROJECT_TAG }, { name: 'Data-Type', value: 'IPFS-Migration-Index' }, { name: 'Source-Protocol', value: 'ipfs' }, ], }, }); console.log(`Migration index: https://arweave.net/${indexUpload.id}`); console.log(`Migration index: ar://${indexUpload.id}`); ``` ## Complete Script Skeleton This skeleton ties the pipeline together. Extend it with your own retry logic, logging, and concurrency controls. **Memory usage:** This example buffers each fetched CID in memory before uploading. That keeps the script compact, but it is best for small or medium-sized files. For large media files, stream to disk first and upload with `turbo.uploadFile` so concurrent workers do not hold many large buffers in memory. ```javascript const IPFS_GATEWAY = process.env.IPFS_GATEWAY ?? 'https://ipfs.io/ipfs'; const STATE_FILE = './migration-state.jsonl'; const CONCURRENCY = 5; const PROJECT_TAG = 'MyProject-Migration-v1'; const jwk = JSON.parse(fs.readFileSync('./wallet.json', 'utf-8')); const turbo = TurboFactory.authenticated({ privateKey: jwk, token: 'arweave' }); const inventory = JSON.parse(fs.readFileSync('./cids.json', 'utf-8')); const completed = new Set( fs.existsSync(STATE_FILE) ? fs.readFileSync(STATE_FILE, 'utf-8') .trim() .split('\n') .filter(Boolean) .map((line) => JSON.parse(line)) .filter((r) => r.status === 'success') .map((r) => r.cid) : [], ); function appendState(record) { fs.appendFileSync(STATE_FILE, JSON.stringify(record) + '\n'); } async function fetchFromIpfs(cid) { const response = await fetch(`${IPFS_GATEWAY}/${cid}`, { redirect: 'follow' }); if (!response.ok) throw new Error(`Fetch failed: ${response.status}`); const buffer = Buffer.from(await response.arrayBuffer()); const contentType = response.headers.get('content-type') ?? 'application/octet-stream'; return { buffer, contentType }; } async function migrateOne({ cid, contentType: declaredType }) { if (completed.has(cid)) { console.log(`Skipping ${cid} (already migrated)`); return; } try { const { buffer, contentType } = await fetchFromIpfs(cid); const result = await turbo.upload({ data: buffer, dataItemOpts: { tags: [ { name: 'Content-Type', value: declaredType ?? contentType }, { name: 'App-Name', value: PROJECT_TAG }, { name: 'Source-Protocol', value: 'ipfs' }, { name: 'Source-CID', value: cid }, { name: 'Migration-Date', value: new Date().toISOString() }, ], }, }); appendState({ cid, txId: result.id, contentType: declaredType ?? contentType, bytes: buffer.length, status: 'success', uploadedAt: new Date().toISOString(), }); console.log(`Migrated ${cid} → ar://${result.id}`); } catch (error) { appendState({ cid, status: 'failed', error: error.message, failedAt: new Date().toISOString(), }); console.error(`Failed ${cid}:`, error.message); } } async function runPool(items, fn, concurrency) { const queue = [...items]; const workers = Array.from({ length: concurrency }, async () => { while (queue.length > 0) { const item = queue.shift(); if (item) await fn(item); } }); await Promise.all(workers); } await runPool(inventory, migrateOne, CONCURRENCY); console.log('Migration complete. Review migration-state.jsonl for results.'); ``` ## NFT Collection Example NFT collections are a common migration target because metadata JSON files typically reference images via `ipfs://` URIs. The workflow has an extra rewrite step. ### Separate assets from metadata Split your inventory into two groups: - **Assets**: images, animations, and other media files - **Metadata**: JSON files containing `name`, `description`, `image`, `attributes`, etc. Upload assets first so you have transaction IDs to reference in metadata. ### Upload assets and build the mapping Run the migration script on all asset CIDs. Your state file now maps each image CID to an Arweave transaction ID. ### Rewrite metadata references For each metadata JSON, replace `ipfs://` URIs with Arweave references using your mapping. Because `ar://` is not yet universally supported by NFT marketplaces and wallets, the most robust current approach is to use gateway URLs for widely consumed fields like `image`, while also keeping `ar://` values in additional fields for applications that support them. ```javascript const GATEWAY_URL = 'https://arweave.net'; function getArweaveReferences(uri, cidToTxId) { if (!uri.startsWith('ipfs://')) return uri; const cid = uri.replace('ipfs://', '').split('/')[0]; const txId = cidToTxId[cid]; if (!txId) { console.warn(`No mapping for CID: ${cid}`); return uri; } return { gatewayUrl: `${GATEWAY_URL}/${txId}`, arUri: `ar://${txId}`, }; } function rewriteMetadata(metadata, cidToTxId) { const rewritten = { ...metadata }; for (const field of ['image', 'animation_url', 'external_url']) { if (rewritten[field]) { const references = getArweaveReferences(rewritten[field], cidToTxId); if (typeof references === 'string') continue; rewritten[field] = references.gatewayUrl; rewritten[`${field}_ar`] = references.arUri; } } return rewritten; } ``` **Use both where practical.** Gateway URLs have the broadest compatibility today. `ar://` URIs are more future-proof and can resolve through [Wayfinder](/build/access/wayfinder), but adoption is still growing. Keeping both gives downstream consumers a stable HTTP URL now and a protocol-native reference for clients that support it. ### Upload rewritten metadata Upload each rewritten metadata JSON via Turbo. Tag with the same provenance tags plus the token identifier if applicable. For large collections (100+ tokens), consider uploading metadata as a [manifest](/build/upload/manifests) so your smart contract can use a single base URI. Use an HTTP gateway base URI when you need maximum marketplace compatibility, and keep the `ar://` manifest URI documented for clients that support it: ```solidity string private constant MANIFEST_ID = "your-manifest-transaction-id"; string private constant GATEWAY = "https://arweave.net/"; function tokenURI(uint256 tokenId) public view returns (string memory) { return string(abi.encodePacked(GATEWAY, MANIFEST_ID, "/", tokenId.toString(), ".json")); } ``` See [Storing NFTs on ar.io](/build/guides/storing-nfts) for the full manifest workflow. ### Update onchain references If NFTs are already minted with `ipfs://` token URIs, updating onchain metadata requires a contract-specific approach (owner update functions, redeployment, or a new base URI if your contract supports it). Plan this step before migrating. ## Validation After migration, verify a sample of uploads and ideally all failed items from your retry pass. A size check is a useful first pass; for stronger validation, compare hashes or the full buffers: ```javascript async function validateUpload(txId, expectedBuffer) { const response = await fetch(`https://turbo-gateway.com/${txId}`); if (!response.ok) throw new Error(`Gateway fetch failed: ${response.status}`); const actual = Buffer.from(await response.arrayBuffer()); if (actual.length !== expectedBuffer.length) { throw new Error(`Size mismatch: expected ${expectedBuffer.length}, got ${actual.length}`); } if (!actual.equals(expectedBuffer)) { throw new Error(`Content mismatch for ${txId}`); } console.log(`Validated ar://${txId} (${actual.length} bytes)`); } ``` For production workloads, [Wayfinder](/build/access/wayfinder) can add cryptographic verification for clients that support it. ## Next Steps } /> } /> } /> } /> } /> } /> # Storing NFTs on ar.io (/build/guides/storing-nfts) Storing NFT assets on ar.io ensures they're permanent, immutable and accessible, forever. We achieve this by providing: - **Permanent Storage**: your assets are stored forever on Arweave. - **One-Time Payment**: Pay once upfront for permanent storage. - **Decentralised Gateways**: a network of 100s of incentivized gateways provide access. - **Routing Protocol**: ar://wayfinder ensures an active gateway is selected for all requests. Once your NFT assets are stored on ar.io, no one can delete, edit or "rug" them. That means Creators can be confident their art will live on forever and Collectors can be confident their investment won't disappear. ## How to Store NFT assets on ar.io First, it's important to understand ar.io doesn't let you mint NFTs. What it does provide is permanent storage and access for the media files (images, video, animations) and metadata (artist, attributes, traits) JSON which is used in NFTs. To mint an NFT you'd typically store your assets and metadata via ar.io and then use another service to "mint" the NFTs onchain. This guide is targeted at developers. If you're not a developer we strongly recommend uploading to ar.io via [ArDrive](https://ardrive.io), check out this amazing video guide from [Hashlips](https://x.com/Hashlipsnft): ## Uploading Your First NFT Let's walk through uploading a complete NFT with its metadata using Turbo, an ar.io gateway run by the ArDrive team. This guide assumes you're in a node.js environment and using an Arweave wallet, we support uploads in the browser and many other wallet types (EVM, SVM) and payment tokens (ETH, USDC, POL etc.). See [using turbo in the browser](/build/guides/using-turbo-in-a-browser) and [advanced uploading with with turbo](/build/upload/advanced-uploading-with-turbo) for more info. ### Prerequisites Before starting, ensure you have: - **Node.js** (v18 or higher) - **Arweave Wallet (JWK file)** - We recommend [Wander](https://www.wander.app/) if you need a wallet. - **Turbo Credits** - Purchase credits to pay for uploads. See [Turbo Credits guide](/build/upload/turbo-credits) - **Your NFT assets** - The image file(s) you want to upload - **Basic JavaScript knowledge** - Familiarity with Node.js and async/await Let's get started: ### Install and Authenticate Install the Turbo SDK and authenticate with your Arweave wallet: ```bash npm install @ardrive/turbo-sdk ``` Then authenticate and upload your NFT image: ```javascript try { // Authenticate with your Arweave wallet const jwk = JSON.parse(fs.readFileSync('./wallet.json', 'utf-8')); const turbo = TurboFactory.authenticated({ privateKey: jwk, token: 'arweave' }); // Upload the NFT image const imageUpload = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream('./my-nft.png'), fileSizeFactory: () => fs.statSync('./my-nft.png').size, dataItemOpts: { tags: [ { name: 'Content-Type', value: 'image/png' } ] } }); // Get the ar:// URL const imageUrl = `ar://${imageUpload.id}`; console.log('Image uploaded:', imageUrl); // Output: ar://Xj9k2Lm8Pq3Rn5Tv7Wz... } catch (error) { console.error('Upload failed:', error.message); // Check wallet balance, file exists, or network connection } ``` ### Create and Upload Metadata Now create your NFT metadata following the OpenSea/ERC-721 standard, using the `ar://` URL for the image. #### Required Fields ```json { "name": "NFT Name", "description": "Description of your NFT", "image": "ar://image-transaction-id" } ``` #### Full Example with Optional Fields ```json { "name": "Cosmic Explorer #42", "description": "A unique space explorer with permanent storage on Arweave", "image": "ar://Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", "animation_url": "ar://Zb1Cd3Ef5Gh7Ij9Kl1Mn3Op5Qr7St9Uv1Wx3Yz", "external_url": "ar://my-collection-website", "attributes": [ { "trait_type": "Class", "value": "Explorer" }, { "trait_type": "Power Level", "value": 9001, "display_type": "number" } ] } ``` **Critical**: All URL fields fetching data from ar.io (`image`, `animation_url`, `external_url`) should use the `ar://` protocol, not hardcoded gateway URLs. This is what makes your NFT future-proof. #### Upload Metadata with Turbo ```javascript // Create metadata with ar:// reference const metadata = { name: 'My Awesome NFT', description: 'Stored permanently on Arweave', image: imageUrl, // ar://transaction-id attributes: [ { trait_type: 'Background', value: 'Cosmic Blue' }, { trait_type: 'Rarity', value: 'Epic' } ] }; // Upload metadata const metadataUpload = await turbo.uploadFile({ fileStreamFactory: () => Buffer.from(JSON.stringify(metadata)), fileSizeFactory: () => Buffer.from(JSON.stringify(metadata)).length, dataItemOpts: { tags: [ { name: 'Content-Type', value: 'application/json' } ] } }); const metadataUrl = `ar://${metadataUpload.id}`; console.log('Metadata uploaded:', metadataUrl); console.log('Use this URL when minting your NFT!'); ``` ### Use in Your Minting Contract Use the metadata URL in your NFT contract's `tokenURI` field: ```solidity function tokenURI(uint256 tokenId) public view returns (string memory) { return "ar://your-metadata-transaction-id"; } ``` For a single NFT, all tokens can share the same metadata. For collections with unique metadata per token, see the manifest section below. **Note:** Uploads are typically available within seconds, but may take a few minutes to propagate across all ar.io gateways. ### Summary This example has shown you how to store your NFT assets and metadata on Arweave via ar.io and then add this to the tokenURI field of your NFT's smart contract. Next we'll explore how to store NFT images and metadata for entire collections. ## Organizing Collections with Manifests For NFT collections, manifests provide an efficient way to organize multiple files under a single transaction ID. ### What Are Manifests? A manifest acts like a folder on Arweave, mapping paths to transaction IDs. For NFT collections, manifests organize metadata access: - `ar://manifest-id/0.json` → Token 0 metadata - `ar://manifest-id/1.json` → Token 1 metadata - `ar://manifest-id/42.json` → Token 42 metadata Each metadata file contains fully qualified `ar://` URIs pointing to permanent image transaction IDs. This approach gives you: - **Organized metadata**: One manifest ID for all collection metadata - **Universal compatibility**: Images use standard `ar://txid` format that works everywhere - **Clean contract code**: Simple `baseURI + tokenId + .json` pattern ### When to Use Manifests - **Large collections** (100+ NFTs): Organize metadata files under a single manifest ID - **Clean contract code**: Use simple `manifest-id/tokenId.json` pattern instead of mapping each token to individual metadata TxIDs - **Future flexibility**: Update by pointing to new manifest without changing image references - **Simplified management**: Track one manifest ID for metadata access **How this workflow balances cost and compatibility:** Images are uploaded once with individual TxIDs (universally compatible), while metadata is organized in a manifest (easy contract integration). For single NFTs or very small collections (under 10), uploading metadata files individually may be simpler. ### Creating a Collection with Manifests ### Organize Your Files Create a folder structure with images and metadata templates: ``` my-collection/ ├── images/ │ ├── 0.png │ ├── 1.png │ ├── 2.png │ └── ... ├── metadata/ │ ├── 0.json │ ├── 1.json │ ├── 2.json │ └── ... ``` Start with metadata templates (image field will be populated in Step 3): ```json { "name": "My NFT #0", "description": "Part of my permanent NFT collection", "image": "", "attributes": [ { "trait_type": "Background", "value": "Blue" } ] } ``` ### Upload Images and Collect Transaction IDs Upload all images individually and collect their permanent transaction IDs: ```javascript // Authenticate const jwk = JSON.parse(fs.readFileSync('./wallet.json', 'utf-8')); const turbo = TurboFactory.authenticated({ privateKey: jwk, token: 'arweave' }); // Upload all images from images folder const imagesDir = './my-collection/images'; const imageFiles = fs.readdirSync(imagesDir).sort(); const imageTxIds = {}; for (const filename of imageFiles) { const filePath = path.join(imagesDir, filename); const upload = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream(filePath), fileSizeFactory: () => fs.statSync(filePath).size, dataItemOpts: { tags: [ { name: 'Content-Type', value: 'image/png' } ] } }); imageTxIds[filename] = upload.id; console.log(`Uploaded ${filename}: ar://${upload.id}`); } console.log('All images uploaded:', imageTxIds); ``` ### Update Metadata with Image Transaction IDs Update each metadata file to reference its permanent image transaction ID: ```javascript // Update metadata files with fully qualified image URIs const metadataDir = './my-collection/metadata'; const metadataFiles = fs.readdirSync(metadataDir).sort(); for (const filename of metadataFiles) { const filePath = path.join(metadataDir, filename); const metadata = JSON.parse(fs.readFileSync(filePath, 'utf-8')); // Map metadata to image: "0.json" -> "0.png" const imageFilename = filename.replace('.json', '.png'); const imageTxId = imageTxIds[imageFilename]; if (!imageTxId) { console.warn(`No image found for ${filename}`); continue; } // Set fully qualified ar:// URI metadata.image = `ar://${imageTxId}`; fs.writeFileSync(filePath, JSON.stringify(metadata, null, 2)); console.log(`Updated ${filename} with image: ${metadata.image}`); } ``` ### Upload Metadata Folder as Manifest Upload only the metadata folder to create a manifest for organized access: ```javascript // Upload metadata folder to create manifest const result = await turbo.uploadFolder({ folderPath: './my-collection/metadata' }); const manifestId = result.manifestId; console.log('Metadata manifest ID:', manifestId); console.log('Token 0 metadata: ar://' + manifestId + '/0.json'); console.log('Token 1 metadata: ar://' + manifestId + '/1.json'); ``` Each metadata file now contains fully qualified `ar://` image references that work with any NFT platform. ### Use in Your Minting Contract Use the metadata manifest ID for clean, organized token URIs: ```solidity string private constant MANIFEST_ID = "your-manifest-transaction-id"; function tokenURI(uint256 tokenId) public view returns (string memory) { return string(abi.encodePacked("ar://", MANIFEST_ID, "/", tokenId.toString(), ".json")); } ``` **How it resolves:** 1. Contract returns: `ar://manifest-id/42.json` 2. Platform fetches metadata containing: `"image": "ar://individual-image-txid"` 3. Platform fetches image from its permanent transaction ID ### Summary By following this workflow, you've created a permanent, organized NFT collection where: 1. **Images stored with permanent transaction IDs** that work universally 2. **Metadata contains fully qualified ar:// image URIs** compatible with all NFT platforms 3. **Metadata organized under one manifest ID** for clean contract integration 4. **Your smart contract uses one manifest ID** to access all token metadata Your NFTs are now stored permanently with guaranteed access, following industry standards, and future-proofed against gateway dependencies. ## Next Steps } /> } /> } /> # Using Turbo SDK with Vanilla HTML (/build/guides/using-turbo-in-a-browser/html) # Using Turbo SDK with Vanilla HTML **Firefox Compatibility**: Some compatibility issues have been reported with the Turbo SDK in Firefox browsers. At this time the below framework examples may not behave as expected in Firefox. ## Overview This guide demonstrates how to integrate the `@ardrive/turbo-sdk` directly into vanilla HTML pages using CDN imports. No build tools, bundlers, or polyfills are required - just modern ES modules support in browsers. **Note**: Vanilla HTML implementation is the simplest way to get started with the Turbo SDK. It's perfect for prototyping, simple applications, or when you want to avoid build complexity. ## Prerequisites - Modern browser with ES modules support (Chrome 61+, Firefox 60+, Safari 10.1+, Edge 16+) - Basic understanding of HTML, CSS, and JavaScript - HTTPS hosting for production (required for browser wallet integrations) Create a basic HTML file with Turbo SDK integration: ```html Turbo SDK Example body { font-family: Arial, sans-serif; max-width: 800px; margin: 0 auto; padding: 20px; } .section { margin: 20px 0; padding: 20px; border: 1px solid #ddd; border-radius: 8px; } .loading { color: #666; font-style: italic; } .error { color: red; } .success { color: green; } button { background: #007cba; color: white; border: none; padding: 10px 20px; border-radius: 4px; cursor: pointer; margin: 5px; } button:hover { background: #005a87; } button:disabled { background: #ccc; cursor: not-allowed; } Turbo SDK - Vanilla HTML Demo Current Rates Loading rates... Upload File Upload File // Initialize Turbo client const turbo = TurboFactory.unauthenticated(); // Fetch and display rates async function loadRates() { try { const rates = await turbo.getFiatRates(); const ratesDiv = document.getElementById("rates"); const ratesText = Object.entries(rates.fiat) .map( ([currency, rate]) => `${currency.toUpperCase()}: $${rate} per GiB` ) .join(""); ratesDiv.innerHTML = ratesText; } catch (error) { document.getElementById( "rates" ).innerHTML = `Error loading rates: ${error.message}`; } } // Handle file upload document .getElementById("uploadForm") .addEventListener("submit", async (e) => { e.preventDefault(); const fileInput = document.getElementById("fileInput"); const uploadBtn = document.getElementById("uploadBtn"); const statusDiv = document.getElementById("uploadStatus"); if (!fileInput.files.length) { statusDiv.innerHTML = 'Please select a file'; return; } const file = fileInput.files[0]; uploadBtn.disabled = true; statusDiv.innerHTML = 'Preparing upload...'; try { // Show upload cost first const costs = await turbo.getUploadCosts({ bytes: [file.size] }); const cost = costs[0]; statusDiv.innerHTML = ` Upload cost: ${cost.winc} winc File size: ${file.size.toLocaleString()} bytes Note: This example cannot complete uploads without wallet authentication. See wallet integration examples below for full upload functionality. `; } catch (error) { statusDiv.innerHTML = `Error: ${error.message}`; } finally { uploadBtn.disabled = false; } }); // Load rates on page load loadRates(); ``` Select the appropriate CDN import method for your needs: **Use esm.sh for best compatibility**: The `unpkg.com` CDN has known issues with ES module exports for complex packages like Turbo SDK. **Latest Version (Recommended for Development)** ```javascript ``` **Specific Version (Recommended for Production)** ```javascript ``` **Alternative CDN Providers** ```javascript // jsDelivr // SkyPack // unpkg.com (not recommended - has ES module issues) ``` Connect your browser wallet to enable file uploads: **Never expose private keys in browser applications!** Always use browser wallet integrations. **Uploading with Wander** **Deprecation Notice**: The signature API used by ArConnect wallets is deprecated and will be removed. Visit [Wander wallet documentation](https://docs.wander.app/api/signature) for alternatives. Complete HTML page with Wander wallet integration: ```html Turbo SDK - Wander Wallet body { font-family: Arial, sans-serif; max-width: 600px; margin: 0 auto; padding: 20px; background: #f5f5f5; } .container { background: white; padding: 30px; border-radius: 10px; box-shadow: 0 2px 10px rgba(0, 0, 0, 0.1); } .wallet-section { border: 2px solid #e0e0e0; border-radius: 8px; padding: 20px; margin: 20px 0; } .connected { border-color: #4caf50; background-color: #f9fff9; } button { background: #000; color: white; border: none; padding: 12px 24px; border-radius: 6px; cursor: pointer; font-size: 16px; margin: 5px; } button:hover { background: #333; } button:disabled { background: #ccc; cursor: not-allowed; } .status { margin: 10px 0; padding: 10px; border-radius: 4px; } .success { background: #d4edda; color: #155724; border: 1px solid #c3e6cb; } .error { background: #f8d7da; color: #721c24; border: 1px solid #f5c6cb; } .info { background: #d1ecf1; color: #0c5460; border: 1px solid #bee5eb; } ⚡ Turbo SDK + Wander Wallet Wander Wallet Connection Connect your Wander wallet to upload files to Arweave using your AR balance. Connect Wander Wallet 📁 File Upload Upload to Arweave import { TurboFactory, ArconnectSigner, } from "https://esm.sh/@ardrive/turbo-sdk"; let connectedAddress = null; let turboClient = null; // Connect to Wander wallet async function connectWanderWallet() { const statusDiv = document.getElementById("walletStatus"); const connectBtn = document.getElementById("connectBtn"); try { if (!window.arweaveWallet) { statusDiv.innerHTML = ` Wander wallet is not installed! Install Wander Wallet `; return; } connectBtn.disabled = true; statusDiv.innerHTML = 'Connecting to Wander wallet...'; // Required permissions for Turbo SDK const permissions = [ "ACCESS_ADDRESS", "ACCESS_PUBLIC_KEY", "SIGN_TRANSACTION", "SIGNATURE", ]; // Connect to wallet await window.arweaveWallet.connect(permissions); // Get wallet address connectedAddress = await window.arweaveWallet.getActiveAddress(); // Create authenticated Turbo client const signer = new ArconnectSigner(window.arweaveWallet); turboClient = TurboFactory.authenticated({ signer }); // Update UI document.getElementById("walletSection").classList.add("connected"); statusDiv.innerHTML = ` ✅ Connected to Wander Wallet Address: ${connectedAddress.slice( 0, 8 )}...${connectedAddress.slice(-8)} `; connectBtn.style.display = "none"; document.getElementById("uploadSection").style.display = "block"; } catch (error) { console.error("Wander wallet connection failed:", error); statusDiv.innerHTML = `Connection failed: ${error.message}`; } finally { connectBtn.disabled = false; } } // Upload file function async function uploadFile() { const fileInput = document.getElementById("fileInput"); const statusDiv = document.getElementById("uploadStatus"); if (!fileInput.files.length) { statusDiv.innerHTML = 'Please select a file first'; return; } if (!turboClient) { statusDiv.innerHTML = 'Please connect Wander wallet first'; return; } const file = fileInput.files[0]; let uploadStartTime = Date.now(); statusDiv.innerHTML = 'Preparing upload...'; try { // Get upload cost first const costs = await turboClient.getUploadCosts({ bytes: [file.size], }); const cost = costs[0]; statusDiv.innerHTML = ` Upload cost: ${cost.winc} winc Starting upload... `; // Upload with comprehensive progress tracking const result = await turboClient.uploadFile({ fileStreamFactory: () => file.stream(), fileSizeFactory: () => file.size, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type || "application/octet-stream", }, { name: "App-Name", value: "Turbo-HTML-Wander-Demo" }, { name: "File-Name", value: file.name }, { name: "Upload-Timestamp", value: new Date().toISOString() }, ], }, events: { onProgress: ({ totalBytes, processedBytes, step }) => { const percent = Math.round((processedBytes / totalBytes) * 100); const elapsed = Math.round( (Date.now() - uploadStartTime) / 1000 ); statusDiv.innerHTML = ` ${step}: ${percent}% Progress: ${processedBytes.toLocaleString()} / ${totalBytes.toLocaleString()} bytes Elapsed: ${elapsed}s `; }, onError: ({ error, step }) => { console.error(`Error during ${step}:`, error); statusDiv.innerHTML = `Error during ${step}: ${error.message}`; }, }, }); const totalTime = Math.round((Date.now() - uploadStartTime) / 1000); // Use original file size for display (result object doesn't contain size info) const displayBytes = file.size; statusDiv.innerHTML = ` 🎉 Upload Successful! Transaction ID: ${ result.id } File Size: ${displayBytes.toLocaleString()} bytes Upload Time: ${totalTime}s Timestamp: ${new Date( result.timestamp ).toLocaleString()} View File: turbo-gateway.com/${result.id} Explorer: ViewBlock `; } catch (error) { console.error("Upload failed:", error); statusDiv.innerHTML = `Upload failed: ${error.message}`; } } // Make functions available globally for onclick handlers window.connectWanderWallet = connectWanderWallet; window.uploadFile = uploadFile; ``` Browser wallet integrations require HTTPS in production: ```html ``` Configure CSP headers to allow CDN imports: ```html ``` Implement comprehensive error handling: ```javascript // Network error handling async function robustApiCall(apiFunction, retries = 3) { for (let i = 0; i setTimeout(resolve, 1000 * Math.pow(2, i)) ); } } } // Usage example const rates = await robustApiCall(() => turbo.getFiatRates()); ``` Optimize for production environments: ```html // Production code here ``` ## Best Practices ### 1. User Experience - **Loading States**: Always show loading indicators during API calls - **Error Recovery**: Provide clear error messages with recovery options - **Progress Tracking**: Show upload progress for large files - **Wallet Detection**: Guide users to install wallets if missing ### 2. Security - **Never expose private keys** in browser applications - **Validate user inputs** before API calls - **Use HTTPS** for all production deployments - **Implement CSP headers** to prevent XSS attacks ### 3. Performance - **Cache API responses** where appropriate (rates, costs) - **Use specific CDN versions** in production - **Implement retry logic** for network failures - **Optimize file handling** for large uploads ### 4. Development - **Use development endpoints** during testing - **Test wallet integrations** across different browsers - **Validate upload functionality** with small files first - **Monitor API rate limits** and implement backoff ## Troubleshooting Common Issues ### CDN Import Errors If you encounter errors like: - `The requested module does not provide an - Module resolution failures **Solution**: Use `esm.sh` instead of `unpkg.com`: ```javascript // ❌ Problematic // ✅ Working ``` ### Function Scope Issues If onclick handlers throw `ReferenceError: function is not defined`: **Solution**: Use explicit global assignment: ```javascript // ❌ Problematic window.myFunction = async function() { ... } // ✅ Working async function myFunction() { ... } window.myFunction = myFunction; ``` ### Upload Result Properties If upload results have undefined properties: **Solution**: Use original file size for display: ```javascript // ❌ Problematic - these properties don't exist in result object const totalBytes = result.totalBytes || result.dataSizeBytes; // ✅ Correct - use original file size const displayBytes = originalFile.size; // Available result properties: id, timestamp, winc, version, // deadlineHeight, dataCaches, fastFinalityIndexes, public, signature, owner ``` ### Upload Cost Properties If cost calculations fail: **Solution**: Use correct cost object structure: ```javascript // ❌ Problematic - adjustedBytes doesn't exist in cost objects const cost = costs[0]; console.log(`Adjusted: ${cost.adjustedBytes.toLocaleString()}`); // ✅ Correct - use available properties const cost = costs[0]; console.log(`Cost: ${cost.winc} winc`); console.log(`File size: ${originalFile.size.toLocaleString()} bytes`); // Available cost properties: winc (string), adjustments (array) ``` ## Testing Your Implementation ### 1. Basic Functionality Test ```javascript // Test CDN import console.log("Testing Turbo SDK import..."); const turbo = TurboFactory.unauthenticated(); console.log("✅ SDK imported successfully"); // Test rate fetching const rates = await turbo.getFiatRates(); console.log("✅ Rates fetched:", rates); ``` ### 2. Wallet Integration Test - Connect to MetaMask/Wander wallet - Verify address display - Test small file upload ( { // Actual structure: { processedBytes: 4326, // number - bytes processed so far totalBytes: 8652, // number - total bytes to process step: "signing" // string - current step: "signing" or "upload" } } } ``` ## Additional Resources - [Turbo SDK Documentation](https://docs.ardrive.io) - [Browser Wallet Security Guide](https://docs.wander.app) - [Arweave Developer Documentation](https://docs.arweave.org) - [CDN Import Best Practices](https://esm.sh) --- For more advanced implementations, see the [Next.js](./nextjs.mdx) and [Vite](./vite.mdx) framework guides, or explore the [Turbo SDK examples](https://github.com/ardriveapp/turbo-sdk) repository. # Using Turbo in a Browser (/build/guides/using-turbo-in-a-browser) Integrate the **Turbo SDK** directly into your web applications for fast, reliable data uploads to Arweave. Choose the approach that best fits your development workflow and framework preferences. ## What You Can Build **With Turbo SDK in browsers, you can:** - **Upload files directly** from web applications to Arweave - **Pay with different tokens** (AR, Ethereum, and more) - **Integrate with popular wallets** (MetaMask, Wander, ArConnect) - **Build permanent web apps** that store data on Arweave - **Create data marketplaces** and decentralized applications ## Getting Started **Start with the simplest approach** - no build tools required **Key topics:** - CDN imports for instant setup - Wallet integration examples - Production deployment considerations - Error handling and troubleshooting **Full-stack React applications** with server-side rendering **Key topics:** - Webpack polyfill configuration - Client-side component setup - TypeScript integration - Production optimization **Fast development** with modern build tools **Key topics:** - Vite plugin configuration - React and TypeScript setup - Hot module replacement - Bundle optimization ## Why Use Turbo SDK? - **Fast uploads** - Upload data to Arweave in seconds, not minutes - **Multiple payment options** - Pay with AR, Ethereum, or other supported tokens - **Wallet integration** - Seamlessly connect with popular browser wallets - **Reliable infrastructure** - Built on Arweave's permanent storage network - **Developer-friendly** - Simple APIs with comprehensive documentation ## Next Steps } /> } /> } /> # Using Turbo SDK with Next.js (/build/guides/using-turbo-in-a-browser/nextjs) # Using Turbo SDK with Next.js **Firefox Compatibility**: Some compatibility issues have been reported with the Turbo SDK in Firefox browsers. At this time the below framework examples may not behave as expected in Firefox. ## Overview This guide demonstrates how to configure the `@ardrive/turbo-sdk` in a Next.js application with proper polyfills for client-side usage. Next.js uses webpack under the hood, which requires specific configuration to handle Node.js modules that the Turbo SDK depends on. **Polyfills**: Polyfills are required when using the Turbo SDK in Next.js applications. The SDK relies on Node.js modules like `crypto`, `buffer`, `process`, and `stream` that are not available in the browser by default. ## Prerequisites - Next.js 13+ (with App Router recommended) - Next.js 15 users: See updated configuration examples throughout this guide - Node.js 18+ - Basic familiarity with Next.js configuration and webpack Install the main Turbo SDK package: ```bash npm install @ardrive/turbo-sdk ``` Add required polyfill packages for browser compatibility: ```bash npm install --save-dev crypto-browserify stream-browserify process buffer ``` **Wallet Integration Dependencies**: The Turbo SDK includes `@dha-team/arbundles` as a peer dependency, which provides the necessary signers for browser wallet integration (like `InjectedEthereumSigner` and `ArconnectSigner`). You can import these directly without additional installation. **Next.js 15+**: Use TypeScript config format (`next.config.ts`) with ESM exports. For earlier versions, use `next.config.js` with `module.exports`. Create or update your `next.config.ts` file to include the necessary polyfills: ```typescript const nextConfig: NextConfig = { webpack: (config, { isServer, webpack }) => { // Only configure polyfills for client-side bundles if (!isServer) { config.resolve.fallback = { ...config.resolve.fallback, crypto: require.resolve("crypto-browserify"), stream: require.resolve("stream-browserify"), buffer: require.resolve("buffer"), process: require.resolve("process/browser"), fs: false, net: false, tls: false, }; // Provide global process and Buffer config.plugins.push( new webpack.ProvidePlugin({ process: "process/browser", Buffer: ["buffer", "Buffer"], }) ); // CRITICAL: Handle node: protocol imports (e.g., node:stream, node:crypto) // The Turbo SDK uses modern Node.js imports that webpack doesn't understand config.plugins.push( new webpack.NormalModuleReplacementPlugin( /^node:/, (resource: any) => { const module = resource.request.replace(/^node:/, ""); const fallbackMap: Record = { crypto: "crypto-browserify", stream: "stream-browserify", buffer: "buffer", process: "process/browser", }; resource.request = fallbackMap[module] || module; } ) ); } return config; }, }; ``` **Why `NormalModuleReplacementPlugin` is required**: The Turbo SDK uses `node:stream`, `node:crypto`, etc. imports. Without this plugin, you'll get `UnhandledSchemeError: Reading from "node:stream"` build errors. **Required for SDK Compatibility**: The Turbo SDK requires ES2017 features. Set your TypeScript `target` to `ES2017` or higher. Update your `tsconfig.json`: ```json { "compilerOptions": { "target": "ES2017", "lib": ["es2017", "dom", "dom.iterable", "esnext"], "module": "esnext", "moduleResolution": "bundler", "jsx": "preserve", "paths": { "@/*": ["./src/*"] } // ... other options } } ``` **TypeScript Wallet Types** Create a `types/wallet.d.ts` file to properly type wallet objects: ```typescript // types/wallet.d.ts interface Window { ethereum?: { request: (args: { method: string; params?: any[] }) => Promise; on?: (event: string, handler: (...args: any[]) => void) => void; removeListener?: (event: string, handler: (...args: any[]) => void) => void; isMetaMask?: boolean; }; arweaveWallet?: { connect: (permissions: string[]) => Promise; disconnect: () => Promise; getActiveAddress: () => Promise; getPermissions: () => Promise; sign: (transaction: any) => Promise; getPublicKey: () => Promise; }; } ``` Select between **client-side wallet** or **server-side signing** integration based on your use case: ### When to Use Client-Side Wallets (Steps 6-7) Use MetaMask or Wander wallet integration when: - Users control their own wallets and pay for their uploads - Building a decentralized app (dApp) where users own their data - Users need to sign transactions with their private keys - You want users to manage their own Turbo credits ### When to Use Server-Side Signing (Step 8) Use server-side signing when: - **Your application pays for uploads** (most common for Next.js apps) - Background jobs or automated uploads - API-driven uploads without user wallets - You need centralized control over signing - Building traditional web apps with server-side authentication **Most Next.js apps use server-side signing**. If you're building a typical web application where your backend handles uploads, skip to Step 8. **Never expose private keys in browser applications!** Client-side patterns use browser wallets (no private keys in code). Server-side patterns keep private keys secure on the server. Create a React component for MetaMask wallet integration: For MetaMask integration, you'll need to use `InjectedEthereumSigner` from `@dha-team/arbundles`, which is available as a peer dependency through the Turbo SDK. ```tsx "use client"; const [connected, setConnected] = useState(false); const [address, setAddress] = useState(""); const [uploading, setUploading] = useState(false); const [uploadResult, setUploadResult] = useState(null); const connectMetaMask = useCallback(async () => { try { if (!window.ethereum) { alert("MetaMask is not installed!"); return; } // Request account access await window.ethereum.request({ method: "eth_requestAccounts", }); // Get the current account const accounts = await window.ethereum.request({ method: "eth_accounts", }); if (accounts.length > 0) { setAddress(accounts[0]); setConnected(true); // Log current chain for debugging const chainId = await window.ethereum.request({ method: "eth_chainId", }); console.log("Connected to chain:", chainId); } } catch (error) { console.error("Failed to connect to MetaMask:", error); } }, []); const uploadWithMetaMask = async (event) => { const file = event.target.files?.[0]; if (!file || !connected) return; setUploading(true); try { // Create a provider wrapper for InjectedEthereumSigner const providerWrapper = { getSigner: () => ({ signMessage: async (message: string | Uint8Array) => { const accounts = await window.ethereum!.request({ method: "eth_accounts", }); if (accounts.length === 0) { throw new Error("No accounts available"); } // Convert message to hex if it's Uint8Array const messageToSign = typeof message === "string" ? message : "0x" + Array.from(message) .map((b) => b.toString(16).padStart(2, "0")) .join(""); return await window.ethereum!.request({ method: "personal_sign", params: [messageToSign, accounts[0]], }); }, }), }; // Create the signer using InjectedEthereumSigner const signer = new InjectedEthereumSigner(providerWrapper); const turbo = TurboFactory.authenticated({ signer, token: "ethereum", // Important: specify token type for Ethereum }); // Upload file with progress tracking const result = await turbo.uploadFile({ fileStreamFactory: () => file.stream(), fileSizeFactory: () => file.size, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type }, { name: "App-Name", value: "My-Next-App" }, { name: "Funded-By", value: "Ethereum" }, ], }, events: { onProgress: ({ totalBytes, processedBytes, step }) => { console.log( `${step}: ${Math.round((processedBytes / totalBytes) * 100)}%` ); }, onError: ({ error, step }) => { console.error(`Error during ${step}:`, error); console.error("Error details:", JSON.stringify(error, null, 2)); }, }, }); setUploadResult(result); } catch (error) { console.error("Upload failed:", error); console.error("Error details:", JSON.stringify(error, null, 2)); alert(`Upload failed: ${error.message}`); } finally { setUploading(false); } }; return ( MetaMask Upload {!connected ? ( Connect MetaMask ) : ( ✅ Connected: {address.slice(0, 6)}...{address.slice(-4)} Select File to Upload: {uploading && ( 🔄 Uploading... Please confirm transaction in MetaMask )} {uploadResult && ( ✅ Upload Successful! Transaction ID: {uploadResult.id} Data Size: {uploadResult.totalBytes} bytes )} )} ); } ``` Create a React component for Wander wallet integration: ```tsx "use client"; const [connected, setConnected] = useState(false); const [address, setAddress] = useState(""); const [uploading, setUploading] = useState(false); const [uploadResult, setUploadResult] = useState(null); const connectWanderWallet = useCallback(async () => { try { if (!window.arweaveWallet) { alert("Wander wallet is not installed!"); return; } // Required permissions for Turbo SDK const permissions = [ "ACCESS_ADDRESS", "ACCESS_PUBLIC_KEY", "SIGN_TRANSACTION", "SIGNATURE", ]; // Connect to wallet await window.arweaveWallet.connect(permissions); // Get wallet address const walletAddress = await window.arweaveWallet.getActiveAddress(); setAddress(walletAddress); setConnected(true); } catch (error) { console.error("Failed to connect to Wander wallet:", error); } }, []); const uploadWithWanderWallet = async (event) => { const file = event.target.files?.[0]; if (!file || !connected) return; setUploading(true); try { // Create ArConnect signer using Wander wallet const signer = new ArconnectSigner(window.arweaveWallet); const turbo = TurboFactory.authenticated({ signer }); // Note: No need to specify token for Arweave as it's the default // Upload file with progress tracking const result = await turbo.uploadFile({ fileStreamFactory: () => file.stream(), fileSizeFactory: () => file.size, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type }, { name: "App-Name", value: "My-Next-App" }, { name: "Funded-By", value: "Arweave" }, ], }, events: { onProgress: ({ totalBytes, processedBytes, step }) => { console.log( `${step}: ${Math.round((processedBytes / totalBytes) * 100)}%` ); }, onError: ({ error, step }) => { console.error(`Error during ${step}:`, error); }, }, }); setUploadResult(result); } catch (error) { console.error("Upload failed:", error); alert(`Upload failed: ${error.message}`); } finally { setUploading(false); } }; return ( Wander Wallet Upload {!connected ? ( Connect Wander Wallet ) : ( ✅ Connected: {address.slice(0, 6)}...{address.slice(-4)} Select File to Upload: {uploading && ( 🔄 Uploading... Please confirm transaction in Wander wallet )} {uploadResult && ( ✅ Upload Successful! Transaction ID: {uploadResult.id} Data Size: {uploadResult.totalBytes} bytes )} )} ); } ``` **Recommended for most Next.js applications**. This pattern keeps your private keys secure on the server while allowing uploads from the client. ### Architecture - **Client**: Uses `walletAdapter` with Turbo SDK, delegates signing to API route - **Server**: API route signs data with `ethers.js` Wallet, returns signature - **Security**: Private keys stay server-side, never exposed to browser ### Installation Install ethers.js (if not already installed): ```bash npm install ethers ``` ### Environment Setup #### Getting a Private Key You'll need an Ethereum-compatible private key for server-side signing: **For Testing/Development:** - Generate a new wallet with any Ethereum wallet (MetaMask, Rainbow, etc.) - Export the private key (usually in wallet settings under "Show private keys") - Copy the private key (starts with `0x`) **For Production:** - Use a dedicated wallet for your application - Fund it with the appropriate token (USDC on Base, ETH, etc.) - Never use a personal wallet with significant funds **Quick Test Key Generation (Node.js):** ```bash node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" # Add 0x prefix: 0x[generated_hex] ``` Create a `.env.local` file in your project root: ```bash # .env.local (NEVER commit this file to git!) ETHEREUM_PRIVATE_KEY=0xYourPrivateKeyHere ``` **Security**: Add `.env.local` to your `.gitignore` file. Never commit private keys to version control. **Funding Your Wallet**: After setting up your private key, you'll need to fund the corresponding wallet address with Turbo credits. Visit [https://turbo.ardrive.io](https://turbo.ardrive.io) to add credits to your wallet. ### Token Selection Choose your token type based on the blockchain you're using: - `"base-usdc"` - USDC on Base Network (recommended for low fees) - `"ethereum"` - ETH on Ethereum Mainnet - `"polygon-usdc"` - USDC on Polygon Network - `"matic"` or `"pol"` - MATIC/POL on Polygon Network - `"solana"` - SOL on Solana - `"solana-usdc"` - USDC on Solana Your private key must match the blockchain you select. For example, if using `"base-usdc"`, use an Ethereum-compatible private key. ### Server API Route Create `app/api/sign/route.ts`: ```typescript import "server-only"; // Validate that private key exists in environment variables if (!process.env.ETHEREUM_PRIVATE_KEY) { throw new Error( "ETHEREUM_PRIVATE_KEY is not set. Please add it to your .env.local file." ); } const wallet = new Wallet(process.env.ETHEREUM_PRIVATE_KEY); async function signData(data: string | Uint8Array): Promise { // ethers.js handles both UTF-8 strings and binary data return await wallet.signMessage(data); } function getPublicKey(): string { // Return uncompressed public key (keep '04' prefix, remove only '0x') return wallet.signingKey.publicKey.slice(2); } return NextResponse.json({ publicKey: getPublicKey() }); } const body = await req.json() as { signatureData: string; isHex?: boolean; }; if (!body?.signatureData) { return NextResponse.json( { error: "Missing signatureData" }, { status: 400 } ); } // Handle both string messages (UTF-8) and binary data (hex-encoded) const dataToSign = body.isHex ? Buffer.from(body.signatureData, "hex") : body.signatureData; const signature = await signData(dataToSign); return NextResponse.json({ signature }); } ``` **Next.js 15 Requirement**: Route handlers can ONLY ### Client Component Create `app/upload/component.tsx`: ```typescript "use client"; // Buffer is available globally from webpack polyfills configured in next.config.ts // No import needed - it's provided by the ProvidePlugin // API endpoint for server-side signing (corresponds to app/api/sign/route.ts) const signEndpoint = "/api/sign"; const [uploading, setUploading] = useState(false); // Create turbo instance with useMemo to avoid recreating on every render // This prevents SSR issues and unnecessary re-initialization const turbo = useMemo(() => { return TurboFactory.authenticated({ token: "base-usdc", // Match the token type from your environment walletAdapter: { getSigner: () => { return { signMessage: async (_message: string | Uint8Array) => { // Determine if message is binary (Uint8Array) or string const isHex = typeof _message !== "string"; const message = isHex ? Buffer.from(_message).toString("hex") : _message; const res = await fetch(signEndpoint, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ signatureData: message, isHex }), }); if (!res.ok) { throw new Error(`Signing failed: ${res.statusText}`); } const { signature } = await res.json(); return signature; }, sendTransaction: async () => { throw new Error("Not implemented"); }, provider: { // Fetch public key from server getPublicKey: async () => { const res = await fetch(signEndpoint); const { publicKey } = await res.json(); return publicKey; }, getSigner: () => ({ // Return address (same as public key for this implementation) getAddress: async () => { const res = await fetch(signEndpoint); const { publicKey } = await res.json(); return publicKey; }, // Sign typed data (used for certain Turbo operations) _signTypedData: async ( _domain: never, _types: never, message: { address: string; "Transaction hash": Uint8Array; } ) => { const convertedMsg = Buffer.from( message["Transaction hash"] ).toString("hex"); const res = await fetch(signEndpoint, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ signatureData: convertedMsg, isHex: true }), }); if (!res.ok) { throw new Error(`Signing failed: ${res.statusText}`); } const { signature } = await res.json(); return signature; }, }), _ready: () => {}, }, } as unknown as EthereumWalletSigner; }, }, }); }, []); // Upload string data const handleUploadText = async () => { setUploading(true); try { const result = await turbo.upload({ data: "Hello, ArDrive Turbo!", }); console.log("Upload successful! ID:", result.id); alert(`Upload successful! ID: ${result.id}`); } catch (error) { console.error("Upload failed:", error); alert(`Upload failed: ${(error as Error).message}`); } finally { setUploading(false); } }; // Upload file const handleFileUpload = async (e: React.ChangeEvent) => { const file = e.target.files?.[0]; if (!file) return; setUploading(true); try { const result = await turbo.uploadFile({ fileStreamFactory: () => file.stream(), fileSizeFactory: () => file.size, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type }, { name: "File-Name", value: file.name }, ], }, }); console.log("File upload successful! ID:", result.id); alert(`File uploaded! ID: ${result.id}`); } catch (error) { console.error("Upload failed:", error); alert(`Upload failed: ${(error as Error).message}`); } finally { setUploading(false); } }; return ( {uploading ? "Uploading..." : "Upload Text"} ); } ``` **Styling**: This is a basic example. Add your own styles, labels, and accessibility features as needed for your application. ### Key Implementation Details 1. **`useMemo` for Turbo Instance**: Creating the turbo instance inside `useMemo` prevents it from being recreated on every render and avoids SSR issues. The turbo client is initialized once when the component mounts. 2. **Provider Implementation**: The `provider` object is required for Ethereum wallets and includes: - `getPublicKey()` - Fetches public key from `GET /api/sign` - `getSigner().getAddress()` - Returns the wallet address (same as public key) - `getSigner()._signTypedData()` - Signs EIP-712 typed data for certain Turbo operations - `_ready()` - Lifecycle method (can be empty) 3. **Buffer Global**: `Buffer` is available globally from the webpack polyfills you configured in Step 3. No import needed - the `ProvidePlugin` makes it available everywhere. 4. **`isHex` Flag**: The SDK passes both UTF-8 strings (e.g., during key setup) and binary data (file content). Use this flag to differentiate handling on the server. 5. **File Uploads**: Use `uploadFile()` with `fileStreamFactory` and `fileSizeFactory` for files. Use `upload()` with `data` for strings or Blobs. 6. **Public Key Format**: Server returns uncompressed public key with `04` prefix byte. Use `.slice(2)` to remove only `0x`, keeping `04`. 7. **Signature Format**: `ethers.js` produces EIP-191 signatures compatible with the SDK's verification. 8. **Token Matching**: Ensure the `token` in your client matches your server's blockchain (e.g., both use `"base-usdc"`). 9. **Security**: Private keys stay in `.env.local` on the server, never exposed to the browser. ### Complete Working Example To use this component, add it to a page: ```typescript // app/page.tsx return ( Upload to ArDrive Turbo ); } ``` Now you can: - Click "Upload Text" to upload a string - Use the file input to upload any file - Both use server-side signing for security ## Common Issues and Solutions ### Build Errors If you encounter build errors related to missing modules: 1. **"UnhandledSchemeError: Reading from 'node:stream'"** ⚠️ **CRITICAL** - The Turbo SDK uses `node:` protocol imports that webpack doesn't understand - **Solution**: Add `NormalModuleReplacementPlugin` to your webpack config (see Step 3) - This is the most common build error in Next.js 15 2. **"Module not found: Can't resolve 'fs'"** - Ensure `fs: false` is set in your webpack fallback configuration 3. **"process is not defined"** - Make sure you have the `ProvidePlugin` configuration for process 4. **"Buffer is not defined"** - Verify the Buffer polyfill is properly configured in `ProvidePlugin` ### Next.js 15 Specific Issues 1. **"Type does not satisfy constraint" in route handlers** - Next.js 15 requires route files to ONLY - **Solution**: Remove `export` from helper functions ```typescript // ❌ INCORRECT // ✅ CORRECT async function helper() { ... } // Not exported ``` 2. **TypeScript compilation errors** - **Solution**: Set `"target": "ES2017"` in `tsconfig.json` - The SDK requires ES2017 features ### Runtime Errors 1. **"crypto.getRandomValues is not a function"** - This usually indicates the crypto polyfill isn't working. Double-check your webpack configuration. 2. **"TypeError: e.startsWith is not a function"** - This indicates incorrect signer usage. For MetaMask integration, use `InjectedEthereumSigner` from `@dha-team/arbundles`, not `EthereumSigner`. - `EthereumSigner` expects a private key string, while `InjectedEthereumSigner` expects a provider wrapper. 3. **"No accounts available" during wallet operations** - Ensure the wallet is properly connected before attempting operations - Add validation to check account availability after connection 4. **Message signing failures with wallets** - For `InjectedEthereumSigner`, ensure your provider wrapper correctly implements the `getSigner()` method - Handle both string and Uint8Array message types in your `signMessage` implementation - Use MetaMask's `personal_sign` method with proper parameter formatting 5. **Server-side rendering issues** - Always use `'use client'` directive for components that use the Turbo SDK - Consider dynamic imports with `ssr: false` for complex cases: ```tsx const TurboUploader = dynamic(() => import("./TurboUploader"), { ssr: false, }); ``` ### Server-Side Signing Issues 1. **"Invalid Data Item" errors (400 Bad Request)** - Usually caused by incorrect signature format or public key issues - **Common causes**: - Public key missing `04` prefix byte - Incorrect handling of string vs binary data - Wrong signature encoding - Token mismatch between client and server 2. **Turbo Instance SSR/Re-render Issues** ```typescript // ❌ INCORRECT - Creates instance at module level const turbo = TurboFactory.authenticated({ ... }); // Uses turbo - will cause SSR issues } // ✅ CORRECT - Create inside component with useMemo const turbo = useMemo(() => TurboFactory.authenticated({ ... }) , []); } ``` - Creating at module level runs on server during SSR - Creates new instance on every render without memoization - **Solution**: Always use `useMemo` hook inside component 3. **Public Key Format Issues** ```typescript // ❌ INCORRECT - Removes both '0x' AND '04' wallet.signingKey.publicKey.slice(4) // ✅ CORRECT - Removes only '0x', keeps '04' prefix wallet.signingKey.publicKey.slice(2) ``` - The uncompressed Ethereum public key format is `04 + x_coord + y_coord` - The SDK expects this full format 4. **String vs Binary Data Handling** - The SDK passes BOTH types to `signMessage`: - **UTF-8 strings** (e.g., "sign this message to connect to Bundlr.Network") during setup - **Binary data** (Uint8Array) for actual file content - **Solution**: Use `isHex` flag to differentiate: ```typescript const isHex = typeof message !== "string"; const messageStr = isHex ? Buffer.from(message).toString("hex") : message; // Send to server with isHex flag fetch("/api/sign", { body: JSON.stringify({ signatureData: messageStr, isHex }) }); ``` 5. **"Buffer is not defined"** - `Buffer` should be available globally from webpack polyfills - **Solution**: Verify `ProvidePlugin` is configured correctly in `next.config.ts` (Step 3) - No import needed - it's provided globally 6. **Using TypedEthereumSigner instead of ethers.js Wallet** - `TypedEthereumSigner` from `@dha-team/arbundles` produces incompatible signature format - **Solution**: Use `ethers.js` Wallet for server-side signing - Ethers produces proper EIP-191 signatures with recovery parameter 7. **Token Mismatch Errors** - Client and server must use the same token type - **Solution**: Ensure `token: "base-usdc"` matches between: - Client `TurboFactory.authenticated()` - Server environment (private key's blockchain) - Your Turbo account funding ### Wallet Integration Issues 1. **Incorrect Signer Import** ```tsx // ❌ INCORRECT - For Node environments // ✅ CORRECT - For browser wallets ``` 2. **Provider Interface Mismatch** ```tsx // ❌ INCORRECT - window.ethereum doesn't have getSigner() const signer = new InjectedEthereumSigner(window.ethereum); // ✅ CORRECT - Use a provider wrapper const providerWrapper = { getSigner: () => ({ signMessage: async (message: string | Uint8Array) => { // Implementation here }, }), }; const signer = new InjectedEthereumSigner(providerWrapper); ``` 3. **Missing Dependencies** If you encounter import errors for `@dha-team/arbundles`, note that it's available as a peer dependency through `@ardrive/turbo-sdk`. You may need to ensure it's properly resolved in your build process. ## Best Practices 1. **Use Client Components**: Always mark components using the Turbo SDK with `'use client'` 2. **Error Handling**: Implement proper error handling for network requests and wallet interactions 3. **Environment Variables**: Store sensitive configuration in environment variables: ```javascript // next.config.js const nextConfig = { env: { TURBO_UPLOAD_URL: process.env.TURBO_UPLOAD_URL, TURBO_PAYMENT_URL: process.env.TURBO_PAYMENT_URL, }, // ... webpack config }; ``` 4. **Bundle Size**: Consider code splitting for large applications to reduce bundle size 5. **Wallet Security**: - **Never expose private keys** in client-side code - Always use browser wallet integrations (MetaMask, Wander, etc.) - Request only necessary permissions from wallets - Validate wallet connections before use - Handle wallet disconnection gracefully ## Production Deployment Checklist For production deployments: 1. **Verify polyfills work correctly** in your build environment 2. **Test wallet connections** with various providers (Wander, MetaMask, etc.) 3. **Monitor bundle sizes** to ensure polyfills don't significantly increase your app size 4. **Use environment-specific configurations** for different Turbo endpoints 5. **Implement proper error boundaries** for wallet connection failures 6. **Add loading states** for wallet operations to improve UX 7. **Test across different browsers** to ensure wallet compatibility ## Implementation Verification To verify your MetaMask integration is working correctly: 1. **Check Console Logs**: After connecting to MetaMask, you should see: ``` Connected to chain: 0x1 (or appropriate chain ID) ``` 2. **Test Balance Retrieval**: Add this to verify your authenticated client works: ```tsx // After creating authenticated turbo client const balance = await turbo.getBalance(); console.log("Current balance:", balance); ``` 3. **Verify Signer Setup**: Your implementation should: - Use `InjectedEthereumSigner` from `@dha-team/arbundles` - Include a proper provider wrapper with `getSigner()` method - Handle both string and Uint8Array message types - Use MetaMask's `personal_sign` method 4. **Common Success Indicators**: - No `TypeError: e.startsWith is not a function` errors - Successful wallet connection and address display - Ability to fetch balance without errors - Upload operations work with proper MetaMask transaction prompts ## Additional Resources - [Turbo SDK Documentation](https://docs.ardrive.io) - [Web Usage Examples](https://docs.ardrive.io) - [Next.js Webpack Configuration](https://nextjs.org/docs/pages/api-reference/next-config-js/webpack) - [ArDrive Examples Repository](https://github.com/ardriveapp/turbo-sdk) --- For more examples and advanced usage patterns, refer to the [Turbo SDK examples directory](https://github.com/ardriveapp/turbo-sdk) or the main [SDK documentation](https://docs.ardrive.io). # Using Turbo SDK with Vite (/build/guides/using-turbo-in-a-browser/vite) # Using Turbo SDK with Vite **Firefox Compatibility**: Some compatibility issues have been reported with the Turbo SDK in Firefox browsers. At this time the below framework examples may not behave as expected in Firefox. ## Overview This guide demonstrates how to configure the `@ardrive/turbo-sdk` in a Vite application with proper polyfills for client-side usage. Vite provides excellent support for modern JavaScript features and can be easily configured to work with the Turbo SDK through plugins. **Polyfills**: Vite simplifies polyfill management compared to other bundlers. The `vite-plugin-node-polyfills` plugin handles most of the complexity automatically. ## Prerequisites - Vite 5+ - Node.js 18+ - React 18+ (or your preferred framework) - Basic familiarity with Vite configuration Install the main Turbo SDK package: ```bash npm install @ardrive/turbo-sdk ``` Add the Vite node polyfills plugin for browser compatibility: ```bash npm install --save-dev vite-plugin-node-polyfills ``` **Wallet Integration Dependencies**: The Turbo SDK includes `@dha-team/arbundles` as a peer dependency, which provides the necessary signers for browser wallet integration (like `InjectedEthereumSigner` and `ArconnectSigner`). You can import these directly without additional installation. Add React and TypeScript dependencies (if using React): ```bash npm install react react-dom npm install --save-dev @vitejs/plugin-react @types/react @types/react-dom ``` Create or update your `vite.config.js` file: ```javascript base: "/", plugins: [ react(), nodePolyfills({ // Enable specific polyfills for Turbo SDK requirements include: ["crypto", "stream", "buffer", "process"], globals: { Buffer: true, global: true, process: true, }, }), ], define: { // Define globals for browser compatibility global: "globalThis", }, }); ``` If you're using TypeScript, update your `tsconfig.json`: ```json { "compilerOptions": { "target": "ESNext", "lib": ["DOM", "DOM.Iterable", "ESNext"], "allowJs": true, "skipLibCheck": true, "esModuleInterop": true, "allowSyntheticDefaultImports": true, "strict": true, "forceConsistentCasingInFileNames": true, "module": "ESNext", "moduleResolution": "Bundler", "isolatedModules": true, "jsx": "react-jsx", "paths": { "buffer/": ["./node_modules/vite-plugin-node-polyfills/shims/buffer"] } }, "include": ["src"] } ``` **TypeScript Wallet Types** Create a `types/wallet.d.ts` file to properly type wallet objects: ```typescript // types/wallet.d.ts interface Window { ethereum?: { request: (args: { method: string; params?: any[] }) => Promise; on?: (event: string, handler: (...args: any[]) => void) => void; removeListener?: (event: string, handler: (...args: any[]) => void) => void; isMetaMask?: boolean; }; arweaveWallet?: { connect: (permissions: string[]) => Promise; disconnect: () => Promise; getActiveAddress: () => Promise; getPermissions: () => Promise; sign: (transaction: any) => Promise; getPublicKey: () => Promise; }; } ``` Select between MetaMask or Wander wallet integration: **Never expose private keys in browser applications!** Always use browser wallet integrations for security. Create a React component for MetaMask wallet integration: ```tsx const [connected, setConnected] = useState(false); const [address, setAddress] = useState(""); const [uploading, setUploading] = useState(false); const [uploadResult, setUploadResult] = useState(null); const connectMetaMask = useCallback(async () => { try { if (!window.ethereum) { alert("MetaMask is not installed!"); return; } // Request account access await window.ethereum.request({ method: "eth_requestAccounts", }); // Get the current account const accounts = await window.ethereum.request({ method: "eth_accounts", }); if (accounts.length > 0) { setAddress(accounts[0]); setConnected(true); // Log current chain for debugging const chainId = await window.ethereum.request({ method: "eth_chainId", }); console.log("Connected to chain:", chainId); } } catch (error) { console.error("Failed to connect to MetaMask:", error); } }, []); const uploadWithMetaMask = async (event) => { const file = event.target.files?.[0]; if (!file || !connected) return; setUploading(true); try { // Create a provider wrapper for InjectedEthereumSigner const providerWrapper = { getSigner: () => ({ signMessage: async (message: string | Uint8Array) => { const accounts = await window.ethereum!.request({ method: "eth_accounts", }); if (accounts.length === 0) { throw new Error("No accounts available"); } // Convert message to hex if it's Uint8Array const messageToSign = typeof message === "string" ? message : "0x" + Array.from(message) .map((b) => b.toString(16).padStart(2, "0")) .join(""); return await window.ethereum!.request({ method: "personal_sign", params: [messageToSign, accounts[0]], }); }, }), }; // Create the signer using InjectedEthereumSigner const signer = new InjectedEthereumSigner(providerWrapper); const turbo = TurboFactory.authenticated({ signer, token: "ethereum", // Important: specify token type for Ethereum }); // Upload file with progress tracking const result = await turbo.uploadFile({ fileStreamFactory: () => file.stream(), fileSizeFactory: () => file.size, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type }, { name: "App-Name", value: "My-Vite-App" }, { name: "Funded-By", value: "Ethereum" }, ], }, events: { onProgress: ({ totalBytes, processedBytes, step }) => { console.log( `${step}: ${Math.round((processedBytes / totalBytes) * 100)}%` ); }, onError: ({ error, step }) => { console.error(`Error during ${step}:`, error); console.error("Error details:", JSON.stringify(error, null, 2)); }, }, }); setUploadResult(result); } catch (error) { console.error("Upload failed:", error); console.error("Error details:", JSON.stringify(error, null, 2)); alert(`Upload failed: ${error.message}`); } finally { setUploading(false); } }; return ( MetaMask Upload {!connected ? ( Connect MetaMask ) : ( ✅ Connected: {address.slice(0, 6)}...{address.slice(-4)} Select File to Upload: {uploading && ( 🔄 Uploading... Please confirm transaction in MetaMask )} {uploadResult && ( ✅ Upload Successful! Transaction ID: {uploadResult.id} Data Size: {uploadResult.totalBytes} bytes )} )} ); } ``` Create a React component for Wander wallet integration: ```tsx const [connected, setConnected] = useState(false); const [address, setAddress] = useState(""); const [uploading, setUploading] = useState(false); const [uploadResult, setUploadResult] = useState(null); const connectWanderWallet = useCallback(async () => { try { if (!window.arweaveWallet) { alert("Wander wallet is not installed!"); return; } // Required permissions for Turbo SDK const permissions = [ "ACCESS_ADDRESS", "ACCESS_PUBLIC_KEY", "SIGN_TRANSACTION", "SIGNATURE", ]; // Connect to wallet await window.arweaveWallet.connect(permissions); // Get wallet address const walletAddress = await window.arweaveWallet.getActiveAddress(); setAddress(walletAddress); setConnected(true); } catch (error) { console.error("Failed to connect to Wander wallet:", error); } }, []); const uploadWithWanderWallet = async (event) => { const file = event.target.files?.[0]; if (!file || !connected) return; setUploading(true); try { // Create ArConnect signer using Wander wallet const signer = new ArconnectSigner(window.arweaveWallet); const turbo = TurboFactory.authenticated({ signer }); // Note: No need to specify token for Arweave as it's the default // Upload file with progress tracking const result = await turbo.uploadFile({ fileStreamFactory: () => file.stream(), fileSizeFactory: () => file.size, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type }, { name: "App-Name", value: "My-Vite-App" }, { name: "Funded-By", value: "Arweave" }, ], }, events: { onProgress: ({ totalBytes, processedBytes, step }) => { console.log( `${step}: ${Math.round((processedBytes / totalBytes) * 100)}%` ); }, onError: ({ error, step }) => { console.error(`Error during ${step}:`, error); }, }, }); setUploadResult(result); } catch (error) { console.error("Upload failed:", error); alert(`Upload failed: ${error.message}`); } finally { setUploading(false); } }; return ( Wander Wallet Upload {!connected ? ( Connect Wander Wallet ) : ( ✅ Connected: {address.slice(0, 6)}...{address.slice(-4)} Select File to Upload: {uploading && ( 🔄 Uploading... Please confirm transaction in Wander wallet )} {uploadResult && ( ✅ Upload Successful! Transaction ID: {uploadResult.id} Data Size: {uploadResult.totalBytes} bytes )} )} ); } ``` Here's a complete `package.json` example for a Vite + React + Turbo SDK project: ```json { "name": "vite-turbo-app", "version": "0.1.0", "private": true, "type": "module", "scripts": { "dev": "vite", "build": "vite build", "preview": "vite preview", "lint": "eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0" }, "dependencies": { "@ardrive/turbo-sdk": "^1.20.0", "react": "^18.3.1", "react-dom": "^18.3.1" }, "devDependencies": { "@types/react": "^18.3.1", "@types/react-dom": "^18.3.0", "@vitejs/plugin-react": "^4.2.1", "typescript": "^5.3.3", "vite": "^5.2.14", "vite-plugin-node-polyfills": "^0.17.0" } } ``` ## Common Issues and Solutions ### Build Errors 1. **"global is not defined"** - Ensure you have `global: 'globalThis'` in your Vite config's `define` section 2. **Buffer polyfill issues** - Make sure `vite-plugin-node-polyfills` is properly configured with Buffer globals - Add the buffer path mapping in your `tsconfig.json` 3. **Module resolution errors** - Use `moduleResolution: "Bundler"` in TypeScript configuration - Ensure you're importing from `@ardrive/turbo-sdk/web` for browser usage ### Runtime Errors 1. **"process is not defined"** - Enable process globals in the node polyfills plugin configuration 2. **Wallet integration errors** - For MetaMask, use `InjectedEthereumSigner` from `@dha-team/arbundles` - For Wander wallet, use `ArconnectSigner` from the Turbo SDK - Always check wallet availability before attempting connection ### Development Experience 1. **Hot reload issues with wallet connections** - Wallet state may not persist across hot reloads - Consider using localStorage to persist connection state 2. **Console warnings about dependencies** - Some peer dependency warnings are normal for wallet libraries - Focus on runtime functionality rather than dependency warnings ## Best Practices 1. **Development vs Production** - Use debug logs during development: `TurboFactory.setLogLevel('debug')` - Remove debug logs in production builds 2. **Error Handling** - Always wrap wallet operations in try-catch blocks - Provide meaningful error messages to users - Log detailed error information for debugging 3. **Performance** - Initialize Turbo clients once and reuse them - Consider lazy loading wallet integration components - Use loading states for better user experience 4. **Security** - Never expose private keys in browser applications - Always validate wallet connections before operations - Use secure wallet connection methods in production ## Production Deployment Checklist For production builds: 1. **Build optimization** - Vite automatically optimizes builds with tree shaking - Polyfills are only included when needed 2. **Testing** - Test wallet connections across different browsers - Verify polyfills work in production builds - Test with actual wallet extensions 3. **Monitoring** - Monitor bundle sizes to ensure polyfills don't bloat your app - Set up error tracking for wallet connection failures ## Implementation Verification To verify your Vite setup is working correctly: 1. **Check Development Server**: Start your dev server and verify no polyfill errors 2. **Test Wallet Connections**: Ensure both MetaMask and Wander wallet integrations work 3. **Build Verification**: Run `npm run build` and check for any build errors 4. **Bundle Analysis**: Use `vite-bundle-analyzer` to inspect your bundle size ## Additional Resources - [Vite Documentation](https://vitejs.dev/) - [vite-plugin-node-polyfills](https://www.npmjs.com/package/vite-plugin-node-polyfills) - [Turbo SDK Documentation](https://docs.ardrive.io) - [Web Usage Examples](https://docs.ardrive.io) - [ArDrive Examples Repository](https://github.com/ardriveapp/turbo-sdk) --- For more examples and advanced usage patterns, refer to the [Turbo SDK examples directory](https://github.com/ardriveapp/turbo-sdk) or the main [SDK documentation](https://docs.ardrive.io). # Working With Primary Names (/build/guides/working-with-arns/arns-primary-names) Create **web3 identity** using ArNS names. Primary names allow you to use human-readable names as your identity in the Arweave ecosystem, making it easy for others to find and interact with you. ## What Are Primary Names? **Primary names** are ArNS names used as identity that: - **Resolve to wallet addresses** - Link human-readable names to wallet addresses - **Provide web3 identity** - Give users friendly names for their Arweave identity - **Are bidirectional** - Can resolve from name to address or address to name - **Require ownership** - Only the owner of an ArNS name can set it as their primary name - **Enable secure verification** - Ownership requirement ensures identity authenticity - **Work across gateways** - Accessible from any ar.io gateway - **Are unique** - A name cannot be the Primary Name for more than one wallet ## How It Works ### 1. Identity Registration **Register a primary name:** - Choose a unique name (e.g., `jonniesparkles`) - Pay the Primary Name fee - Link the name to your wallet address - Use as your web3 identity The Primary Name fee is equivalent to a single undername purchase on a 51-character name of the same purchase type, adjusted by the current Demand Factor. ### 2. Bidirectional Resolution **Name to address resolution:** - `jonniesparkles` → `OU48aJtcq3KjsEqSUWDVpynh1xP2Y1VI-bwiSukAktU` - Others can find your wallet using your name - Use in dApps and applications **Address to name resolution:** - `OU48aJtcq3KjsEqSUWDVpynh1xP2Y1VI-bwiSukAktU` → `jonniesparkles` - Find the name associated with any wallet - Verify identity in transactions ### 3. Application Integration **Use in supported apps:** - **Send tokens to "jonniesparkles"** instead of copying long wallet addresses - **Display friendly names** as usernames when connecting wallets - **Apps resolve names** to wallet addresses using the ar.io SDK - **Seamless user experience** with human-readable identifiers ## Basic Integration ### Using the ar.io SDK **Get a primary name by address:** ```javascript const ario = new ARIO(); // Get the primary name for a wallet address const nameData = await ario.getPrimaryName({ address: "OU48aJtcq3KjsEqSUWDVpynh1xP2Y1VI-bwiSukAktU", }); console.log(nameData.name); // e.g., "jonniesparkles" ``` **Get primary name data:** ```javascript const ario = new ARIO(); // Get primary name data for a name const nameData = await ario.getPrimaryName({ name: "jonniesparkles", }); console.log(nameData.owner); // e.g., "OU48aJtcq3KjsEqSUWDVpynh1xP2Y1VI-bwiSukAktU" console.log(nameData.name); // e.g., "jonniesparkles" ``` ## How Apps Use Primary Names **Token transfers:** - Send tokens to "jonniesparkles" instead of copying `OU48aJtcq3KjsEqSUWDVpynh1xP2Y1VI-bwiSukAktU` - Apps automatically resolve the name to the wallet address - Much more user-friendly than long wallet addresses **User interfaces:** - Display "jonniesparkles" as username when wallet is connected - Show friendly names in transaction histories - Make interactions more personal and memorable **Developer integration:** - Use the [ar.io SDK](/sdks/ar-io-sdk/primary-names#getprimaryname) to resolve names - Support primary names in your dApp - Enhance user experience with human-readable identifiers - **Trust identity ownership** - Only name owners can set primary names, ensuring secure verification **Ownership controls:** - A wallet can have one Primary Name at a time - A name can only be the Primary Name for one wallet - The base name's ANT owner can remove any Primary Name set on one of its undernames ## Benefits - **Web3 identity** - Use human-readable names as your identity - **Easy discovery** - Others can find you by name instead of wallet address - **Bidirectional resolution** - Resolve name to address or address to name - **Secure verification** - Only name owners can set primary names, preventing impersonation - **Network uniqueness** - A name can identify only one wallet as its Primary Name - **Permanent ownership** - Own your identity forever - **App integration** - Works in any app that supports primary names ## Ready to Learn More? } > Learn about using undernames for website versioning. } > Use the ar.io SDK to programmatically manage ArNS records. } > Learn how to deploy and host permanent apps on Arweave. # Buying and managing an ArNS name in ar.io Console (/build/guides/working-with-arns/buy-manage-arns-console) The [ar.io Console](https://console.ar.io) registers ArNS names and manages them alongside uploads, deploys and credits. This guide describes Console 4.11.0. To use the standalone ArNS app instead, see [Purchasing an ArNS name using arns.ar.io](/build/guides/working-with-arns/purchase-arns-ui). ## Before you start An ArNS name is owned by a **Solana wallet**. The Console gets one for you in one of these ways: - **Sign in with a Solana wallet**, such as Phantom, Solflare, or the Solana account in MetaMask. That wallet owns your names. - **Sign in with email.** The Console creates a Solana wallet for your account. - **Sign in with an Arweave or Ethereum wallet, then link a Solana wallet.** Your signed-in wallet stays your account and pays with its credits; the linked Solana wallet owns the names and approves changes to them. Link one from the Account page, or when the checkout asks. **Credits are spent from the account you signed in with**, not from the wallet that owns the name. On an Arweave or Ethereum session, top up the signed-in account, not the linked Solana wallet. ## Register a name Go to [console.ar.io/arns](https://console.ar.io/arns) and search for the name you want. If it is available, choose a **lease** of 1 to 5 years or **Permabuy**, a permanent registration. Optional: choose what the name points at. Pick one of your recent deploys, Pages or uploads, or paste an Arweave transaction ID. It is set as the name is registered, at no extra cost. Leave it alone to point the name at the default placeholder, and change it later from the name's page. Choose how to pay: - **Credits**: paid from your credit balance. - **Card**: paid by card through Stripe. The card buys credits first (at least $5; anything over the price stays on your balance), then the name is bought from those credits with one wallet approval. - **Crypto**: a token from the list. Every token except ARIO, such as SOL or USDC on Solana, is sent from your signed-in wallet to buy credits first, then the name is bought from those credits: two wallet approvals, with the amount worked out for you. **ARIO is a row in the Crypto list, and works differently.** It is paid from the Solana wallet that owns the name (the linked wallet on an Arweave or Ethereum session) straight to the ArNS registry, with no Turbo infrastructure fee. That wallet also needs a little SOL for Solana account rent and the network fee. ARIO is marked "Best price" when it is the cheapest route for that name, counting the SOL. ARIO is the only route where your wallet pays the Solana costs of the purchase; with credits, a card or another token, Turbo pays them. Sending SOL or USDC on Solana still spends a little SOL on that transfer's own network fee. The checkout shows the costs before you pay. Approve each step in your wallet when it asks. The name appears under [Manage Domains](https://console.ar.io/my-domains). ### If a purchase does not finish If a purchase paid with credits, a card, or a token does not finish because you closed the wallet prompt, approved too late, or left the page, [Manage Domains](https://console.ar.io/my-domains) shows what happened to that attempt and its credits: when they return to your balance, or that they are already back. Once the credits are back, it offers **Try again**. The notice appears in the browser where you started the purchase, for 30 minutes. ## Manage a name Open a name from [Manage Domains](https://console.ar.io/my-domains), or go to `console.ar.io/domains/NAME`. Replace NAME with the name. Each section of the page has its own actions: | Section | Actions | | --- | --- | | Overview | **Renew or upgrade** a lease: extend it, make it permanent, or add undername slots. A permanent name has **Add undername slots** | | Details | **Edit** the name's nickname, ticker, logo, description, and keywords | | Ownership | **Transfer** the name to another wallet, **Reassign** it to a different name token, or **Release** a permanent name back to the registry | | Controllers | **Manage** the wallets allowed to edit its records | | Records | **Add record**, edit a record with its pencil, or remove an undername | The header also has **Set as primary**, which makes the name your primary name, and **Visit**, which opens it. Transfer, Reassign, and Release are available only to the owner, and each asks you to confirm before it goes ahead. A transfer cannot be undone. ### Edit records The Records table lists the root record (`@`, the name itself) and every undername. Undernames use an underscore: `UNDERNAME_NAME.ar.io`. Select a record's name to open what it serves. To change where the name points, select the pencil on the `@` row. To add an undername, select **Add record** and enter the undername. Enter the **Target (Arweave TX ID)** and the **Cache time (TTL, seconds)**, then save. The change shows in the table as soon as it is saved, while the page confirms it against the chain. ### Who pays for a change On a name's page, record changes, transfers, and controller changes have a small cost, paid one of two ways: - **If the wallet that owns the name holds enough SOL**, it signs the change and pays the Solana network fee, usually a fraction of a cent. A transfer or a new controller can also pay rent on accounts it creates, up to about 0.005 SOL, so those use SOL only when the wallet holds at least that much. **Pay with credits instead** switches to credits when both would work. - **Otherwise, credits pay**, and Turbo pays the Solana fee. For these changes, a wallet with no SOL, such as an email account that holds no SOL, uses credits. **Editing the name's details, setting a primary name, reassigning and releasing are not paid by Turbo.** Your wallet signs and pays a small Solana fee, so it needs a little SOL. Setting a primary name also charges an ARIO fee from that wallet. For record changes, transfers, and controller changes, the cost is shown before you confirm. Editing details, setting a primary name, reassigning, and releasing say that your wallet pays a Solana fee, without an amount. IPFS targets and record priority are always signed by your wallet and need a little SOL; the editor says so before you save. **Controllers** can edit records, edit the name's details, and renew or upgrade it, but cannot set it as their primary name, transfer, reassign, release, or manage controllers. For record and detail changes, a controller signs and pays the Solana fee from their own wallet. Renewing, upgrading, and adding undername slots are paid with credits, a card, a token, or ARIO. A card pays for the change in one step. A token buys credits first; once they arrive, select **Continue** to pay for the change from them. ## Assign a name to an upload, deploy or capture When you upload a file, deploy a site, capture a page, or publish a Page in the Console, you can point one of your names at it. For a name you own, assigning it here is paid in credits; a controller's wallet signs and pays the Solana fee. After an upload, deploy, or capture, the success message has **Visit**, which opens the name, and **Manage NAME**, which opens the name's page. ## Gateway operators A gateway operator gets a 20% discount on ArNS purchases, renewals, upgrades, and undername slots **when the wallet paying in ARIO is the gateway's operator wallet**, and the gateway: - is joined to the network, - has run for at least 180 days, and - has passed at least 90% of its epochs. The gateway's operations wallet qualifies too, but only on a gateway at schema 1.2.0 or later. The discount never applies to primary names. The checkout shows it as its own line. With credits, a card, or another token, Turbo pays the registry, and the discount does not apply. ## Next steps } > Buy credits, and see which tokens and wallets can pay. } > Update records from code with the ar.io SDK. # Working with ArNS (/build/guides/working-with-arns) The **Ar.io Name System (ArNS)** provides human-readable names for Arweave content, making it easy to access and manage your permanent websites and applications. Learn how to register, manage, and use ArNS names effectively. ## What is ArNS? ArNS is a decentralized naming system that allows you to: - **Register human-readable names** like `myapp` instead of using long transaction IDs - **Point names to content** stored on Arweave for permanent access - **Create subdomains** (undernames) for organizing different versions and components - **Establish web3 identity** with primary names linked to wallet addresses - **Transfer ownership** of names through the ANT (Ar.io Name Token) system — ANTs are Metaplex Core NFTs tradeable on Solana marketplaces ## Getting Started } title="Purchase an ArNS Name" description="Learn how to register and purchase ArNS names using the arns.ar.io interface" href="/build/guides/working-with-arns/purchase-arns-ui" /> } title="Manage ArNS Names" description="Update records, transfer ownership, and configure your ArNS names" href="/build/guides/working-with-arns/manage-arns-ui" /> } title="Use ArNS in ar.io Console" description="Register a name, edit its records, and manage it in the ar.io Console" href="/build/guides/working-with-arns/buy-manage-arns-console" /> } title="Primary Names" description="Set up web3 identity using ArNS names that resolve to wallet addresses" href="/build/guides/working-with-arns/arns-primary-names" /> ## Advanced Usage } title="Register ArNS Names Programmatically" description="Use the ar.io SDK to programmatically register and purchase ArNS names" href="/build/guides/working-with-arns/register-arns-programmatically" /> } title="Set Records Programmatically" description="Use the ar.io SDK to programmatically set and manage ArNS records" href="/build/guides/working-with-arns/set-arns-records-programmatically" /> } title="Undername Versioning" description="Use undernames to manage different versions and components of your permanent website" href="/build/guides/hosting-decentralised-apps/using-undernames-for-versioning" /> ## Key Concepts ### ArNS Names Human-readable identifiers that point to Arweave content, making it easy to access your permanent websites and applications. ### Undernames Subdomains under your main ArNS name that allow you to organize different versions, components, and content types. ### Primary Names ArNS names used as web3 identity that resolve to wallet addresses, enabling human-readable usernames across applications. ### ANT (Ar.io Name Token) A Metaplex Core NFT on Solana that represents ownership and control of an ArNS name. Each ANT stores DNS-like records pointing to Arweave data. ANTs are managed by the `ario-ant` Solana program and are tradeable on NFT marketplaces like Tensor and Magic Eden. ## Common Use Cases - **Permanent websites** - Host decentralized websites with memorable names - **Application versioning** - Maintain multiple versions of your app using undernames - **Web3 identity** - Use primary names as your identity across the ecosystem - **Content organization** - Structure different types of content using undernames - **API endpoints** - Create dedicated subdomains for different services ## Next Steps # Managing ArNS names using arns.ar.io (/build/guides/working-with-arns/manage-arns-ui) ## arns.ar.io The simplest way to register or manage an ArNS name is by using the user interface maintained by the ar.io team at [arns.ar.io](https://arns.ar.io). This site can also be accessed as an ArNS name itself on any ar.io gateway via ar://arns. To register and manage names in the ar.io Console instead, see [Buying and managing an ArNS name in ar.io Console](/build/guides/working-with-arns/buy-manage-arns-console). Visit [arns.ar.io](https://arns.ar.io) in your browser and connect your wallet using the "Connect" button at the top right of the screen. Use the "Manage Assets" button at the top right of the screen to view and manage your registered ArNS names. The "Manage Assets" page will display a list of your registered ArNS names. You can click on an ArNS name to view the details of the name, or click the "gear" icon to the right of the name to view the settings for the name. The "Settings" page will display the settings for the selected ArNS name. You can transfer the name to a different wallet, add or remove controllers, set info about the name like the description, keywords, and logo, extend the lease duration, and more from this page. The most common setting you will want to change is the data that the name is pointing to. You can change this by using the "Target ID" field. Once you have copied in a new Arweave transaction ID to the "Target ID" field, you can click the "Save" button to save and deploy the changes. Any updates to the ArNS name will require a signature from the wallet that owns the name, or an authorized controller. ## Next Steps } > Use the ar.io SDK to programmatically register ArNS names. } > Use the ar.io SDK to programmatically manage ArNS records. } > Set up web3 identity with primary names. # Purchasing an ArNS name using arns.ar.io (/build/guides/working-with-arns/purchase-arns-ui) ## arns.ar.io The simplest way to register or manage an ArNS name is by using the user interface maintained by the ar.io team at [arns.ar.io](https://arns.ar.io). This site can also be accessed as an ArNS name itself on any ar.io gateway via ar://arns. To register and manage names in the ar.io Console instead, see [Buying and managing an ArNS name in ar.io Console](/build/guides/working-with-arns/buy-manage-arns-console). Visit [arns.ar.io](https://arns.ar.io) in your browser and connect your wallet using the "Connect" button at the top right of the screen. Search for an ArNS name in the search bar in the center of the screen to see if your chosen name is available. If it is, click the "Register" button to register it. On the registration page, you will be able to configure the lease duration for registering the name, or choose to permabuy the name. The displayed price for the registration will automatically recalculate based on the selected purchase type and lease duration. Once you are satisfied with the configuration, click the "Next" button to proceed to the payment page. On the payment page, you will be able to review the details of the registration and the price you will be charged. You will also be able to select the payment method between a credit card, ARIO tokens, or Turbo Credits. If you do not have enough ARIO tokens or Turbo Credis for your chosen registration configuration, the payment button will display "Insufficient Balance" and you will not be able to proceed using that payment method. Once you are satisfied with the configuration, click the "Pay Now" button to complete the registration and celebrate your new ArNS name! Use the "Manage Assets" button at the top right of the screen to view and manage your registered ArNS names. ## Next Steps } > Learn how to update records and configure your ArNS names. } > Use the ar.io SDK to programmatically register ArNS names. } > Set up web3 identity with primary names. # Registering ArNS Names Programmatically (/build/guides/working-with-arns/register-arns-programmatically) Use the **ar.io SDK** to programmatically register and purchase ArNS names. This approach allows you to automate the registration process, check availability, calculate costs, and complete purchases programmatically. ## Prerequisites **Install the required packages:** ```bash npm install @ar.io/sdk ``` **Required:** - A Solana wallet with ARIO tokens and SOL for transaction fees ## Basic Setup **Initialize the SDK with your Solana wallet:** ```javascript const { ARIO } = require("@ar.io/sdk"); // Initialize ARIO with a Solana signer for transactions // See the SDK Configuration page for how to create a signer from @solana/kit const ario = ARIO.mainnet({ signer }); ``` ## Complete Registration Process **Check Name Availability** First, verify that the ArNS name you want to register is available: ```javascript async function checkNameAvailability(name) { try { const record = await ario.getArNSRecord({ name }); if (record) { console.log(`❌ Name '${name}' is already registered`); console.log("Record details:", record); return false; } } catch (error) { if (error.message.includes("not found") || error.message.includes("404")) { console.log(`✅ Name '${name}' is available`); return true; } throw error; } } // Usage const isAvailable = await checkNameAvailability("myapp"); ``` **Calculate Registration Cost** Get the cost for registering the name before proceeding: ```javascript async function getRegistrationCost(name, type = "lease", years = 1) { try { const cost = await ario.getTokenCost({ intent: "Buy-Name", name: name, type: type, years: years, }); // Convert to ARIO for readability const arioCost = new mARIOToken(cost).toARIO(); console.log(`Cost to register '${name}': ${arioCost} ARIO`); return { cost: arioCost, rawCost: cost, type: type, years: years, }; } catch (error) { console.error("Error calculating cost:", error); throw error; } } // Usage examples const leaseCost = await getRegistrationCost("myapp", "lease", 1); const permabuyCost = await getRegistrationCost("myapp", "permabuy"); ``` **Register the ArNS Name** Complete the registration process with full error handling: ```javascript async function registerArNSName(name, options = {}) { const { type = "lease", years = 1, processId = null, referrer = null, tags = [], } = options; try { console.log(`Starting registration process for '${name}'...`); // Step 1: Check availability console.log("1. Checking name availability..."); const isAvailable = await checkNameAvailability(name); if (!isAvailable) { throw new Error(`Name '${name}' is not available`); } // Step 2: Calculate cost console.log("2. Calculating registration cost..."); const costInfo = await getRegistrationCost(name, type, years); console.log(`Cost: ${costInfo.cost} ARIO`); // Step 3: Register the name console.log("3. Registering ArNS name..."); const record = await ario.buyRecord( { name: name, type: type, years: years, processId: processId, referrer: referrer, }, { tags: [{ name: "App-Name", value: "ArNS-Registration" }, ...tags], onSigningProgress: (step, event) => { console.log(`Signing progress: ${step}`); if (step === "spawning-ant") { console.log("Spawning ANT:", event); } if (step === "registering-ant") { console.log("Registering ANT:", event); } if (step === "verifying-state") { console.log("Verifying state:", event); } if (step === "buying-name") { console.log("Buying name:", event); } }, } ); console.log(`✅ Successfully registered '${name}'`); console.log("Record details:", record); return { success: true, record: record, cost: costInfo.cost, name: name, }; } catch (error) { console.error(`❌ Failed to register '${name}':`, error.message); return { success: false, error: error.message, name: name, }; } } // Usage const result = await registerArNSName("myapp", { type: "lease", years: 1, referrer: "my-app.com", }); ``` ## Advanced Registration Features ### Cost Details and Funding **Get detailed cost breakdown:** ```javascript async function getDetailedCost(name, type = "lease", years = 1) { try { const costDetails = await ario.getCostDetails({ intent: "Buy-Name", fromAddress: "your-wallet-address", fundFrom: "stakes", // or "balance" or "any" name: name, type: type, years: years, }); console.log("Cost details:", costDetails); return costDetails; } catch (error) { console.error("Error getting cost details:", error); throw error; } } ``` ### Batch Registration **Register multiple names:** ```javascript async function registerMultipleNames(names, options = {}) { const results = []; for (const name of names) { console.log(`\n--- Registering ${name} ---`); const result = await registerArNSName(name, options); results.push(result); // Add delay between registrations to avoid rate limiting await new Promise((resolve) => setTimeout(resolve, 2000)); } const successful = results.filter((r) => r.success); const failed = results.filter((r) => !r.success); console.log(`\n📊 Registration Summary:`); console.log(`✅ Successful: ${successful.length}`); console.log(`❌ Failed: ${failed.length}`); return results; } // Usage const names = ["myapp", "myapi", "mydocs"]; const results = await registerMultipleNames(names, { type: "lease", years: 1, }); ``` ### Registration with Validation **Enhanced registration with pre-checks:** ```javascript async function registerWithValidation(name, options = {}) { const { type = "lease", years = 1, maxCost = null, // Maximum cost in ARIO validateName = true, } = options; try { // Validate name format if (validateName) { if (!/^[a-z0-9-]+$/.test(name)) { throw new Error( "Name must contain only lowercase letters, numbers, and hyphens" ); } if (name.length 63) { throw new Error("Name must be between 3 and 63 characters"); } } // Check availability const isAvailable = await checkNameAvailability(name); if (!isAvailable) { throw new Error(`Name '${name}' is not available`); } // Get cost const costInfo = await getRegistrationCost(name, type, years); // Check cost limit if (maxCost && costInfo.cost > maxCost) { throw new Error( `Cost ${costInfo.cost} ARIO exceeds maximum ${maxCost} ARIO` ); } // Proceed with registration return await registerArNSName(name, options); } catch (error) { console.error(`Validation failed for '${name}':`, error.message); return { success: false, error: error.message, name: name, }; } } // Usage with validation const result = await registerWithValidation("myapp", { type: "lease", years: 1, maxCost: 1000, // Maximum 1000 ARIO validateName: true, }); ``` ## Registration Types ### Lease Registration **Register with a time-limited lease:** ```javascript // 1-year lease const leaseResult = await registerArNSName("myapp", { type: "lease", years: 1, }); // 5-year lease const longLeaseResult = await registerArNSName("myapp", { type: "lease", years: 5, }); ``` ### Permanent Registration **Register with permanent ownership:** ```javascript const permabuyResult = await registerArNSName("myapp", { type: "permabuy", }); ``` ## Error Handling **Comprehensive error handling:** ```javascript async function safeRegister(name, options = {}) { try { return await registerArNSName(name, options); } catch (error) { if (error.message.includes("insufficient funds")) { console.error("❌ Insufficient funds for registration"); } else if (error.message.includes("already registered")) { console.error("❌ Name is already registered"); } else if (error.message.includes("invalid name")) { console.error("❌ Invalid name format"); } else if (error.message.includes("network")) { console.error("❌ Network error, please try again"); } else { console.error(`❌ Unexpected error: ${error.message}`); } return { success: false, error: error.message, name: name, }; } } ``` ## Ready to Register? } > Learn how to set and manage ArNS records after registration. } > Set up web3 identity with your registered ArNS names. } > Use undernames to organize different versions of your content. # Setting ArNS Records Programmatically (/build/guides/working-with-arns/set-arns-records-programmatically) Use the **ar.io SDK** to programmatically set and manage ArNS records. This approach gives you full control over your ArNS operations and allows for automation and integration into your applications. ## Prerequisites **Install the required packages:** ```bash npm install @ar.io/sdk ``` **Required:** - A Solana wallet with ARIO tokens and SOL for transaction fees ## Basic Setup **Initialize the SDK:** ```javascript const { ARIO, ANT } = require("@ar.io/sdk"); // Initialize ARIO for mainnet const ario = ARIO.mainnet(); // See the Configuration page for how to create a signer from @solana/kit // const signer = await createKeyPairSignerFromBytes(keypairBytes); ``` ## Setting Base Name Records **Set the main domain record:** ```javascript async function setBaseNameRecord(arnsName, transactionId) { try { // Get the ArNS record for your domain const pid = await ario.getArNSRecord({ name: arnsName }); if (!pid) { throw new Error(`ArNS name '${arnsName}' not found`); } // Initialize ANT contract const ant = ANT.init({ processId: pid.processId, signer, }); // Set the base name record const response = await ant.setBaseNameRecord({ transactionId: transactionId, ttlSeconds: 900, // 15 minutes TTL }); console.log("Base name record updated:", response); return response; } catch (error) { console.error("Error setting base name record:", error); throw error; } } // Usage await setBaseNameRecord("myapp", "abc123...def789"); ``` ## Setting Undername Records **Set subdomain records:** ```javascript async function setUndernameRecord(arnsName, undername, transactionId) { try { // Get the ArNS record for your domain const pid = await ario.getArNSRecord({ name: arnsName }); if (!pid) { throw new Error(`ArNS name '${arnsName}' not found`); } // Initialize ANT contract const ant = ANT.init({ processId: pid.processId, signer, }); // Set the undername record const response = await ant.setUndernameRecord({ undername: undername, transactionId: transactionId, ttlSeconds: 900, // 15 minutes TTL }); console.log(`Undername '${undername}' record updated:`, response); return response; } catch (error) { console.error("Error setting undername record:", error); throw error; } } // Usage examples await setUndernameRecord("myapp", "v1", "abc123...def789"); await setUndernameRecord("myapp", "api", "def456...ghi012"); await setUndernameRecord("myapp", "docs", "ghi789...jkl345"); ``` ## Complete Deployment Example **Upload and set records in one function:** ```javascript const { TurboFactory } = require("@ardrive/turbo-sdk"); async function deployAndSetRecord(arnsName, undername, distFolderPath) { try { // Initialize Turbo for file uploads const turbo = TurboFactory.authenticated({ privateKey: jwk }); // Upload your website folder const { manifestResponse } = await turbo.uploadFolder({ folderPath: distFolderPath, dataItemOpts: { tags: [{ name: "App-Name", value: "ar.io docs deploy" }], }, manifestOptions: { fallbackFile: "index.html", }, }); console.log(`Manifest uploaded: ${manifestResponse.id}`); // Set the appropriate record if (undername === "@") { await setBaseNameRecord(arnsName, manifestResponse.id); } else { await setUndernameRecord(arnsName, undername, manifestResponse.id); } console.log(`Deployment complete!`); console.log( `Access at: https://${arnsName}.turbo-gateway.com${undername === "@" ? "" : `/${undername}`}` ); } catch (error) { console.error("Deployment failed:", error); throw error; } } // Usage await deployAndSetRecord("myapp", "@", "./build"); await deployAndSetRecord("myapp", "v1", "./build-v1"); await deployAndSetRecord("myapp", "api", "./api-docs"); ``` ## Managing Multiple Records **Set up a complete website structure:** ```javascript async function setupWebsiteStructure(arnsName) { const records = [ { undername: "@", folder: "./build", description: "Main site" }, { undername: "v1", folder: "./build-v1", description: "Version 1" }, { undername: "v2", folder: "./build-v2", description: "Version 2" }, { undername: "api", folder: "./api-docs", description: "API documentation", }, { undername: "docs", folder: "./user-docs", description: "User documentation", }, ]; for (const record of records) { try { console.log(`Deploying ${record.description}...`); await deployAndSetRecord(arnsName, record.undername, record.folder); console.log(`✅ ${record.undername} deployed successfully`); } catch (error) { console.error(`❌ Failed to deploy ${record.undername}:`, error.message); } } } // Usage await setupWebsiteStructure("myapp"); ``` ## Advanced Configuration **Custom TTL and error handling:** ```javascript async function setRecordWithOptions( arnsName, undername, transactionId, options = {} ) { const { ttlSeconds = 900, retries = 3, retryDelay = 1000 } = options; for (let attempt = 1; attempt setTimeout(resolve, retryDelay)); } } } // Usage with custom options await setRecordWithOptions("myapp", "v1", "abc123...def789", { ttlSeconds: 3600, // 1 hour retries: 5, retryDelay: 2000, }); ``` ## Reading Records **Get current record information:** ```javascript async function getRecordInfo(arnsName, undername = "@") { try { const pid = await ario.getArNSRecord({ name: arnsName }); if (!pid) { throw new Error(`ArNS name '${arnsName}' not found`); } const ant = ANT.init({ processId: pid.processId, signer }); if (undername === "@") { return await ant.getBaseNameRecord(); } else { return await ant.getUndernameRecord({ undername }); } } catch (error) { console.error("Error getting record info:", error); throw error; } } // Usage const mainRecord = await getRecordInfo("myapp", "@"); const v1Record = await getRecordInfo("myapp", "v1"); console.log("Main record:", mainRecord); console.log("V1 record:", v1Record); ``` ## Error Handling Best Practices **Comprehensive error handling:** ```javascript async function safeSetRecord(arnsName, undername, transactionId) { try { // Validate inputs if (!arnsName || !undername || !transactionId) { throw new Error("Missing required parameters"); } // Check if ArNS name exists const pid = await ario.getArNSRecord({ name: arnsName }); if (!pid) { throw new Error( `ArNS name '${arnsName}' not found or not owned by this wallet` ); } // Set the record const ant = ANT.init({ processId: pid.processId, signer }); let response; if (undername === "@") { response = await ant.setBaseNameRecord({ transactionId, ttlSeconds: 900, }); } else { response = await ant.setUndernameRecord({ undername, transactionId, ttlSeconds: 900, }); } console.log(`✅ Successfully set ${undername} record for ${arnsName}`); return response; } catch (error) { if (error.message.includes("not found")) { console.error(`❌ ArNS name '${arnsName}' not found`); } else if (error.message.includes("insufficient funds")) { console.error("❌ Insufficient funds for transaction"); } else if (error.message.includes("signature")) { console.error("❌ Invalid wallet signature"); } else { console.error(`❌ Unexpected error: ${error.message}`); } throw error; } } ``` ## Ready to Get Started? } > Learn about using undernames for website versioning. } > Set up web3 identity with primary names. } > Explore the full ar.io SDK documentation. # Get Started (/build) Welcome to ar.io's developer documentation. This section will guide you through everything you need to know for building on top of ar.io from uploading and accessing your data, to operating and extending your own infrastructure to support your unique use-case. If you're unfamiliar with Arweave's permanent storage and ar.io we recommend reading this [introduction section](/learn) first. ## Get Started Building with ar.io } title="Uploading Data" description="Learn how to permanently store files, websites, and application data on Arweave" href="/build/upload" /> } title="Accessing Data" description="Query, retrieve, and interact with data stored on the permanent web" href="/build/access" /> } title="Running Your Own Gateway" description="Deploy and operate ar.io gateway infrastructure to support the network" href="/build/run-a-gateway" /> } title="Testnet Sandbox" description="Try the full stack — upload, pay, buy ArNS names, and serve data — free on Solana devnet, no mainnet" href="/build/testnet" /> } title="AI Agents & LLMs" description="Machine-readable docs — llms.txt, SKILL.md, and per-SDK llm.txt — for building with AI coding agents" href="/build/agents" /> ## Developer Resources } title="Turbo SDK Reference" description="TypeScript SDK for Turbo, an ar.io gateway run by the ArDrive team" href="/sdks/turbo-sdk" /> } title="Wayfinder SDK Reference" description="Decentralized data access SDK with built-in verification and routing" href="/sdks/wayfinder" /> } title="ar.io SDK Reference" description="Complete SDK documentation for interacting with ar.io protocols" href="/sdks/ar-io-sdk" /> } title="Gateway API Reference" description="Direct API access for custom implementations and integrations" href="/apis" /> ## Use Cases & Guides } title="Hosting Unstoppable Apps" description="Build and deploy permanent, censorship-resistant applications with comprehensive guides" href="/build/guides/hosting-decentralised-apps" /> } title="ArNS Primary Names" description="Set up primary names for user-friendly wallet addresses" href="/build/guides/working-with-arns/arns-primary-names" /> } title="ArNS Undernames & Versioning" description="Manage subdomains and versioning for your ArNS names" href="/build/guides/hosting-decentralised-apps/using-undernames-for-versioning" /> } title="ArNS Marketplace" description="Explore the ArNS marketplace for name trading and management" href="/build/guides/arns-marketplace" /> } title="Verifiable AI" description="Anchor MLflow lifecycle proofs and verify them through ar.io gateways" href="/build/verifiable-ai" /> ## Get Help } title="Browse Examples" description="Explore code examples and sample applications" href="https://github.com/ar-io" /> } title="Join the Community" description="Connect with other developers on Discord" href="https://discord.com/invite/HGG52EtTc2" /> Ready to build on the permanent web? Choose your path above and start creating applications that last forever. # Run a Gateway (/build/run-a-gateway) Join the decentralized network that powers permanent data access. Run your own **ar.io Gateway** to support the permaweb infrastructure and earn rewards. ## Gateway Options Choose the deployment approach that fits your needs - from local testing to production infrastructure. Production Gateway Earn Rewards} description={ Join ar.io and earn ARIO tokens Earn ARIO token rewards Serve Wayfinder traffic Cache and serve Arweave data } href="/build/run-a-gateway/quick-start#production-setup-with-custom-domain" icon={} /> For development and testing Quick Docker setup Test gateway features No commitment required } href="/build/run-a-gateway/quick-start" icon={} /> Optimize for specific use cases • Data filtering options • Performance tuning • Advanced features } href="/build/run-a-gateway/manage/filters" icon={} /> ## Why Run a Gateway? **Economic Benefits** - Earn ARIO tokens through network participation - Set custom pricing for premium services - Build sustainable infrastructure business **Technical Advantages** - Full control over data access and caching - Custom configuration for your applications - Direct integration with your services **Network Impact** - Support decentralized web infrastructure - Increase network reliability and redundancy - Enable censorship-resistant data access ## Try It Locally You can run a gateway on your own machine with one `docker run` command, then test it with `curl`. The [quick start](/build/run-a-gateway/quick-start#quickstart) has both commands, and the full setup for a production gateway. ## Learn Before You Build Understanding gateways helps you make informed infrastructure decisions. } /> } /> } /> ## Ready to Deploy? Whether you're exploring gateway capabilities or ready to join the network, we have resources to help: # Join the Network (/build/run-a-gateway/join-the-network) Take control of the permanent web by running your own **ar.io Gateway**. Join the decentralized network that powers the permaweb and earn rewards for providing infrastructure services. ## Prerequisites ### Running Gateway Required You must have a fully functional ar.io Gateway running with a custom domain and SSL certificates. **Don't have a gateway yet?** Follow our [Production Setup Guide](/build/run-a-gateway/quick-start#production-setup-with-custom-domain) to get your gateway running with proper DNS configuration. **Requirements:** - Gateway accessible via your custom domain (e.g., `https://yourdomain.com`) - SSL certificates properly configured - ArNS subdomain resolution working - Gateway responding to test requests ### Minimum Stake Requirement To join the network as a gateway operator, you need **20,000 ARIO tokens** as the minimum stake requirement. **Need to acquire ARIO tokens?** Visit our [Get the Token guide](/learn/token/get-the-token) to learn about all available methods including exchanges, DEXs, and network participation. **Acquisition Options:** - Purchase on centralized exchanges like Gate.io - Trade on Solana DEXs (Jupiter, Raydium) - Use a Solana wallet (Phantom, Solflare, Backpack) to manage tokens - Earn through network participation and community programs You will also need SOL in your wallet for Solana transaction fees when joining the network. ## Join the Network Choose your preferred method to register your gateway: ### Visit the Network Portal Go to [gateways.ar.io](https://gateways.ar.io/#/gateways) to access the ar.io's Network Portal. The portal shows all active gateways on the network and provides the interface to register your own gateway. ### Connect Your Wallet Click the "Start your own gateway" button to begin the registration process. You'll be prompted to connect your wallet. Choose your preferred Solana wallet (Phantom, Solflare, or Backpack) to connect. Use the same wallet address that you configured in your gateway's `AR_IO_WALLET` environment variable. This wallet will be the owner of your gateway registration. ### Fill Out Gateway Information Complete the gateway registration form with your gateway details: **Required Fields:** - **Label**: A display name for your gateway (e.g., "My New Gateway") - **Address**: Your gateway's domain with port (e.g., `https://fastandfurious.io:443`) - **Observer Wallet**: The public address of your observer wallet - **Properties ID**: Transaction ID of your gateway properties - **Stake (ARIO)**: Minimum stake required (20,000 ARIO) - **Delegated Staking**: Enable to allow others to delegate stake to your gateway - **Minimum Delegated Stake**: Set minimum delegation amount (e.g., 100 ARIO) - **Reward Share Ratio**: Percentage of rewards shared with delegators (e.g., 50%) - **Note**: Additional information about your gateway (e.g., "ar.io rules!") ### Confirm Registration Review all information carefully and click "Confirm" to submit your gateway registration to the network. **What happens next:** - Your gateway will be added to the Gateway Address Registry - Observers will start observing your gateway at the next Epoch (day) - You will begin to receive rewards based on your gateway performance - You can monitor your gateway's performance in the portal **Confirm your gateway registration:** Your gateway should now be viewable at `gateways.ar.io/#/` with the wallet address you used to join. This dashboard shows your gateway's information, stats, and performance metrics including join date, uptime, operator stake, and delegated stake details. ### Install the ar.io CLI First, install the ar.io CLI tool if you haven't already: ```npm npm install -g @ar.io/sdk ``` ### Run the Join Network Command Use the `ar.io join-network` command with your gateway configuration. The operator key can be supplied either as a JSON keypair file (`--wallet-file`) or as a base58-encoded 64-byte secret key string (`--private-key`). The latter is the format Phantom and similar browser wallets export, so most operators can paste it directly. ```bash ar.io join-network \ --wallet-file ./path/to/solana-keypair.json \ -t solana \ --mainnet \ --rpc-url https://api.mainnet-beta.solana.com \ --core-program-id 73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh \ --gar-program-id 89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf \ --arns-program-id 2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ \ --ant-program-id 2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 \ --operator-stake 20000 \ --auto-stake \ --allow-delegated-staking \ --min-delegated-stake 100000000 \ --delegate-reward-share-ratio 10 \ --label "My Gateway" \ --note "Production ar.io gateway" \ --observer-address 7xKXtR2qpZm8FjvKNsG3kL9p5yMnYhVdEbxQ4oWc2Rn \ --fqdn my-gateway.example.com \ --port 443 \ --protocol https ``` '`. The SDK decodes the 64-byte secret, derives the public key, and signs in-memory — no JSON file required. The public key derived from your `--wallet-file` / `--private-key` becomes your `AR_IO_WALLET` (operator). The `--observer-address` must be a **unique** Solana address; no two gateways can share an observer. If you don't have a separate observer key, set `--observer-address` to your operator address. The gateway accepts this single-key setup. **Parameter explanations:** - `--operator-stake 20000` — operator stake in whole ARIO units (CLI converts to mARIO internally). Minimum is 20,000 ARIO on mainnet. - `--auto-stake` — flag-only (no value). When set, operator rewards auto-restake to your gateway. - `--allow-delegated-staking` — flag-only. Permits other wallets to delegate stake to you. - `--min-delegated-stake 100000000` — minimum delegation in mARIO (100 ARIO × 1,000,000). Unlike `--operator-stake`, this argument is in mARIO directly. - `--delegate-reward-share-ratio 10` — percentage of rewards shared with delegators (0-100). - `--observer-address` — Solana address of the observer key (often the same as the operator address). - `--fqdn` — your gateway's public domain name. - `-t solana` — required to switch the CLI into Solana mode (default is Arweave). - `--mainnet` — selects Solana mainnet behavior. The explicit `--core-program-id`, `--gar-program-id`, `--arns-program-id`, `--ant-program-id`, and `--rpc-url` flags pin the command to the canonical mainnet deployment. ### Verify Registration Verify the on-chain registration via the CLI: ```bash ar.io get-gateway \ --mainnet \ -t solana \ --rpc-url https://api.mainnet-beta.solana.com \ --core-program-id 73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh \ --gar-program-id 89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf \ --arns-program-id 2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ \ --ant-program-id 2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 \ --address ``` The result should show `"status": "joined"` and the settings (FQDN, stake, allow-delegated, etc.) you passed at join time. Once your gateway is running and pointed at the same network, cross-check from the gateway side too: ```bash curl https:///ar-io/info | jq '{wallet, programIds}' ``` - `wallet` should match the operator pubkey you joined with. - `programIds.{core,gar,arns,ant}` should match the network you joined (mainnet program IDs, or whichever staging/devnet set you're using). If those don't line up, your gateway is querying a different network than the one you joined. That's the usual reason a freshly-joined gateway doesn't show up as observing. You can also verify in the portal: `gateways.ar.io/#/`. ## What Happens After Registration After joining the network: - Your gateway will be monitored by the Observer system - You'll earn rewards for providing reliable service - You can monitor your gateway's performance and earnings in the portal - You may be selected as an Observer to help monitor other gateways ## Next Steps Your gateway is now part of ar.io! Here are some next steps to maximize your participation: } /> } /> } /> } /> # CDB64 Root Transaction Index (/build/run-a-gateway/manage/cdb64) ## Overview When your gateway receives a request for a data item (content inside an ANS-104 bundle), it needs to find the root Arweave transaction containing that data. The CDB64 index answers this lookup in constant time, so historical data items resolve without a search of the network. **Default Behavior**: As of Release 67, CDB64 is enabled by default with no configuration required. As of release r70, the gateway ships three pre-built indexes by default, covering close to 3.8 billion data items combined. ## How It Works The gateway checks multiple sources when resolving a data item ID to its root transaction. The order is controlled by `ROOT_TX_LOOKUP_ORDER`: 1. **db** - Your local SQLite database (fastest, but requires locally parsing ANS-104 bundles to index discovered items) 2. **gateways** - HEAD requests to other ar.io gateways 3. **graphql** - GraphQL queries to trusted gateways 4. **hyperbeam** - A HyperBEAM endpoint's `/raw/{id}/offset`, resolved to a root transaction against your local database 5. **cdb** - CDB64 file-based index (O(1) lookup from local files or cached remote data) The default configuration tries each source in order until a match is found: ```bash ROOT_TX_LOOKUP_ORDER=db,gateways,graphql,hyperbeam,cdb ``` With local CDB64 indexes, such as bands from [Index Sharing](/build/run-a-gateway/manage/index-sharing), move `cdb` right after `db` so the local index answers before any network source: `ROOT_TX_LOOKUP_ORDER=db,cdb,gateways,graphql`. ## Default Coverage The gateway ships three CDB64 indexes by default: - A non-AO index covering data items with content types (approximately 964 million data items) - A non-AO index covering data items without content types (approximately 1.2 billion data items) - An AO index (approximately 1.6 billion data items) The two non-AO indexes exclude Redstone data items. All three cover block heights 0 through 1,820,000. This means most historical ArDrive, Akord, AO, and similar application data can be resolved via the CDB64 index. The default shipped indexes store partition data on Arweave, so network requests are made to fetch CDB data, with byte-range caching. For zero network latency, you can download the CDB files locally. ## Fresher Indexes from Other Gateways The shipped index stops at block 1,820,000. With [Index Sharing](/build/run-a-gateway/manage/index-sharing), your gateway can subscribe to another gateway's indexes instead, such as turbo-gateway.com's, which covers block 0 to the chain tip with offsets and is updated as new data arrives. Every file is signed and checked before your gateway uses it. ## Configuration Options ### Disabling CDB64 If you want to disable CDB64 lookups (not recommended), remove `cdb` from the lookup order: ```bash ROOT_TX_LOOKUP_ORDER=db,gateways,graphql,hyperbeam ``` ### Using Custom Index Sources You can configure custom CDB64 index sources to supplement or replace the default indexes: ```bash CDB64_ROOT_TX_INDEX_SOURCES=/path/to/custom-index.cdb ``` ```bash # Directory containing multiple .cdb files or a partitioned index CDB64_ROOT_TX_INDEX_SOURCES=/path/to/index-directory/ ``` ```bash CDB64_ROOT_TX_INDEX_SOURCES=https://cdn.example.com/index.cdb ``` ```bash # 43-character base64url transaction ID CDB64_ROOT_TX_INDEX_SOURCES=ABC123def456xyz789ABC123def456xyz789ABC12 ``` ```bash # Sources are tried in order until a match is found CDB64_ROOT_TX_INDEX_SOURCES=/local/index.cdb,https://cdn.example.com/index/,TxId123... ``` ### Remote Index Configuration When using HTTP or Arweave-stored indexes, you can tune the caching and request behavior: ```bash # Caching settings CDB64_REMOTE_CACHE_MAX_REGIONS=100 # Max cached byte-range regions per source CDB64_REMOTE_CACHE_TTL_MS=300000 # Cache TTL (5 minutes) # Request settings CDB64_REMOTE_REQUEST_TIMEOUT_MS=30000 # Request timeout CDB64_REMOTE_MAX_CONCURRENT_REQUESTS=4 # Max concurrent HTTP requests # Retrieval order for fetching CDB files from Arweave (default: chunks) CDB64_REMOTE_RETRIEVAL_ORDER=chunks ``` ### File Watching For local CDB64 directories, the gateway automatically watches for new or removed `.cdb` files: ```bash # Enable/disable automatic reloading (default: true) CDB64_ROOT_TX_INDEX_WATCH=true ``` When enabled, you can add new index files to the directory without restarting your gateway. ## Partitioned Indexes Large CDB64 indexes can be split across up to 256 partition files for better manageability. Records are partitioned by the first byte of the binary data item ID, represented as a hex prefix (00-ff). A partitioned index consists of: - `manifest.json` - Describes all partitions and their locations - `00.cdb` through `ff.cdb` - Partition files (only populated prefixes exist) Partitions can be stored in different locations (local files, HTTP, Arweave), allowing flexible deployment strategies. ```bash # Point to directory containing manifest.json CDB64_ROOT_TX_INDEX_SOURCES=/path/to/partitioned-index/ ``` ```bash # HTTP URL to manifest CDB64_ROOT_TX_INDEX_SOURCES=https://cdn.example.com/index/manifest.json ``` ```bash # Append :manifest to transaction ID CDB64_ROOT_TX_INDEX_SOURCES=ABC123def456xyz789ABC123def456xyz789ABC12:manifest ``` ## Generating Custom Indexes If you need to create CDB64 indexes for specific data sets, the gateway includes CLI tools: ```bash # Generate from CSV file ./tools/generate-cdb64-root-tx-index --input data.csv --output index.cdb # Generate partitioned index (creates manifest.json automatically) ./tools/generate-cdb64-root-tx-index --input data.csv --partitioned --output-dir ./index/ # Export from local SQLite database ./tools/export-sqlite-to-cdb64 --output index.cdb # Check that every record in the CSV is in the index ./tools/verify-cdb64 --cdb64 index.cdb --csv data.csv ``` The `--partitioned` flag automatically shards records by ID prefix and generates the `manifest.json` with local file locations. For high-throughput generation, a Rust-backed tool is also available: ```bash ./tools/generate-cdb64-root-tx-index-rs --input data.csv --output index.cdb ``` ## Publishable Bands **Release Requirement**: the `ar-io-node` CLI comes with Release 85, which is not released yet. On Release 84, build bands with `export-sqlite-to-cdb64` as [Index Sharing](/build/run-a-gateway/manage/index-sharing#by-hand) shows. A band you publish through [Index Sharing](/build/run-a-gateway/manage/index-sharing) is a partitioned index with a few extra rules. Most publishers let `index-export` build bands. To build one yourself from CSV records, use the `ar-io-node` CLI, run from the gateway's directory. It deduplicates the records, names the band, checks a sample of its headers against their root transactions, and only then renames it into place: ```bash ./tools/ar-io-node index-band-build --input - --skip-header \ --publisher --kind d --height-range 2010500,tip \ --gateway-url https://turbo-gateway.com \ --gateway-url https://turbo-gateway.com ``` The [CLI reference](https://github.com/ar-io/ar-io-node/blob/develop/docs/cli.md) lists every option. ## Uploading Indexes to Arweave You can upload partitioned CDB64 indexes to Arweave for permanent, decentralized storage: ```bash ./tools/upload-cdb64-to-arweave \ --input ./partitioned-index/ \ --wallet ./wallet.json \ --concurrency 5 ``` This tool: 1. Uploads each partition file to Arweave via Turbo 2. Resolves the bundle IDs and byte offsets for each partition 3. Updates the manifest with `arweave-bundle-item` locations The resulting manifest can be shared with other gateway operators or uploaded to Arweave for decentralized index distribution. ## Performance Considerations - **O(1) lookups** - Each lookup requires only 2-3 file reads regardless of index size - **Byte-range caching** - The 4KB header is cached permanently; other regions use LRU caching - **Lazy loading** - Partitioned indexes only open accessed partitions, reducing memory usage - **Circuit breakers** - If CDB64 lookups fail repeatedly, the gateway automatically falls back to other sources ## Troubleshooting ### CDB64 lookups not working 1. Verify `cdb` is in your `ROOT_TX_LOOKUP_ORDER` 2. Check that index files exist and are readable 3. Review gateway logs for CDB64-related errors ### Slow remote index performance 1. Increase `CDB64_REMOTE_CACHE_MAX_REGIONS` for frequently accessed indexes 2. Consider downloading the index locally for best performance 3. Check network connectivity to remote sources ### Missing data items in index The default shipped indexes exclude Redstone data. For these, you'll need to: - Generate a custom index covering the desired data - Rely on other lookup sources (db, gateways, graphql, hyperbeam) For the complete list of CDB64 environment variables, see [Environment Variables Reference](/build/run-a-gateway/manage/environment-variables#cdb64-root-transaction-index). # Content Moderation (/build/run-a-gateway/manage/content-moderation) ## Overview Arweave is a network designed for permanent storage of data. It is a practical impossibility for data to be wholly removed from the network once it has been uploaded. Ar.io has adopted Arweave's voluntary content moderation model, whereby every participant of the network has the autonomy to decide which content they want to (or can legally) store, serve, and see. Each gateway operating on the network has the right and ability to blocklist any content, ArNS name, or address that is deemed in violation of its content policies or is non-compliant with local regulations. Gateway operators may set content to be blocked by their gateway by submitting a `PUT` request to their gateway defining the content to be blocked. This requires that the `ADMIN_API_KEY` environmental variable to be set in order to authenticate the moderation request. The simplest method for submitting moderation requests to a gateway is to use `curl` in a terminal. ## Quick Start ### Set Up Admin API Key Configure your admin API key in your `.env` file: ```bash # Set a secure admin API key ADMIN_API_KEY=your_secure_admin_key_here ``` Choose a strong, unique admin API key. This key provides administrative access to your gateway and should be kept secure. ### Test API Access Verify your admin API key is working: ```bash # Test admin endpoint access curl -H "Authorization: Bearer your_secure_admin_key_here" \ http://localhost:3000/ar-io/admin/debug ``` ### Block Your First Content Block a specific transaction ID: ```bash curl -X 'PUT' \ 'http://localhost:3000/ar-io/admin/block-data' \ -H 'accept: */*' \ -H 'Authorization: Bearer your_secure_admin_key_here' \ -H 'Content-Type: application/json' \ -d '{ "id": "3lyxgbgEvqNSvJrTX2J7CfRychUD5KClFhhVLyTPNCQ", "notes": "Content violates our policies", "source": "Manual Review" }' ``` ## Authentication Moderation requests must contain the gateway's `ADMIN_API_KEY` in the request Header, as `Authorization: Bearer`. For example, if a gateway's `ADMIN_API_KEY` is set to `secret`, any request must contain `Authorization: Bearer secret` in the Header. ## Block Data Specific data items can be blocked by a gateway operator by submitting a `PUT` request containing a json object with three keys: - **id**: The Arweave transaction Id of the data item to be blocked. - **notes**: Any note the gateway operator wants to leave him/herself as to the reason the content is blocked. - **source**: A note as to where the content was identified as requiring moderation. i.e. a public block list. Requests to block data must be submitted to the gateway's `/ar-io/admin/block-data` endpoint. ```bash {{ title: 'curl' }} curl -X 'PUT' \ 'http://localhost:3000/ar-io/admin/block-data' \ -H 'accept: */*' \ -H 'Authorization: Bearer secret' \ -H 'Content-Type: application/json' \ -d '{ "id": "3lyxgbgEvqNSvJrTX2J7CfRychUD5KClFhhVLyTPNCQ", "notes": "This content is offensive", "source": "Public Block list" }' ``` ### Unblock Data At this time, blocked data items can only be unblocked by manually deleting the corresponding row from the `data/sqlite/moderation.db` database. The Arweave transaction Id of the blocked data item is stored in the database as raw bytes, which sqlite3 accepts as a BLOB (Binary Large OBject), and so cannot be accessed easily using the original transaction Id, which is a base64url. Sqlite3 is able to interact with a hexadecimal representation of the BLOB, by using a BLOB literal. To do so, wrap a hexadecimal representation of the Arweave transaction Id in single quotes, and prepend an `X` i.e. `X'de5cb181b804bea352bc9ad35f627b09f472721503e4a0a51618552f24cf3424'`. Where possible, consider using the `notes` or `source` values to identify rows for deletion rather than the `id`. ```bash {{ title: 'id' }} sqlite3 data/sqlite/moderation.db "DELETE FROM blocked_ids WHERE id=X'de5cb181b804bea352bc9ad35f627b09f472721503e4a0a51618552f24cf3424';" # Note that the id in this command is a BLOB literal using the hexadecimal representation of the Arweave transaction Id, not the transaction Id in its normal base64url format ``` ```bash {{ title: 'source' }} sqlite3 data/sqlite/moderation.db "DELETE FROM blocked_ids WHERE block_source_id = (SELECT id FROM block_sources WHERE name='Public Block List');" # This command uses a subquery to look up the id in block_sources where name='Public Block List' # This command will unblock ALL data items marked with this source value ``` ## Block ArNS Name ArNS names can be blocked so that a gateway will refuse to serve their associated content even if the name holder updates the Arweave transaction Id that the name points at. This is done via an authenticated `PUT` request to the endpoint `/ar-io/admin/block-name` containing a json object with three keys: - **name**: The ArNS name to be blocked. - **notes**: Any note the gateway operator wants to leave him/herself as to the reason the content is blocked. - **source**: A note as to where the content was identified as requiring moderation. i.e. a public block list. ```bash {{ title: 'curl'}} curl -X 'PUT' \ 'http://localhost:3000/ar-io/admin/block-name' \ -H 'accept: */*' \ -H 'Authorization: Bearer secret' \ -H 'Content-Type: application/json' \ -d '{ "name": "i-bought-a-potato", "notes": "Potatoes are offensive", "source": "Public Block list" }' ``` For moderation purposes, each [undername](/learn/arns) of an ArNS name is treated as a separate name and must be moderated separately. ### Unblock ArNS Name Gateway operators can unblock ArNS names that were previously blocked. This is done via an authenticated `PUT` request to the endpoint `/ar-io/admin/unblock-name` containing a json object with a single key: - **name**: The ArNS name to be unblocked ```bash {{title: 'curl'}} curl -X 'PUT' \ 'http://localhost:3000/ar-io/admin/unblock-name' \ -H 'accept: */*' \ -H 'Authorization: Bearer secret' \ -H 'Content-Type: application/json' \ -d '{ "name": "i-bought-a-potato", }' ``` # Environment Variables (/build/run-a-gateway/manage/environment-variables) **Default Values**: Most variables have defaults that suit a typical gateway. Set a variable only to change its behavior. Set these in the `.env` file next to `docker-compose.yaml`. The **Default** column is what you get under the shipped `docker-compose.yaml`. Where Docker Compose sets a different default from the gateway's own code, the cell gives both, and the code's value applies only when you run the gateway outside Docker. A few variables have a different name in `.env` from the one the service reads, because Docker Compose renames them (for example `CORE_LOG_LEVEL` becomes the core service's `LOG_LEVEL`). The tables use the `.env` name. ## Core ar.io Node The main ar.io Gateway service that handles data retrieval, indexing, and serving. ### Server Configuration | Variable | Type | Default | Description | | -------------------------- | ------- | ----------------- | ------------------------------------------------------------------------------ | | `CORE_PORT` | number | `4000` | Host port for the core service. Inside its container, the core service listens on 4000; outside Docker, set `PORT` | | `NODE_ENV` | string | `production` | Node.js environment | | `CORE_LOG_LEVEL` | string | `info` | Logging level (error, warn, info, debug). `LOG_LEVEL` outside Docker | | `CORE_LOG_FORMAT` | string | `simple` | Log format (simple, json). `LOG_FORMAT` outside Docker | | `CORE_LOG_FILTER` | string | `{"always":true}` | Log filtering configuration. `LOG_FILTER` outside Docker | | `CORE_LOG_ALL_STACKTRACES` | boolean | `false` | Include full stack traces in logs. `LOG_ALL_STACKTRACES` outside Docker | | `INSTANCE_ID` | string | - | Unique instance identifier | ### Authentication & Security | Variable | Type | Default | Description | | -------------------- | ------ | --------- | -------------------------------------------------------------------------- | | `ADMIN_API_KEY` | String | Generated | API key for admin endpoints (generated at startup if not set). The ClickHouse auto-import service uses it too | | `ADMIN_API_KEY_FILE` | String | - | Path to file containing admin API key | ### HTTP Message Signatures (RFC 9421) Configure [response signing](/build/run-a-gateway/manage/verification-headers#http-message-signatures-rfc-9421) to provide cryptographic proof that your gateway produced a given response. If `OBSERVER_KEYPAIR_PATH` or `OBSERVER_PRIVATE_KEY` is set, the observer's Ed25519 Solana key signs responses directly. The key's Solana address is already in the on-chain Gateway Registry, so verifiers can confirm the signer with a single GAR lookup. If neither is set, the gateway auto-generates a standalone Ed25519 key at `HTTPSIG_KEY_FILE`; responses are still signed but can't be tied back to the registry. Setting both at once is rejected at startup as ambiguous. | Variable | Type | Default | Description | | ----------------------- | ------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `HTTPSIG_ENABLED` | boolean | `true` | Enable RFC 9421 response signing | | `HTTPSIG_KEY_FILE` | string | `data/keys/httpsig.pem` | Path to standalone Ed25519 private key PEM. Auto-generated on first startup if missing. Ignored when `OBSERVER_KEYPAIR_PATH` or `OBSERVER_PRIVATE_KEY` is set | | `HTTPSIG_BIND_REQUEST` | boolean | `true` | Include request method and path in signature (prevents replay) | | `OBSERVER_KEYPAIR_PATH` | string | - | Path to a 64-byte Solana keypair JSON file (e.g. `solana-keygen new` output). When set, used as the HTTPSIG signing key. The observer reads it too | | `OBSERVER_PRIVATE_KEY` | string | - | Alternative to `OBSERVER_KEYPAIR_PATH`: base58-encoded 64-byte Solana secret key (the format Phantom and other browser wallets export) | ### Network Configuration | Variable | Type | Default | Description | | --------------------------------------- | ------- | ---------------------------------------------------------------------------------------- | ------------------------------------ | | `TRUSTED_NODE_URL` | string | `http://envoy:3000`; `https://arweave.net` outside Docker | Trusted Arweave node URL. Under Docker, requests go through Envoy, which forwards them to `TRUSTED_NODE_HOST` | | `TRUSTED_GATEWAY_URL` | string | - | A single trusted gateway URL. When set and `TRUSTED_GATEWAYS_URLS` is not, it becomes the only trusted gateway | | `TRUSTED_GATEWAYS_URLS` | JSON | `{"https://turbo-gateway.com": 1, "https://arweave.net": {"priority": 2, "trusted": false}}` | Weighted trusted gateway URLs. With `TRUSTED_GATEWAY_URL` set, the default is that URL alone | | `TRUSTED_GATEWAYS_REQUEST_TIMEOUT_MS` | number | `10000` | Request timeout for trusted gateways | | `GATEWAYS_THROTTLE_BACKOFF_ENABLED` | boolean | `true` | Release 85: skip a trusted gateway that answers `429` until its `Retry-After` has passed | | `GATEWAYS_THROTTLE_BACKOFF_DEFAULT_MS` | number | `30000` | Release 85: how long to skip a gateway whose `429` has no usable `Retry-After` | | `GATEWAYS_THROTTLE_BACKOFF_MAX_MS` | number | `300000` | Release 85: the longest a `Retry-After` may make the gateway skip a trusted gateway | | `ARWEAVE_NODE_IGNORE_URLS` | string | - | Comma-separated URLs to ignore | ### Chunk Management | Variable | Type | Default | Description | | -------------------------------- | ------ | ------- | ----------------------------------- | | `CHUNK_POST_MIN_SUCCESS_COUNT` | number | `3` | Min successful chunk posts required | | `CHUNK_POST_RESPONSE_TIMEOUT_MS` | number | - | Chunk POST response timeout | | `CHUNK_POST_ABORT_TIMEOUT_MS` | number | - | Chunk POST abort timeout | ### Data Sources | Variable | Type | Default | Description | | ---------------------------- | ------ | ------------------------------------------------------------ | ---------------------------------- | | `ON_DEMAND_RETRIEVAL_ORDER` | string | `trusted-gateways,ar-io-network,chunks-offset-aware,tx-data` | On-demand data retrieval priority | | `BACKGROUND_RETRIEVAL_ORDER` | string | `chunks` | Background data retrieval priority | ### CDB64 Root Transaction Index The CDB64 index looks up which root Arweave transaction holds a data item, in one read. The gateway ships three pre-built indexes by default. See [CDB64 Root TX Index](/build/run-a-gateway/manage/cdb64) for what they cover and how to use them. | Variable | Type | Default | Description | | -------------------------------------- | ------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `ROOT_TX_LOOKUP_ORDER` | string | `db,gateways,graphql,hyperbeam,cdb` | Comma-separated root TX lookup sources: `db`, `peers`, `gateways`, `graphql`, `hyperbeam`, `cdb`, `turbo`. With local indexes, such as [Index Sharing](/build/run-a-gateway/manage/index-sharing) bands, put `cdb` right after `db` | | `CDB64_ROOT_TX_INDEX_SOURCES` | string | the three shipped `resources/` indexes | Comma-separated CDB64 sources: local files, directories (including a directory of bands), HTTP URLs, Arweave TX IDs, or bundle data items. When you add a source, list the shipped ones too if you want to keep them | | `CDB64_ROOT_TX_INDEX_DATA_PATH` | string | `./data/cdb64-root-tx-index` | Host directory mounted at `data/cdb64-root-tx-index` in the core container, for your own index files | | `CDB64_ROOT_TX_INDEX_WATCH` | boolean | `true` | Watch every local CDB64 directory source, so new files load without a restart | | `CDB64_REMOTE_RETRIEVAL_ORDER` | string | `chunks` | Data sources for fetching remote CDB64 files: `gateways`, `chunks`, `tx-data` | | `CDB64_REMOTE_CACHE_MAX_REGIONS` | number | `100` | Maximum byte-range regions to cache per remote source | | `CDB64_REMOTE_CACHE_TTL_MS` | number | `300000` | TTL for cached byte-range regions (5 minutes) | | `CDB64_REMOTE_REQUEST_TIMEOUT_MS` | number | `30000` | Request timeout for remote CDB64 sources | | `CDB64_REMOTE_MAX_CONCURRENT_REQUESTS` | number | `4` | One limit on concurrent HTTP requests, shared by all remote CDB64 sources | | `CDB64_REMOTE_SEMAPHORE_TIMEOUT_MS` | number | `5000` | Maximum wait for a request slot before failing | ### Index Sharing Settings for the `index-swarm` sidecar, which subscribes to other gateways' indexes and publishes your own. Available from Release 84. See [Index Sharing](/build/run-a-gateway/manage/index-sharing) for how each one is used. | Variable | Type | Default | Description | | --- | --- | --- | --- | | `INDEX_SWARM_SUBSCRIBE` | JSON | unset | Publishers to subscribe to, by gateway wallet: `[{"publisher":"","name":"root-tx-index"}]`. `name` may be one index or a list. Without `name`, you take every index the publisher offers except opt-in kinds; from Release 85, `parquet-l1` is opt-in and must be named. An optional `url` fetches from another address, such as a fleet's signing node; the signature is still checked against the registered key. See [Take L1 Bands](/build/run-a-gateway/manage/index-sharing#take-l1-bands-parquet-l1) | | `INDEX_SWARM_PUBLISH` | JSON | unset | Indexes this gateway publishes: `[{"name":"root-tx-index","kind":"cdb64-root-tx"}]` | | `INDEX_SWARM_MAX_DISK_BYTES` | number | unset (no ceiling; the setup script writes 50 GiB) | Ceiling on the disk the sidecar takes for bands. A band that would go over it is skipped. See [Disk](/build/run-a-gateway/manage/index-sharing#disk) | | `INDEX_SWARM_OBSERVER_KEYPAIR_FILE` | string | unset | Publishers only: host path of the observer keypair file. Set this or `OBSERVER_PRIVATE_KEY`, not both | | `INDEX_SWARM_TRUSTED_PUBLISHERS` | string | unset | Comma-separated wallets. When set, only these publishers are accepted | | `INDEX_SWARM_ALLOWED_FILE_ORIGINS` | string | unset | Other servers (`http(s)://host[:port]`) a publisher may send band files from, such as its CDN. Anything else is refused | | `INDEX_SWARM_POLL_INTERVAL_SECONDS` | number | `300` | How often each publisher is checked for new bands | | `INDEX_SWARM_DOWNLOAD_RATE_LIMIT_BYTES_PER_SEC` | number | unset | Cap on download speed, shared across all files of a band | | `INDEX_SWARM_DOWNLOAD_STALL_TIMEOUT_SECONDS` | number | `60` | Give up on a download that receives nothing for this long; it resumes next poll | | `INDEX_SWARM_DOWNLOAD_CONCURRENCY` | number | `4` | Parallel file downloads within one band | | `INDEX_SWARM_DATA_PATH` | string | `./data/indexes` | Host directory for published, downloading and installed bands. The gateway mounts the same directory | | `INDEXES_PUBLISHED_DIR` | string | `data/indexes/published` | Where the gateway's `/ar-io/indexes` routes serve from, inside the core container | | `INDEX_SWARM_PUBLISH_TTL_SECONDS` | number | `86400` | How long a publication is valid. See [Index Sharing](/build/run-a-gateway/manage/index-sharing#publish-your-indexes) | | `INDEX_SWARM_PUBLISH_SCAN_INTERVAL_SECONDS` | number | `60` | How often the publisher looks for new or changed bands | | `INDEX_SWARM_SUPERSEDE_GRACE_SECONDS` | number | `300` | How long a retired band's files stay on disk after it stops being served | | `INDEX_SWARM_MANIFEST_FETCH_TIMEOUT_MS` | number | `30000` | Give up on a publisher that has not answered in this long | | `INDEX_SWARM_REGISTRY_CACHE_TTL_SECONDS` | number | `300` | How long one read of the gateway's `/ar-io/peers`, where the sidecar gets registry records, is reused | | `INDEX_SWARM_DATA_DIR` | string | `data/indexes` | The index directory inside the sidecar's container. To move the directory on the host, set `INDEX_SWARM_DATA_PATH` instead | | `INDEX_SWARM_CORE_URL` | string | `http://core:4000` | Where the sidecar reaches the gateway for its release check | | `INDEX_SWARM_MIN_CORE_RELEASE` | number | `84` | Gateway release needed to load installed bands. Below it, the subscriber installs nothing until the gateway is upgraded | | `INDEX_SWARM_METRICS_PORT` | number | `9101` | Port for the sidecar's `/metrics` and `/healthz`, inside its container | | `INDEX_SWARM_METRICS_HOST` | string | `0.0.0.0` | Bind address for metrics, inside the container. Nothing reaches the host unless the port is mapped | | `INDEX_SWARM_SHUTDOWN_TIMEOUT_MS` | number | `10000` | How long to let work finish on shutdown before exiting anyway | | `INDEX_SWARM_LOG_LEVEL` | string | `info` | Sidecar log level | | `INDEX_SWARM_LOG_FORMAT` | string | `simple` | Sidecar log format (`simple` or `json`) | | `INDEX_SWARM_LOG_MAX_SIZE` | string | `50m` | Docker log rotation size for the sidecar, the torrent engine and its init container, and, from Release 85, `index-export` | | `INDEX_SWARM_LOG_MAX_FILE` | number | `3` | Docker log rotation file count for the same containers | #### BitTorrent Settings for the optional torrent engine (compose profile `index-swarm-torrent`). Setting `INDEX_SWARM_ENGINE_AUTH` turns it on; `./tools/index-swarm-setup --torrent` generates it. See [Tuning the Torrent Engine](/build/run-a-gateway/manage/index-sharing#tuning-the-torrent-engine). | Variable | Type | Default | Description | | --- | --- | --- | --- | | `INDEX_SWARM_ENGINE_AUTH` | string | unset | `user:password` for the engine's Web API, for example `swarm:`. The password must be at least 16 characters. Setting it turns the engine on | | `INDEX_SWARM_ENGINE_URL` | string | `http://index-swarm-engine:8080` with `INDEX_SWARM_ENGINE_AUTH`, else unset | The engine's Web API. Set it only for an engine run outside the compose profile | | `INDEX_SWARM_ENGINE_PORT` | number | `6881` | Peer port, published on the host over TCP and UDP. Keep it below Linux's ephemeral range (32768–60999) | | `INDEX_SWARM_ENGINE_PUBLIC_HOST` | string | host of the first `INDEX_SWARM_TRACKERS` URL | The host or IP peers reach this node's engine on. Set it when the tracker is behind a load balancer that does not forward the peer port | | `INDEX_SWARM_UPLOAD_LIMIT_BYTES_PER_SEC` | number | `10000000` | Cap on upload to peers (10 MB/s). `0` is unlimited. See [Bounding Upload](/build/run-a-gateway/manage/index-sharing#bounding-upload) | | `INDEX_SWARM_UPLOAD_DAILY_LIMIT_BYTES` | number | `100000000000` | Most the engine may upload in a UTC day (100 GB). `0` is no budget. See [Bounding Upload](/build/run-a-gateway/manage/index-sharing#bounding-upload) | | `INDEX_SWARM_TORRENT_TIMEOUT_SECONDS` | number | `3600` | How long a torrent may go without progress before the band is fetched over HTTP instead | | `INDEX_SWARM_WEBSEED_AFTER_SECONDS` | number | `120` | How long a torrent may stall before the publisher's WebSeed is added | | `INDEX_SWARM_TRACKERS` | string | unset | Comma-separated announce URLs written into every torrent this node publishes; normally its own tracker, `http://:6969/announce` | | `INDEX_SWARM_TRACKER_PORT` | number | `6969` | Port the tracker listens on, published on the host by the sidecar. Only a node that publishes torrents listens on it | | `INDEX_SWARM_TRACKER_TRUSTED_PROXIES` | string | unset | Comma-separated IPs or CIDRs of proxies in front of the tracker whose `X-Forwarded-For` it believes. See [Publishing from a Fleet](/build/run-a-gateway/manage/index-sharing#publishing-from-a-fleet) | | `INDEX_SWARM_ALLOWED_TRACKERS` | string | unset | Tracker URLs, exactly as written, that a subscriber hands its engine even though their host is private. Only useful with `INDEX_SWARM_ENGINE_BLOCK_PRIVATE=false` | | `INDEX_SWARM_ENGINE_BLOCK_PRIVATE` | boolean | `true` | Have the engine refuse peers, trackers and WebSeeds on private, loopback, link-local and carrier-grade NAT addresses. See [Sharing Between Your Own Gateways](/build/run-a-gateway/manage/index-sharing#sharing-between-your-own-gateways) | | `INDEX_SWARM_ENGINE_NETWORK_NAME` | string | `ar-io-index-swarm-engine` | Docker network the engine runs on, shared only with the sidecar | | `INDEX_SWARM_ENGINE_UID` / `INDEX_SWARM_ENGINE_GID` | number | `1000` | User and group the engine runs as. Set the same values for the sidecar and the engine | | `INDEX_SWARM_ENGINE_CONFIG_PATH` | string | `./data/index-swarm-engine` | Host directory for the engine's configuration and resume data | | `INDEX_SWARM_ENGINE_CONFIG_DIR` | string | `/config` | Where `index-swarm-engine-init` writes the engine's configuration, inside its container. Set it only when running the init outside compose | #### Building Bands (Release 85) **Release Requirement**: The `index-export` service and these settings are part of gateway Release 85, which is not released yet. Settings for the `index-export` service (compose profile `index-export`), which builds your gateway's bands for the sidecar to publish. It logs by `LOG_LEVEL` and `LOG_FORMAT`. See [What the Daily Run Does](/build/run-a-gateway/manage/index-sharing#what-the-daily-run-does). | Variable | Type | Default | Description | | --- | --- | --- | --- | | `INDEX_EXPORT_KINDS` | string | `root-tx-index` | Comma-separated indexes to build: `root-tx-index`, `parquet-l1`. See [Publish L1 Bands](/build/run-a-gateway/manage/index-sharing#publish-l1-bands) | | `INDEX_EXPORT_START_HEIGHT` | number | unset | The lowest height to build. Root-TX bands need it before the first run; `./tools/index-swarm-setup --publish --start-height ` sets it | | `INDEX_EXPORT_HEADER_CHECK_URL` | string | unset | Required for root-TX bands: the gateway a sample of each band's entries is checked against before publishing. `./tools/index-swarm-setup --publish` writes `https://turbo-gateway.com` | | `INDEX_EXPORT_HEADER_CHECK_TIMEOUT_MS` | number | `30000` | Timeout for each read of that check | | `INDEX_EXPORT_SOURCES` | JSON | unset (this gateway's ClickHouse when `CLICKHOUSE_URL` is set, else its SQLite) | Where root-TX records come from | | `INDEX_EXPORT_SECRETS_DIR` | string | unset | Host directory of password files for the ClickHouse sources `INDEX_EXPORT_SOURCES` names | | `INDEX_EXPORT_CORE_DB` | string | `data/sqlite/core.db` | The `core.db` L1 bands are built from, opened read-only | | `INDEX_EXPORT_RUN_AT_UTC` | string | `04:00` | Time of the daily run, `HH:MM` UTC | | `INDEX_EXPORT_RECENT_MAX_BLOCKS` | number | `100000` | Span at which a recent root-TX band is frozen and a new one starts | | `INDEX_EXPORT_L1_RUN_BUDGET_MINUTES` | number | `240` | How long a run keeps starting whole L1 bands. Raise it for a bootstrap | | `INDEX_EXPORT_METRICS_PORT` | number | `9102` | Port for `index-export`'s `/metrics` and `/healthz`, inside its container | | `INDEX_EXPORT_IMAGE_TAG` | string | the value of `CORE_IMAGE_TAG` | Core image tag for `index-export`, when it should run a newer image than the gateway | | `LOG_LEVEL` | string | `info` | `index-export` log level | | `LOG_FORMAT` | string | `simple` | `index-export` log format (`simple` or `json`) | | `INDEXES_ADVERTISE_FROM_URL` | string | unset | On a fleet node that serves your indexes but does not sign them: the signing node's base URL (for example `http://10.0.0.1:4000`), so `/ar-io/info` advertises the same publication on every node. Leave it unset on the signing node. See [Giving the Other Nodes the Index](/build/run-a-gateway/manage/index-sharing#giving-the-other-nodes-the-index) | ### Indexing & Synchronization | Variable | Type | Default | Description | | -------------------------------- | ------- | ---------- | ---------------------------------- | | `START_WRITERS` | boolean | `true` | Enable indexing processes | | `START_HEIGHT` | number | `0` | Starting block height for indexing | | `STOP_HEIGHT` | number | `Infinity` | Stopping block height for indexing | | `SKIP_CACHE` | boolean | `false` | Bypass header cache | | `SIMULATED_REQUEST_FAILURE_RATE` | number | `0` | Rate of simulated request failures | ### ANS-104 Bundle Processing | Variable | Type | Default | Description | | ------------------------- | ------- | ----------------------------------------------------------------------------------------- | ------------------------------------- | | `ANS104_UNBUNDLE_FILTER` | JSON | `{"never": true}` | Filter for bundles to unbundle | | `ANS104_INDEX_FILTER` | JSON | `{"always": true}` while background verification is on, else `{"never": true}` | Filter for data items to index | | `ANS104_UNBUNDLE_WORKERS` | number | `1`, or `0` when background verification is off and `ANS104_UNBUNDLE_FILTER` matches nothing | Number of unbundling workers | | `ANS104_DOWNLOAD_WORKERS` | number | `5`, or `0` in the same case | Number of download workers | | `FILTER_CHANGE_REPROCESS` | boolean | `false` | Reprocess old bundles with new filter | | `BACKFILL_BUNDLE_RECORDS` | boolean | `false` | Backfill bundle records | ### Data Management | Variable | Type | Default | Description | | --------------------------------------- | ------- | -------- | ---------------------------------- | | `WRITE_ANS104_DATA_ITEM_DB_SIGNATURES` | boolean | `false` | Write data item signatures to DB | | `WRITE_TRANSACTION_DB_SIGNATURES` | boolean | `false` | Write transaction signatures to DB | | `ENABLE_DATA_DB_WAL_CLEANUP` | boolean | `false` | Enable data DB WAL cleanup | | `MAX_DATA_ITEM_QUEUE_SIZE` | number | `100000` | Max data items in queue | | `BUNDLE_DATA_IMPORTER_QUEUE_SIZE` | number | `1000` | Max bundles in import queue | | `VERIFICATION_DATA_IMPORTER_QUEUE_SIZE` | number | `1000` | Max verification items in queue | | `DATA_ITEM_FLUSH_COUNT_THRESHOLD` | number | `1000` | Data items threshold for flushing | | `MAX_FLUSH_INTERVAL_SECONDS` | number | `600` | Max interval between flushes | ### File System Cleanup | Variable | Type | Default | Description | | ------------------------------------------ | ------ | ---------- | ------------------------------------ | | `FS_CLEANUP_WORKER_BATCH_SIZE` | number | `2000` | Files per cleanup batch | | `FS_CLEANUP_WORKER_BATCH_PAUSE_DURATION` | number | `5000` | Pause between cleanup batches (ms) | | `FS_CLEANUP_WORKER_RESTART_PAUSE_DURATION` | number | `14400000` | Pause before restarting cleanup (ms) | ### Background Verification | Variable | Type | Default | Description | | ------------------------------------------------ | ------- | ------- | ----------------------------------- | | `ENABLE_BACKGROUND_DATA_VERIFICATION` | boolean | `true` | Enable background data verification | | `BACKGROUND_DATA_VERIFICATION_INTERVAL_SECONDS` | number | `600` | Verification interval | | `BACKGROUND_DATA_VERIFICATION_WORKER_COUNT` | number | `1` | Number of verification workers | | `BACKGROUND_DATA_VERIFICATION_STREAM_TIMEOUT_MS` | number | `30000` | Stream timeout for verification | ### Bundle Repair | Variable | Type | Default | Description | | -------------------------------------------------- | ------ | ------- | ----------------------------- | | `BUNDLE_REPAIR_RETRY_INTERVAL_SECONDS` | number | `300` | Bundle repair retry interval | | `BUNDLE_REPAIR_UPDATE_TIMESTAMPS_INTERVAL_SECONDS` | number | `300` | Timestamp update interval | | `BUNDLE_REPAIR_BACKFILL_INTERVAL_SECONDS` | number | `900` | Backfill interval | | `BUNDLE_REPAIR_FILTER_REPROCESS_INTERVAL_SECONDS` | number | `300` | Filter reprocess interval | | `BUNDLE_REPAIR_RETRY_BATCH_SIZE` | number | `5000` | Batch size for repair retries | ### ArNS Configuration | Variable | Type | Default | Description | | ------------------------------------ | ------ | --------------------------------------- | ------------------------------------------------- | | `ARNS_ROOT_HOST` | string | - | Root hostname for ArNS. Envoy and the ClickHouse auto-import service read it too | | `SANDBOX_PROTOCOL` | string | - | Protocol for sandboxing redirects (http or https) | | `AR_IO_SDK_LOG_LEVEL` | string | `none` | ar.io SDK log level | | `ARNS_CACHE_TYPE` | string | `redis`; `node` outside Docker | ArNS cache type | | `ARNS_CACHE_TTL_SECONDS` | number | `86400` | ArNS cache TTL | | `ARNS_CACHE_MAX_KEYS` | number | `10000` | Max ArNS cache keys | | `ARNS_RESOLVER_PRIORITY_ORDER` | string | `on-demand,gateway` | ArNS resolver priority | | `ARNS_COMPOSITE_RESOLVER_TIMEOUT_MS` | number | `3000` | Composite resolver timeout | | `ARNS_NAMES_CACHE_TTL_SECONDS` | number | `3600` | Names cache TTL | | `ARNS_MAX_CONCURRENT_RESOLUTIONS` | number | `1` | Max concurrent resolutions | ### ar.io The ar.io protocol runs on five Solana programs (see [protocol architecture](/learn/protocol-architecture)), and the gateway talks to four of them: `ario-core`, `ario-gar`, `ario-arns`, and `ario-ant`. The fifth, `ario-ant-escrow`, has no gateway-side program ID below because a gateway never calls it; it is a wallet-to-wallet escrow, not something a gateway needs to resolve. Each program ID below is configured independently, so the same image can run against mainnet, staging-devnet, or a local devnet. The defaults are the mainnet IDs, listed in the [Token](/learn/token#mainnet-addresses) docs. To confirm which set a running gateway is using, `GET /ar-io/info` returns the resolved `programIds` object. | Variable | Type | Default | Description | | -------------------------- | ------- | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `AR_IO_WALLET` | string | - | Operator Solana public key (base58). Display label surfaced on `/ar-io/info` | | `SOLANA_RPC_URL` | string | `https://api.mainnet-beta.solana.com` | Solana JSON-RPC endpoint, for the gateway and the observer | | `SOLANA_KEYPAIR_PATH` | string | - | Path to the operator's 64-byte Solana keypair JSON file. Signs `join_network`, `update_gateway_settings`, and cranker instructions. Inside the container the path must start with `/app/wallets/` | | `SOLANA_PRIVATE_KEY` | string | - | Alternative to `SOLANA_KEYPAIR_PATH`: base58-encoded 64-byte secret (Phantom | `ENABLE_EPOCH_CRANKING` | boolean | `false` | When `true`, the observer runs permissionless epoch instructions (`close_observation`, `tick_epoch`, etc.) | | `ARIO_CORE_PROGRAM_ID` | string | `73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh` | `ario-core` program ID (token, staking, epoch state) | | `ARIO_GAR_PROGRAM_ID` | string | `89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf` | `ario-gar` program ID (Gateway Registry; joins, observations, distributions) | | `ARIO_ARNS_PROGRAM_ID` | string | `2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ` | `ario-arns` program ID (ArNS name registry) | | `ARIO_ANT_PROGRAM_ID` | string | `2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5` | `ario-ant` program ID (ANT records — Metaplex Core NFTs that route names to data) | | `AR_IO_NODE_RELEASE` | string | the release of the image | The release the gateway and observer report. Leave it unset | | `APEX_TX_ID` | string | - | Transaction served at the apex domain. See [Setting an Apex Domain](/build/run-a-gateway/manage/setting-apex-domain) | | `APEX_ARNS_NAME` | string | - | ArNS name served at the apex domain | | `ARNS_NOT_FOUND_TX_ID` | string | - | Not found transaction ID | | `ARNS_NOT_FOUND_ARNS_NAME` | string | `unregistered_arns` | Not found ArNS name | The default public Solana RPC is rate-limited and may block `getProgramAccounts` queries needed for full registry enumeration. For production gateways, use a dedicated RPC provider such as [Helius](https://helius.dev), [Triton](https://triton.one), or [QuickNode](https://quicknode.com). ### Caching | Variable | Type | Default | Description | | ----------------------------------------- | ------- | ------------------------------------------------------ | --------------------------------------- | | `CHAIN_CACHE_TYPE` | string | `redis`; `lmdb` outside Docker | Chain cache type (lmdb, fs, redis) | | `REDIS_CACHE_URL` | string | `redis://redis:6379`; `redis://localhost:6379` outside Docker | Redis cache URL | | `REDIS_USE_TLS` | boolean | `false` | Use TLS for Redis | | `REDIS_CACHE_TTL_SECONDS` | number | `28800` | Redis cache TTL | | `ENABLE_FS_HEADER_CACHE_CLEANUP` | boolean | `false` | Enable FS header cache cleanup | | `CONTIGUOUS_DATA_CACHE_CLEANUP_THRESHOLD` | string | - | Contiguous data cache cleanup threshold | ### Webhooks | Variable | Type | Default | Description | | ------------------------ | ------ | ----------------- | -------------------------------------- | | `WEBHOOK_TARGET_SERVERS` | string | - | Comma-separated webhook target servers | | `WEBHOOK_INDEX_FILTER` | JSON | `{"never": true}` | Webhook index filter | | `WEBHOOK_BLOCK_FILTER` | JSON | `{"never": true}` | Webhook block filter | ### Mempool Watcher | Variable | Type | Default | Description | | ----------------------------- | ------- | ------- | ------------------------ | | `ENABLE_MEMPOOL_WATCHER` | boolean | `false` | Enable mempool watcher | | `MEMPOOL_POLLING_INTERVAL_MS` | number | `30000` | Mempool polling interval | ### AWS S3 | Variable | Type | Default | Description | | ------------------------------- | ------ | ------- | ----------------------------- | | `AWS_ACCESS_KEY_ID` | string | - | AWS access key ID | | `AWS_SECRET_ACCESS_KEY` | string | - | AWS secret access key | | `AWS_SESSION_TOKEN` | string | - | AWS session token | | `AWS_REGION` | string | - | AWS region | | `AWS_ENDPOINT` | string | - | AWS endpoint | | `AWS_S3_CONTIGUOUS_DATA_BUCKET` | string | - | S3 bucket for contiguous data | | `AWS_S3_CONTIGUOUS_DATA_PREFIX` | string | - | S3 prefix for contiguous data | ### ClickHouse | Variable | Type | Default | Description | | --------------------- | ------ | ------- | --------------------------------------------------------------------------- | | `CLICKHOUSE_URL` | string | - | ClickHouse URL | | `CLICKHOUSE_USER` | string | - | ClickHouse username, for the gateway, the ClickHouse server and the auto-import service. Set it explicitly (usually `default`) when you set a password | | `CLICKHOUSE_PASSWORD` | string | - | ClickHouse password, for the same three services. Required by the auto-import service | ### PostgreSQL (Legacy) | Variable | Type | Default | Description | | ------------------------------------- | ------- | ------- | ----------------------------------- | | `LEGACY_PSQL_CONNECTION_STRING` | string | - | PostgreSQL connection string | | `LEGACY_PSQL_PASSWORD_FILE` | string | - | Path to PostgreSQL password file | | `LEGACY_PSQL_SSL_REJECT_UNAUTHORIZED` | boolean | `true` | Reject unauthorized SSL connections | ### Circuit Breaker | Variable | Type | Default | Description | | ----------------------------------------------------------------- | ------ | --------- | -------------------------- | | `ARIO_PROCESS_DEFAULT_CIRCUIT_BREAKER_TIMEOUT_MS` | number | `60000` | Circuit breaker timeout | | `ARIO_PROCESS_DEFAULT_CIRCUIT_BREAKER_ERROR_THRESHOLD_PERCENTAGE` | number | `30` | Error threshold percentage | | `ARIO_PROCESS_DEFAULT_CIRCUIT_BREAKER_ROLLING_COUNT_TIMEOUT_MS` | number | `600000` | Rolling count timeout | | `ARIO_PROCESS_DEFAULT_CIRCUIT_BREAKER_RESET_TIMEOUT_MS` | number | `1200000` | Reset timeout | ### Performance Tuning | Variable | Type | Default | Description | | ----------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | | `NODE_MAX_OLD_SPACE_SIZE` | number | `2048`, or `8192` when `ANS104_UNBUNDLE_WORKERS` is above 1 | Node.js heap limit for the core service, in MB | | `WEIGHTED_PEERS_TEMPERATURE_DELTA` | number | `2` | Weighted peers temperature delta | | `GATEWAY_PEERS_WEIGHTS_CACHE_DURATION_MS` | number | `5000` | Gateway peers weights cache duration | | `GATEWAY_PEERS_REQUEST_WINDOW_COUNT` | number | `20` | Gateway peers request window count | | `TAG_SELECTIVITY` | JSON | `{"Parent-Folder-Id": 20, "Message": 20, "Drive-Id": 10, "Process": 10, "Recipient": 10, "App-Name": -10, "Content-Type": -10, "Data-Protocol": -10}` | Tag selectivity configuration | ### Data Paths Host directories mounted into the core service. Other services that share a directory mount the same path. | Variable | Type | Default | Description | | ---------------------- | ------ | ------------------- | ---------------------------------------------------------- | | `CHUNKS_DATA_PATH` | string | `./data/chunks` | Path to chunks data | | `CONTIGUOUS_DATA_PATH` | string | `./data/contiguous` | Path to contiguous data | | `HEADERS_DATA_PATH` | string | `./data/headers` | Path to headers data | | `SQLITE_DATA_PATH` | string | `./data/sqlite` | Path to SQLite data (also backed up by Litestream) | | `DUCKDB_DATA_PATH` | string | `./data/duckdb` | Path to DuckDB data | | `TEMP_DATA_PATH` | string | `./data/tmp` | Path to temporary data (also used by the observer) | | `LMDB_DATA_PATH` | string | `./data/lmdb` | Path to LMDB data | | `PARQUET_DATA_PATH` | string | `./data/parquet` | Path to Parquet data (also used by ClickHouse auto-import) | ### Rate Limiter | Variable | Type | Default | Description | | ----------------------------------------- | ------- | ------------------------------------------------------ | ----------------------------------- | | `ENABLE_RATE_LIMITER` | boolean | `false` | Enable rate limiting system | | `RATE_LIMITER_TYPE` | string | `redis`; `memory` outside Docker | Rate limiter type (memory or redis) | | `RATE_LIMITER_REDIS_ENDPOINT` | string | `redis://redis:6379`; `localhost:6379` outside Docker | Redis endpoint for rate limiter | | `RATE_LIMITER_IP_TOKENS_PER_BUCKET` | number | `100000` | IP bucket token capacity | | `RATE_LIMITER_IP_REFILL_PER_SEC` | number | `20` | IP bucket refill rate per second | | `RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET` | number | `1000000` | Resource bucket token capacity | | `RATE_LIMITER_RESOURCE_REFILL_PER_SEC` | number | `100` | Resource bucket refill rate | | `RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST` | string | - | Comma-separated IP/CIDR allowlist, exempt from rate limits and x402. Matched against the client address only | | `TRUSTED_PROXIES` | string | loopback, private, CGNAT and link-local ranges | Proxies (IPs/CIDRs) whose `X-Forwarded-For` and `X-Real-IP` are believed when working out the client address. Add a CDN's or public load balancer's ranges when one is in front; `none` trusts no proxy, only for a core that clients reach directly, with no Envoy in front. See [x402 setup](/build/run-a-gateway/manage/x402-setup) | | `RATE_LIMITER_ARNS_ALLOWLIST` | string | - | Comma-separated ArNS allowlist | ### x402 Payment Protocol | Variable | Type | Default | Description | | -------------------------------------- | ------- | ------------------------------ | ------------------------------------------ | | `ENABLE_X_402_USDC_DATA_EGRESS` | boolean | `false` | Enable x402 USDC payments | | `X_402_USDC_NETWORK` | string | `base-sepolia` | Base network (base-sepolia or base) | | `X_402_USDC_WALLET_ADDRESS` | string | - | USDC wallet address for receiving payments | | `X_402_USDC_FACILITATOR_URL` | string | `https://x402.org/facilitator` | Payment facilitator URL | | `X_402_USDC_PER_BYTE_PRICE` | number | `0.0000000001` | Price per byte in USDC | | `X_402_USDC_DATA_EGRESS_MIN_PRICE` | number | `0.001` | Minimum payment amount | | `X_402_USDC_DATA_EGRESS_MAX_PRICE` | number | `1.00` | Maximum payment amount | | `X_402_RATE_LIMIT_CAPACITY_MULTIPLIER` | number | `10` | Paid token capacity multiplier | | `X_402_APP_NAME` | string | `AR.IO Gateway` | Application name for paywall | | `X_402_APP_LOGO` | string | - | Application logo URL for paywall | | `X_402_CDP_CLIENT_KEY` | string | - | **PUBLIC**: CDP client key for Onramp (optional testnet, required mainnet) | | `CDP_API_KEY_ID` | string | - | **SECRET**: CDP API key ID for Onramp (optional testnet, required mainnet) | | `CDP_API_KEY_SECRET` | string | - | **SECRET**: CDP API secret for Onramp (use CDP_API_KEY_SECRET_FILE instead) | | `CDP_API_KEY_SECRET_FILE` | string | - | **SECRET**: Path to CDP secret file for Onramp (takes precedence, recommended) | | `CHUNK_GET_BASE64_SIZE_BYTES` | number | `368640` | Fixed size for chunk pricing | ### Autoheal | Variable | Type | Default | Description | | -------------- | ------- | ------- | -------------------------------------------------------------------------------------- | | `RUN_AUTOHEAL` | boolean | `false` | Label the gateway's containers so the `autoheal` service restarts them when unhealthy | ## OpenTelemetry Tracing ### Basic Configuration | Variable | Type | Default | Description | | --------------------------------- | ------ | ------------ | ---------------------------------- | | `OTEL_SERVICE_NAME` | string | `ar-io-node` | OpenTelemetry service name | | `OTEL_EXPORTER_OTLP_ENDPOINT` | string | - | OTLP exporter endpoint | | `OTEL_EXPORTER_OTLP_HEADERS` | string | - | OTLP exporter headers | | `OTEL_EXPORTER_OTLP_HEADERS_FILE` | string | - | Path to OTLP exporter headers file | ### Tracing Performance | Variable | Type | Default | Description | | ------------------------------------------------ | ------ | ------- | --------------------------------------------------------- | | `OTEL_BATCH_LOG_PROCESSOR_SCHEDULED_DELAY_MS` | number | `2000` | Batch log processor scheduled delay | | `OTEL_BATCH_LOG_PROCESSOR_MAX_EXPORT_BATCH_SIZE` | number | `10000` | Max | `OTEL_TRACING_SAMPLING_RATE_DENOMINATOR` | number | `1` | Tracing sampling rate denominator (`1` keeps every trace) | ## Observer Service ### Basic Configuration | Variable | Type | Default | Description | | -------------------- | ------ | --------- | --------------------------------------------------------- | | `OBSERVER_PORT` | number | `5050` | Host port for the observer, and the port Envoy forwards to | | `OBSERVER_LOG_LEVEL` | string | `verbose` | Observer log level | | `OBSERVER_WALLET` | string | - | Observer wallet | ### Observer Operation | Variable | Type | Default | Description | | ------------------------------------- | ------- | ------------------------------ | ------------------------------------------ | | `RUN_OBSERVER` | boolean | `true` | Run observer service | | `SUBMIT_CONTRACT_INTERACTIONS` | boolean | `true`; `false` outside Docker | Submit observations to Solana programs. Pre-flight no-ops unless your pubkey is in `epoch.prescribed_observers`, so it is harmless before `join_network` | | `NUM_ARNS_NAMES_TO_OBSERVE_PER_GROUP` | number | `8` | Number of ArNS names per observation group | | `REPORT_GENERATION_INTERVAL_MS` | number | `3600000` | Report generation interval (one hour) | | `REPORT_DATA_SINK` | string | `turbo` | Where reports are saved | | `TURBO_UPLOAD_SERVICE_URL` | string | - | Turbo upload service URL | | `MIN_RELEASE_NUMBER` | number | `0` | Minimum release number | ### Upload Wallet Identities The observer uploads report bundles to Turbo. The upload signer is resolved from the first matching env in the [precedence chain](/build/run-a-gateway/manage/solana-migration#upload-signing-precedence). Setting envs from more than one chain group at once is rejected at startup. If your observer logs warn that `TurboReportSink` is not configured, explicitly set a Solana upload signer. Most operators can use the same base58 secret for both `OBSERVER_PRIVATE_KEY` and `SOLANA_UPLOAD_PRIVATE_KEY`. | Variable | Type | Default | Description | | --------------------------------- | ------ | ------- | --------------------------------------------------------------------------- | | `ARWEAVE_UPLOAD_KEY_FILE` | string | - | Path to an Arweave JWK file. Highest priority for upload signing | | `ARWEAVE_UPLOAD_JWK` | string | - | Inline Arweave JWK JSON. Lower priority than the file form | | `ETHEREUM_UPLOAD_PRIVATE_KEY_FILE`| string | - | Path to a 32-byte hex private key (with or without `0x` prefix) | | `ETHEREUM_UPLOAD_PRIVATE_KEY` | string | - | Inline hex private key. Lower priority than the file form | | `SOLANA_UPLOAD_KEYPAIR_PATH` | string | - | Path to a separate Solana keypair JSON for uploads. Ignored when any `ARWEAVE_UPLOAD_*` or `ETHEREUM_UPLOAD_*` is set | | `SOLANA_UPLOAD_PRIVATE_KEY` | string | - | Alternative to above: base58 secret. Mutually exclusive with the file form | When none of the above are set, uploads fall back to the observer key, then the operator key. For production observers, prefer setting `SOLANA_UPLOAD_KEYPAIR_PATH` or `SOLANA_UPLOAD_PRIVATE_KEY` explicitly so report uploads do not depend on fallback behavior. ### Observer Data Paths | Variable | Type | Default | Description | | ------------------- | ------ | ---------------- | -------------------- | | `REPORTS_DATA_PATH` | string | `./data/reports` | Path to reports data | | `WALLETS_PATH` | string | `./wallets` | Path to wallets | ## Envoy Proxy Docker Compose also passes Envoy `CORE_PORT`, `OBSERVER_PORT` and `ARNS_ROOT_HOST`, described above. | Variable | Type | Default | Description | | ------------------- | ------ | ------------- | -------------------------------------------- | | `ENVOY_PORT` | number | `3000` | Host port Envoy listens on | | `ENVOY_LOG_LEVEL` | string | `info` | Envoy log level | | `TRUSTED_NODE_HOST` | string | `arweave.net` | Arweave node Envoy forwards node requests to | | `TRUSTED_NODE_PORT` | number | `443` | Port of that node | | `GRAPHQL_HOST` | string | `core` | Where Envoy sends GraphQL requests | | `GRAPHQL_PORT` | number | `4000` | Port for GraphQL requests | ## Redis Cache | Variable | Type | Default | Description | | ------------------- | ------ | --------------------------- | ------------------ | | `REDIS_MAX_MEMORY` | string | `256mb` | Redis max memory | | `EXTRA_REDIS_FLAGS` | string | `--save "" --appendonly no` | Extra Redis flags | | `REDIS_DATA_PATH` | string | `./data/redis` | Path to Redis data | ## ClickHouse Server `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD` are under [ClickHouse](#clickhouse) in the core section. | Variable | Type | Default | Description | | ---------------------- | ------ | ------------------- | ------------------------------------------ | | `CLICKHOUSE_PORT` | number | `9000` | Host port for ClickHouse's native protocol | | `CLICKHOUSE_DATA_PATH` | string | `./data/clickhouse` | Path to ClickHouse data | | `CLICKHOUSE_LOGS_PATH` | string | `./logs/clickhouse` | Path to ClickHouse logs | ### ClickHouse Auto-Import | Variable | Type | Default | Description | | ------------------------------------------ | ------ | ------------ | --------------------------------- | | `CLICKHOUSE_DEBUG` | string | - | ClickHouse debug flag | | `CLICKHOUSE_HOST` | string | `clickhouse` | ClickHouse host | | `CLICKHOUSE_AUTO_IMPORT_SLEEP_INTERVAL` | string | - | Auto-import sleep interval | | `CLICKHOUSE_AUTO_IMPORT_HEIGHT_INTERVAL` | string | - | Auto-import height interval | | `CLICKHOUSE_AUTO_IMPORT_MAX_ROWS_PER_FILE` | string | - | Max rows per file for auto-import | ## Litestream Backup | Variable | Type | Default | Description | | ------------------------------------------ | ------ | ------- | ----------------------------------- | | `AR_IO_SQLITE_BACKUP_S3_BUCKET_NAME` | string | - | S3 bucket name for SQLite backups | | `AR_IO_SQLITE_BACKUP_S3_BUCKET_REGION` | string | - | S3 bucket region for SQLite backups | | `AR_IO_SQLITE_BACKUP_S3_BUCKET_ACCESS_KEY` | string | - | S3 access key for SQLite backups | | `AR_IO_SQLITE_BACKUP_S3_BUCKET_SECRET_KEY` | string | - | S3 secret key for SQLite backups | | `AR_IO_SQLITE_BACKUP_S3_BUCKET_PREFIX` | string | - | S3 prefix for SQLite backups | ## Image Tags Each service's image tag can be overridden. The defaults change with each release, so they are not listed here: they are in [`docker-compose.yaml` on `main`](https://github.com/ar-io/ar-io-node/blob/main/docker-compose.yaml). | Variable | Type | Default | Description | | ---------------------------------- | ------ | ------------------------- | ------------------------------------------------------ | | `CORE_IMAGE_TAG` | string | see `docker-compose.yaml` | Core image tag, also used by the `index-swarm` sidecar | | `ENVOY_IMAGE_TAG` | string | see `docker-compose.yaml` | Envoy image tag | | `OBSERVER_IMAGE_TAG` | string | see `docker-compose.yaml` | Observer image tag | | `REDIS_IMAGE_TAG` | string | see `docker-compose.yaml` | Redis image tag | | `CLICKHOUSE_IMAGE_TAG` | string | see `docker-compose.yaml` | ClickHouse image tag | | `CLICKHOUSE_AUTO_IMPORT_IMAGE_TAG` | string | see `docker-compose.yaml` | ClickHouse auto-import image tag | | `LITESTREAM_IMAGE_TAG` | string | see `docker-compose.yaml` | Litestream image tag | `INDEX_EXPORT_IMAGE_TAG` (Release 85) is under [Building Bands](#building-bands-release-85). ## Usage Notes - All environment variables are optional unless otherwise specified - Default values are shown in the "Default" column - Boolean values should be set to `true` or `false` - JSON values should be valid JSON strings - Path values should be absolute or relative to the project root - Some variables are only used in specific deployment scenarios (e.g., ClickHouse, Litestream) - Data paths can be customized based on your storage requirements ## Configuration Examples ### Basic Gateway Setup ```bash # Core configuration CORE_LOG_LEVEL=info ADMIN_API_KEY=your-admin-key-here # Network configuration TRUSTED_GATEWAY_URL=https://turbo-gateway.com # Data paths CHUNKS_DATA_PATH=/data/chunks CONTIGUOUS_DATA_PATH=/data/contiguous SQLITE_DATA_PATH=/data/sqlite ``` ### Advanced Gateway with ClickHouse ```bash # Core configuration CORE_LOG_LEVEL=info ADMIN_API_KEY=your-admin-key-here # ClickHouse configuration CLICKHOUSE_URL=http://clickhouse:8123 CLICKHOUSE_USER=default CLICKHOUSE_PASSWORD=your-password # Bundle processing ANS104_UNBUNDLE_FILTER={"and": [{"equals": {"App-Name": "MyApp-v1.0"}}]} ANS104_INDEX_FILTER={"and": [{"equals": {"App-Name": "MyApp-v1.0"}}]} ANS104_UNBUNDLE_WORKERS=2 ANS104_DOWNLOAD_WORKERS=5 ``` ### Gateway with Redis Caching Under Docker Compose, the chain cache and the ArNS cache already use the bundled Redis. Outside Docker: ```bash # Redis configuration CHAIN_CACHE_TYPE=redis REDIS_CACHE_URL=redis://localhost:6379 REDIS_USE_TLS=false REDIS_CACHE_TTL_SECONDS=28800 # ArNS configuration ARNS_ROOT_HOST=your-domain.com ARNS_CACHE_TYPE=redis ``` # Gateway Filters (/build/run-a-gateway/manage/filters) Configure your ar.io Gateway to efficiently process and index only the data you need. This guide covers filtering techniques, performance, and worked use cases. ## Overview The ar.io Gateway uses a flexible JSON-based filtering system to control data processing and indexing. The system provides precise control over which bundles are processed and which data items are indexed for querying. ## Understanding the Filtering System The ar.io Gateway uses two primary filters to control data processing: 1. **ANS104_UNBUNDLE_FILTER** - Controls which bundles are processed and unbundled 2. **ANS104_INDEX_FILTER** - Controls which data items from unbundled bundles are indexed for querying By default, gateways process no bundles and index no data items. You must explicitly configure filters to start processing data. ## Core Environment Variables ### Configure Data Management Optimize data storage and processing: ```bash # Number of new data items before flushing to stable storage DATA_ITEM_FLUSH_COUNT_THRESHOLD=1000 # Maximum time between flushes (in seconds) MAX_FLUSH_INTERVAL_SECONDS=600 # Maximum number of data items to queue for indexing MAX_DATA_ITEM_QUEUE_SIZE=100000 # Enable background verification ENABLE_BACKGROUND_DATA_VERIFICATION=true ``` ### Set Up GraphQL Configuration Choose between local-only or proxied queries: ```bash # For new gateways - proxy to turbo-gateway.com for complete index GRAPHQL_HOST=turbo-gateway.com GRAPHQL_PORT=443 # For local-only queries (uncomment to use) # GRAPHQL_HOST= ``` ## Filter Construction While the filters below are displayed on multiple lines for readability, they must be stored in the `.env` file as a single line for proper processing. ### Basic Filters The simplest filters you can use are `"always"` and `"never"` filters. The `"never"` filter is the default behavior and will match nothing, while the `"always"` filter matches everything. ```json {{title: "Never Match"}} { "never": true //default behavior } ``` ```json {{title: "Always Match"}} { "always": true } ``` ### Tag Filters Tag filters allow you to match items based on their tags in three different ways. You can match exact tag values, check for the presence of a tag regardless of its value, or match tags whose values start with specific text. All tag values are automatically base64url-decoded before matching. ```json {{title: "Exact Match"}} { "tags": [ { "name": "Content-Type", "value": "image/jpeg" } ] } ``` ```json {{title: "Match Tag Name Only"}} { "tags": [ { "name": "App-Name" } ] } ``` ```json {{title: "Starts With Match"}} { "tags": [ { "name": "Protocol", "valueStartsWith": "AO" } ] } ``` ### Attribute Filters Attribute filtering allows you to match items based on their metadata properties. The system automatically handles owner public key to address conversion, making it easy to filter by owner address. You can combine multiple attributes in a single filter: ```json {{title: "Basic Attributes"}} { "attributes": { "owner_address": "xyz123...", "data_size": 1000 } } ``` ### Nested Bundle Filter The `isNestedBundle` filter is a specialized filter that checks whether a data item is part of a nested bundle structure. It's particularly useful when you need to identify or process data items in bundles that are contained within other bundles. ```json {{title: "Basic Nested Bundle"}} { "isNestedBundle": true } ``` **Note**: When processing nested bundles, be sure to include filters that match the nested bundles in both `ANS104_UNBUNDLE_FILTER` and `ANS104_INDEX_FILTER`. The bundle data items (nested bundles) need to be indexed to be matched by the unbundle filter. ### Complex Filters Using Logical Operators For more complex scenarios, the system provides logical operators (AND, OR, NOT) that can be combined to create sophisticated filtering patterns. These operators can be nested to any depth: ```json {{title: "AND Operation"}} { "and": [ { "tags": [ { "name": "App-Name", "value": "ArDrive-App" } ] }, { "tags": [ { "name": "Content-Type", "valueStartsWith": "image/" } ] } ] } ``` ```json {{title: "OR Operation"}} { "or": [ { "tags": [ { "name": "App-Name", "value": "ArDrive-App" } ] }, { "attributes": { "data_size": 1000 } } ] } ``` ```json {{title: "NOT Operation"}} { "not": { "tags": [ { "name": "Content-Type", "value": "application/json" } ] } } ``` ## Filter Configuration Strategies ### Process Everything ```json { "always": true } ``` ### Process Nothing (Default) ```json { "never": true } ``` ### Process Specific App Data ```json { "tags": [ { "name": "App-Name", "valueStartsWith": "MyApp" } ] } ``` ### Single Application ```json { "tags": [ { "name": "App-Name", "value": "MyApp-v1.0" } ] } ``` ### Multiple Applications ```json { "or": [ { "tags": [ { "name": "App-Name", "value": "MyApp-v1.0" } ] }, { "tags": [ { "name": "App-Name", "value": "AnotherApp-v2.1" } ] } ] } ``` ### Application with Version Range ```json { "tags": [ { "name": "App-Name", "valueStartsWith": "MyApp" } ] } ``` ### Content Type Filtering ```json { "tags": [ { "name": "Content-Type", "valueStartsWith": "image/" } ] } ``` ### Specific File Types ```json { "or": [ { "tags": [ { "name": "Content-Type", "value": "application/json" } ] }, { "tags": [ { "name": "Content-Type", "value": "text/plain" } ] } ] } ``` ### File Size Filtering ```json { "attributes": { "data_size": 1000000 } } ``` ### Single Owner ```json { "attributes": { "owner_address": "YOUR_WALLET_ADDRESS" } } ``` ### Multiple Owners ```json { "or": [ { "attributes": { "owner_address": "WALLET_ADDRESS_1" } }, { "attributes": { "owner_address": "WALLET_ADDRESS_2" } } ] } ``` ### Exclude Specific Owners ```json { "not": { "attributes": { "owner_address": "UNWANTED_WALLET_ADDRESS" } } } ``` ### Complex Multi-Condition Filter ```json { "and": [ { "tags": [ { "name": "App-Name", "valueStartsWith": "MyApp" } ] }, { "attributes": { "owner_address": "YOUR_WALLET_ADDRESS" } }, { "not": { "tags": [ { "name": "Content-Type", "value": "application/octet-stream" } ] } } ] } ``` ### Exclude Common Bundlers ```json { "and": [ { "not": { "or": [ { "tags": [ { "name": "Bundler-App-Name", "value": "Warp" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "Redstone" } ] }, { "attributes": { "owner_address": "-OXcT1sVRSA5eGwt2k6Yuz8-3e3g9WJi5uSE99CWqsBs" } } ] } }, { "tags": [ { "name": "App-Name", "valueStartsWith": "MyApp" } ] } ] } ``` ## Real-World Use Cases ### Personal Data Gateway For individuals who want to process only their own data: **Unbundle Filter:** ```json { "and": [ { "not": { "or": [ { "tags": [ { "name": "Bundler-App-Name", "value": "Warp" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "Redstone" } ] } ] } }, { "tags": [ { "name": "App-Name", "valueStartsWith": "MyApp" } ] } ] } ``` **Index Filter:** ```json { "attributes": { "owner_address": "YOUR_WALLET_ADDRESS" } } ``` ### Application-Specific Service Ideal for building services around specific applications: **Unbundle Filter:** ```json { "tags": [ { "name": "App-Name", "valueStartsWith": "MyApp" } ] } ``` **Index Filter:** ```json { "or": [ { "tags": [ { "name": "ArFS", "value": "0.10" } ] }, { "tags": [ { "name": "ArFS", "value": "0.11" } ] }, { "tags": [ { "name": "ArFS", "value": "0.12" } ] } ] } ``` ### Content-Type Focused Gateway For gateways specializing in specific content types: **Unbundle Filter:** ```json { "tags": [ { "name": "Content-Type", "valueStartsWith": "image/" } ] } ``` **Index Filter:** ```json { "and": [ { "tags": [ { "name": "Content-Type", "valueStartsWith": "image/" } ] }, { "attributes": { "data_size": 100000 } } ] } ``` ## Performance Optimization ### Worker Configuration ### Understanding Default Worker Settings The gateway uses sensible defaults that work well for most users: ```bash # Default values (no need to set unless customizing) # ANS104_UNBUNDLE_WORKERS=1 (default: 0, or 1 if filters are set) # ANS104_DOWNLOAD_WORKERS=5 (default: 5) # Only adjust if you have specific hardware requirements # or want to optimize for your system's capabilities ``` **When to Adjust Workers:** Only modify worker counts if you have high-performance hardware and want to maximize throughput, or if you're experiencing resource constraints and need to reduce load. ### Optimize Data Flushing Balance between memory usage and database performance: ```bash # For high-memory systems, increase threshold DATA_ITEM_FLUSH_COUNT_THRESHOLD=2000 # For low-memory systems, decrease threshold DATA_ITEM_FLUSH_COUNT_THRESHOLD=500 # Adjust flush interval based on data volume MAX_FLUSH_INTERVAL_SECONDS=300 ``` ### Enable Background Processing ```bash # Enable background verification ENABLE_BACKGROUND_DATA_VERIFICATION=true # Enable WAL cleanup for better performance ENABLE_DATA_DB_WAL_CLEANUP=true ``` ## Webhook Filters There are also two filters available that are used to trigger webhooks. When a transaction is processed that matches one of the webhook filters, the gateway will send a webhook to the specified `WEBHOOK_TARGET_SERVERS` urls containing the transaction data. ```bash WEBHOOK_INDEX_FILTER="" WEBHOOK_BLOCK_FILTER="" ``` The `WEBHOOK_INDEX_FILTER` is used to trigger a webhook when a transaction is indexed. The `WEBHOOK_BLOCK_FILTER` is used to trigger a webhook when a block is processed. ## Important Notes - All tag names and values are base64url-decoded before matching - Owner addresses are automatically converted from owner public keys - Empty or undefined filters default to "never match" - Tag matching requires all specified tags to match - Attribute matching requires all specified attributes to match - The filter system supports nested logical operations to any depth, allowing for very precise control over what data gets processed ## Best Practices ### Filter Design 1. **Start Simple** - Begin with basic filters and gradually add complexity 2. **Test Thoroughly** - Use `FILTER_CHANGE_REPROCESS=true` when changing filters 3. **Monitor Performance** - Watch system resources during processing 4. **Document Changes** - Keep track of filter modifications and their effects ### Maintenance 1. **Regular Monitoring** - Check gateway logs for errors and warnings 2. **Resource Cleanup** - Periodically clean up old data and logs 3. **Filter Optimization** - Refine filters based on actual data patterns 4. **Backup Configuration** - Keep copies of working filter configurations ### Troubleshooting If your gateway stops processing data after changing filters, check: - Filter syntax is valid JSON - Required environment variables are set - Gateway has been restarted after changes - System has sufficient resources ## Next Steps Now that you understand gateway filtering, continue building your infrastructure: } title="Set Up Monitoring" description="Deploy Grafana to visualize your gateway's performance metrics" href="/build/extensions/grafana" /> } title="Add ClickHouse" description="Improve query performance with ClickHouse and Parquet integration" href="/build/extensions/clickhouse" /> } title="Deploy Bundler" description="Accept data uploads directly through your gateway" href="/build/extensions/bundler" /> # Index Sharing (/build/run-a-gateway/manage/index-sharing) To serve a data item, your gateway first has to find which Arweave transaction holds it, and where inside that transaction it sits. [CDB64 indexes](/build/run-a-gateway/manage/cdb64) answer that locally. **Index Sharing** keeps those indexes fresh by letting gateways share them: one gateway publishes its indexes, others subscribe, and every file is signed and checked before it is used. Bands move over HTTP, and, with the optional torrent engine, peer to peer over BitTorrent. For how it works and why it can be trusted, see [Index Sharing](/learn/gateways/index-sharing) in Learn. **Release Requirement**: Index Sharing is available from gateway Release 84. Both your gateway and the publisher you subscribe to must be on Release 84 or later. Index Sharing runs in an optional sidecar container, `index-swarm`, beside your gateway; its scripts, compose profiles and settings all use that name. Two scripts in the gateway repository do the work, run from the gateway's directory (where `.env` and `docker-compose.yaml` are). They need only Docker, since they run in the core image: - `./tools/index-swarm-setup` edits `.env` and, with `--restart`, restarts what the changes need. - `./tools/index-swarm-status` checks that everything works and says how to fix what doesn't. ## HTTP or BitTorrent? Decide this first; you can change it later by running the setup script again. | | HTTP only | With BitTorrent (`--torrent`) | | --- | --- | --- | | First download | From the publisher's HTTP routes, through its rate limits and x402. Can take hours | From peers first, unmetered; the publisher's HTTP routes are the fallback | | Ports to open | None | None required. Opening 6881, TCP and UDP, is recommended: it lets peers connect to you | | You upload | Nothing | You seed what you install to other gateways, capped at 10 MB/s and 100 GB a day by default | | Extra memory | About 60 MB (the sidecar) | About 250 MB (the sidecar and the torrent engine), plus the files it seeds mapped into memory: the kernel can reclaim those pages, but they count against a container memory limit | BitTorrent is the better default: your first pull is faster and free, and each gateway that seeds makes the next one's faster. It works behind NAT or a closed port too, because your gateway connects out to the publisher and to other peers; an open port adds the peers that can only be reached, not reach you. Choose HTTP only if you don't want to upload. [Network and Disk](#network-and-disk) has the details. What runs on your gateway once it is set up. Dashed boxes are optional: the torrent engine runs only with `--torrent`, and `index-export` (Release 85) only on a publisher that builds its own bands. The tracker answers only when you publish torrents: ```mermaid graph LR NET[Clients andother gateways] PEERS[BitTorrent peers] subgraph Host ["Your gateway"] ENVOY[Envoy :3000] CORE[Core Service] SIDE[index-swarm sidecarpublisher · subscribertracker :6969] ENG[Torrent engine :6881own Docker network] EXP[index-export] DIR[data/indexes] end NET -->|/ar-io/indexes| ENVOY ENVOY --> CORE CORE -->|reads root-TX bands| DIR EXP -->|bands into published/| DIR SIDE -->|publishes, installs| DIR SIDE -->|registry| CORE SIDE -->|bands over HTTP| NET SIDE --> ENG ENG -->|writes swarm/ only| DIR ENG |pieces| PEERS PEERS -->|announce| SIDE classDef service fill:#10b981,stroke:#059669,stroke-width:2px,color:#fff classDef proxy fill:#f59e0b,stroke:#d97706,stroke-width:2px,color:#fff classDef external fill:#8b5cf6,stroke:#7c3aed,stroke-width:2px,color:#fff classDef storage fill:#ef4444,stroke:#dc2626,stroke-width:2px,color:#fff classDef optional fill:#10b981,stroke:#059669,stroke-width:2px,color:#fff,stroke-dasharray: 5 5 class CORE,SIDE service class ENG,EXP optional class ENVOY proxy class NET,PEERS external class DIR storage ``` ## Subscribe to a Publisher Subscribing is what most operators want: your gateway downloads a publisher's index bands, verifies them, and answers lookups from them. ### Prerequisites - A running ar.io gateway on Release 84 or later, with Docker, and a checkout of the gateway repository at that release (the scripts live in `tools/`) - 50 GiB of free disk on the filesystem that holds `data/indexes`, the budget the setup script sets. The bands turbo-gateway.com publishes take about 21 GB once installed; the rest is room for a band to be on disk twice while it arrives and for a new band beside the one it replaces. This is on top of what your gateway already uses, and it doesn't change the gateway's minimum requirements. See [Network and Disk](#network-and-disk) - For BitTorrent: an open peer port is recommended, not required. See [Ports](#ports) ### Choose a Publisher A publisher is identified by its **gateway wallet**, not its URL. Your gateway looks the wallet up in the gateway registry to learn where to fetch from and which key must have signed. The turbo-gateway.com gateway publishes a root transaction index spanning block 0 to the chain tip, with offsets. It covers the bundles turbo-gateway.com has indexed, so it is densest for recent data: ```text 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 ``` To see whether another gateway publishes, check its `/ar-io/info` for an `indexes` entry (`curl -s https:///ar-io/info | jq .indexes`), and take its wallet from the registry. ### Run the Setup Script See what it would change first, then run it for real: ```bash ./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --torrent --dry-run ./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --torrent --restart ``` This subscribes to the publisher's root transaction index, points your gateway at the installed bands, puts `cdb` right after `db` in the lookup order, generates the torrent engine's password, and restarts only what needs it, by service name. Leave out `--torrent` to move bands over HTTP only. ### Open the Peer Port With `--torrent`, open the engine's peer port to the internet if you can, as [Ports](#ports) describes. It isn't required: without it, your gateway still downloads from peers and seeds to the peers it connects to. Behind a home router, forward the port yourself; the engine doesn't use UPnP. ### Check It ```bash ./tools/index-swarm-status ``` Each line says `ok`, `WARN` or `FAIL`, and every problem comes with the fix. When it says `All good`, your gateway is answering root transaction lookups from the installed bands. Bands arrive newest heights first, because most lookups are for recent data. A first pull of a full index takes minutes to hours. After that, new bands from the publisher install, and replace the ones they supersede, by themselves; the gateway loads each within 30 seconds, without a restart. ### What to Expect The status script's `CDB64 lookups ... found X of Y` line shows how many root transaction lookups the installed bands answered. Don't expect every lookup to hit. A publisher's index covers the bundles it indexed, densest for recent data, and the tip band is rebuilt periodically, so the newest items aren't in it yet. A miss costs nothing: your gateway falls back to its usual lookup order. Following more than one publisher widens coverage. **The first download can take hours over HTTP.** Publishers meter their index files with the same rate limits and [x402 payments](/build/run-a-gateway/manage/x402-setup) as any data. With BitTorrent, bands come from peers first, and peers are not metered; the publisher's HTTP routes are the fallback. Downloads resume where they stopped, so nothing is lost if your gateway restarts part way through. ### Take L1 Bands (`parquet-l1`) **Release Requirement**: L1 bands need Release 85 on your gateway and on the publisher. Release 85 is not released yet. A publisher can also offer **L1 bands**: the Arweave base layer (blocks, transactions, tags and wallets) in Parquet, one height range per band. Your gateway doesn't read them. They are for two jobs: - Bootstrapping a new gateway's L1 index with `index-l1-import`, instead of indexing the chain block by block. See [Bootstrap from L1 Bands](/build/run-a-gateway/manage/index-snapshots#bootstrap-from-l1-bands). - Analytics: query the installed files with DuckDB or Polars, or read them over HTTP. See [Reading Shared Indexes](/build/advanced/index-publications). A subscription takes L1 bands only when it names them. The setup script subscribes to root-TX bands only, so add the name by hand: ```bash INDEX_SWARM_SUBSCRIBE='[{"publisher":"34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5","name":["root-tx-index","parquet-l1"]}]' ``` The whole chain is 24 L1 bands, about 15 GB more disk. They share `INDEX_SWARM_MAX_DISK_BYTES` with root-TX bands, so raise the budget too, for example with `--max-disk-gib 100`. Then restart the sidecar by name, with your compose `-f` files: ```bash docker compose --profile index-swarm up -d --no-deps index-swarm ``` ## Publish Your Indexes Any registered gateway can publish. Other gateways then find it in the registry and subscribe by its wallet. Publishing is for gateways that unbundle, so their index holds data items to build bands from. The sidecar publishes the bands under `data/indexes/published/`; it doesn't build them. From Release 85 the `index-export` service builds them from your gateway's own index, once a day. On Release 84, you build them [by hand](#by-hand). ### Prerequisites - A gateway **registered** in the gateway registry, reachable at its registered URL, on Release 84 or later. The routes that serve bands live in the gateway, not the sidecar - Its registered **observer key**, which signs what it publishes - For `index-export`: Release 85, and a gateway that unbundles (`ANS104_UNBUNDLE_FILTER`) - For BitTorrent: the peer and tracker ports open, and this node's public IP. See [Ports](#ports) ### Set the Signing Key Set your gateway's wallet: ```bash AR_IO_WALLET= ``` The sidecar signs with your registered observer key. Set **one** of these, not both: ```bash OBSERVER_PRIVATE_KEY= ``` ```bash # Host path of the observer keypair file; only this file is mounted INDEX_SWARM_OBSERVER_KEYPAIR_FILE=/path/to/observer-keypair.json ``` If one Solana keypair does everything ([Pattern 1](/build/run-a-gateway/manage/solana-migration#supported-configurations)), the gateway reads it through `SOLANA_KEYPAIR_PATH` or `OBSERVER_KEYPAIR_PATH`, and the sidecar sees neither. Set `INDEX_SWARM_OBSERVER_KEYPAIR_FILE` to that keypair file's host path. The setup script refuses to publish, and writes nothing, until the key and `AR_IO_WALLET` are set. Don't use your observer key in a wallet that signs messages for dApps. The sidecar's signatures can't be confused with Solana transactions, but a wallet asked to sign an arbitrary message could produce one. ### Run the Setup Script **Release Requirement**: `--start-height`, `--header-check-url` and the `index-export` service need Release 85, which is not released yet. On Release 84, put your bands in place [by hand](#by-hand) and run the script without those two flags. See what it would change first, then write it to `.env`, without restarting yet: ```bash ./tools/index-swarm-setup --publish --start-height 1950000 --torrent \ --public-host --dry-run ./tools/index-swarm-setup --publish --start-height 1950000 --torrent \ --public-host ``` - `--start-height` is the lowest height `index-export` builds. Choose the lowest height your index holds data items for. - `INDEX_EXPORT_HEADER_CHECK_URL` is the gateway `index-export` checks a sample of each band against before publishing. The script sets it to `https://turbo-gateway.com`; pass `--header-check-url http://core:4000` only if your own gateway holds the root transactions' data. - `--public-host` is where peers reach this node's torrent engine and its tracker. ### Dry-Run the Export On Release 85, see what `index-export` would build before it builds anything, with the same `-f` files your gateway was started with: ```bash docker compose --profile index-``` The report lists each band, its rows, the time and disk it needs, and the result of the header check. Nothing is published. ### Start It Run the setup script again with `--restart`. It changes nothing more, and restarts what publishing needs by service name: the sidecar, the torrent engine and, on Release 85, `index-export`. ```bash ./tools/index-swarm-setup --publish --torrent --restart ``` The first scan reads and hashes every band once, which takes a few minutes for tens of gigabytes. ### Open the Ports With `--torrent`, open the peer port and the tracker port to the internet. See [Ports](#ports). ### Check What You Publish ```bash ./tools/index-swarm-status curl -s https:///ar-io/indexes | jq '{sequence, publisher, bands: [.indexes[].bands[].id]}' curl -s https:///ar-io/info | jq .indexes ``` A gateway can do both: pass `--subscribe` and `--publish` together. If you run more than one node behind a load balancer, read [Publishing from a Fleet](#publishing-from-a-fleet) before anyone subscribes. ### What the Daily Run Does **Release Requirement**: `index-export` needs Release 85, which is not released yet. `index-export` first runs about five minutes after it starts, then daily at `INDEX_EXPORT_RUN_AT_UTC` (04:00 UTC by default). It reads your ClickHouse if `CLICKHOUSE_URL` is set, otherwise your SQLite (`INDEX_EXPORT_SOURCES` changes this). Each run: - builds the history bands once, from `--start-height` up, and never again - rebuilds a daily band from just below the newest history or recent band to the tip, published only when its content changed - once a week, folds the daily rows into a recent band, which keeps rows your index has since expired - checks a sample of each band against `INDEX_EXPORT_HEADER_CHECK_URL`, and publishes it only if the check passes New bands supersede the ones they replace, so subscribers retire the old ones by themselves. A failed run withdraws nothing already published. A rejected band waits for you: `./tools/index-swarm-status` shows it as `FAIL` with the reason. After you fix the cause, run it once by hand: ```bash docker compose --profile index-``` If `published/` already holds bands built another way, `index-export` builds nothing until you hand them over with `--adopt`. The [`index-export` reference](https://github.com/ar-io/ar-io-node/blob/develop/docs/index-export.md) covers adoption, disk and memory limits, metrics and troubleshooting. ### Publish L1 Bands **Release Requirement**: L1 bands need Release 85, which is not released yet. `index-export` also builds L1 bands from your `core.db`, if it holds every block from height 0. A gateway that started above it (`START_HEIGHT`) can't build them. Add both lines to `.env`: ```bash INDEX_EXPORT_KINDS=root-tx-index,parquet-l1 INDEX_SWARM_PUBLISH='[{"name":"root-tx-index","kind":"cdb64-root-tx"},{"name":"parquet-l1","kind":"parquet-l1"}]' ``` The sidecar entry is needed too: the sidecar retires each superseded tip band, so without it `published/parquet-l1/` keeps every one. The chain is cut into the same fixed ranges by every publisher: a band per 100,000 heights, a band per 5,000 heights inside the newest range, and a tip band rebuilt each run. Bands are layout `l1-3`, with [lookup files](/build/advanced/index-publications#lookup-files). The first build of the whole chain takes many hours and reads all of `core.db` once, so start it off-peak. A run starts no new band after `INDEX_EXPORT_L1_RUN_BUDGET_MINUTES` (240 by default), and the next run carries on 15 minutes later. Dry-run it first, capped with `--to`: ```bash docker compose --profile index-``` ### By Hand On Release 84, or to publish bands built some other way, put each band in its own directory under `data/indexes/published/root-tx-index/`, built under a name ending in `.tmp` and renamed into place: ```bash ./tools/export-sqlite-to-cdb64 --partitioned \ --output-dir data/indexes/published/root-tx-index/band-tip.tmp mv data/indexes/published/root-tx-index/band-tip.tmp data/indexes/published/root-tx-index/band-tip ``` This tool runs on the host, so it needs Node.js 20 and `yarn install` in the checkout. Give each band a `heightRange` in its manifest's `metadata` so subscribers install the newest heights first, and replace a band by publishing a new one with `"supersedes": ""`. See [Partitioned Indexes](/build/run-a-gateway/manage/cdb64#partitioned-indexes). ## The Setup and Status Scripts ### `index-swarm-setup` The script edits `.env` and nothing else, unless given `--restart`. | Flag | Effect | | --- | --- | | `--subscribe ` | Adds the publisher to `INDEX_SWARM_SUBSCRIBE`. Repeatable; existing entries are kept. Sets `INDEX_SWARM_MAX_DISK_BYTES` to 50 GiB if unset. Puts `data/indexes/installed/root-tx-index` first in `CDB64_ROOT_TX_INDEX_SOURCES`, keeping what was there (or, if unset, the shipped default), and moves `cdb` right after `db` in `ROOT_TX_LOOKUP_ORDER` (if unset: `db,cdb,gateways,graphql`) | | `--publish` | Adds `root-tx-index` to `INDEX_SWARM_PUBLISH`. Refuses, writing nothing, without a registered key or `AR_IO_WALLET`. With `--torrent` and a public host, sets `INDEX_SWARM_TRACKERS` to this node's tracker. From Release 85, also sets `INDEX_EXPORT_HEADER_CHECK_URL` (default `https://turbo-gateway.com`) and starts `index-export` | | `--start-height ` | Release 85. With `--publish`: sets `INDEX_EXPORT_START_HEIGHT`, the lowest height `index-export` builds | | `--header-check-url ` | Release 85. With `--publish`: the gateway `index-export` checks sampled headers against | | `--torrent` | Generates `INDEX_SWARM_ENGINE_AUTH` if unset (`swarm:` and 48 random hex characters; never printed) | | `--public-host ` | Sets `INDEX_SWARM_ENGINE_PUBLIC_HOST`. From Release 85 it works on its own; on Release 84, pass it with `--torrent` | | `--engine-port ` | Sets `INDEX_SWARM_ENGINE_PORT` (default 6881). From Release 85 it works on its own; on Release 84, pass it with `--torrent` | | `--max-disk-gib ` | Sets `INDEX_SWARM_MAX_DISK_BYTES`. Works on its own too, to change an existing subscriber's budget | | `--no-gateway` | Leaves `CDB64_ROOT_TX_INDEX_SOURCES` and `ROOT_TX_LOOKUP_ORDER` alone | | `--dry-run` | Shows the changes and writes nothing | | `--restart` | Then recreates what needs it: the gateway only when those two settings differ from what it runs with, then the sidecar (and the engine, with torrents), by service name, with the compose files the running gateway was started with | | `--env-file ` | A file other than `.env`, relative to the gateway's directory | It is idempotent: a second run changes only what is missing, so it is also how to add a publisher or turn on torrents later. It never replaces a value it cannot parse or a password it did not write; it stops and says what to fix. Before writing, it copies `.env` to `.env.bak-index-swarm-`, readable only by its owner, since it holds secrets. It warns when an explicit `ROOT_TX_LOOKUP_ORDER` keeps `hyperbeam`, which fails unless you run the `hb` profile, but does not remove it. `INDEX_SWARM_ENGINE_AUTH` alone turns the torrent engine on: with a password set, `INDEX_SWARM_ENGINE_URL` defaults to the engine in the compose file. Set the URL only for an engine you run some other way. ### `index-swarm-status` The status script is read-only. It runs inside the sidecar, so it sees exactly what the sidecar sees, and checks: - that the sidecar is up and the gateway's release is new enough - per publisher: the sequence accepted and its age, any `signature_failed`, `replayed` or `verify_failed`, and failed downloads - installed bands and their size, and bands skipped or nearly skipped for the disk budget - that the gateway reads the installed directory, has every installed root-TX band loaded, and sends lookups to them. It compares root-TX bands only, since the gateway never loads L1 bands - publishing: the document served, its expiry, and how many bands seed - the torrent engine: that it answers, whether any peer has connected in (so a closed port shows), and the day's upload against the budget - from Release 85, `index-export`: that it answers, when each kind of band last succeeded, retries, and rejected bands, for root-TX and L1 bands It exits `1` when a check fails, so it can run from cron or a health script. ## Doing It by Hand What the setup script writes, if you would rather edit `.env` yourself: ```bash INDEX_SWARM_SUBSCRIBE='[{"publisher":"34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5","name":"root-tx-index"}]' INDEX_SWARM_MAX_DISK_BYTES=53687091200 # 50 GiB; about twice what the publisher offers CDB64_ROOT_TX_INDEX_SOURCES=data/indexes/installed/root-tx-index,resources/cdb64-root-tx-index-non-ao-non-redstone-with-content-type-to-height-1820000,resources/cdb64-root-tx-index-non-ao-non-redstone-without-content-type-to-height-1820000,resources/cdb64-root-tx-index-ao-to-height-1820000 ROOT_TX_LOOKUP_ORDER=db,cdb,gateways,graphql INDEX_SWARM_ENGINE_AUTH=swarm: # only for BitTorrent: at least 16 characters, e.g. openssl rand -hex 24 ``` The installed bands go **first** in `CDB64_ROOT_TX_INDEX_SOURCES`, so a fresh band answers before the older shipped indexes. The three `resources/` entries are the shipped default; keep them to go on searching them, or leave them off, since each lookup against them fetches from Arweave and a subscribed set that covers the whole chain makes them redundant. `ROOT_TX_LOOKUP_ORDER` matters most. By default, `cdb` is asked last, after every network source, so your gateway would barely use the bands it downloaded. Drop `hyperbeam` unless you run the `hb` profile. Then restart by service name, with the same `-f` files your gateway was started with: ```bash docker compose up -d --no-deps core docker compose --profile index-swarm --profile index-swarm-torrent \ up -d --no-deps index-swarm-engine-init index-swarm-engine index-swarm ``` Without BitTorrent, leave out the `index-swarm-torrent` profile and the two engine services. Name the services as shown: a bare `docker compose up` also starts or recreates every default service, including your observer. ## Network and Disk ### Ports | Port | Who needs it open | What for | | --- | --- | --- | | Your gateway's HTTP port (via Envoy or NGINX) | Publishers | Serving `/ar-io/indexes`, the band files and the `.torrent` files. Nothing new: it is the port your gateway already serves on | | 6881, TCP and UDP (`INDEX_SWARM_ENGINE_PORT`) | Publishers; recommended for subscribers | Peers connecting to your torrent engine. A publisher needs it, since subscribers must reach its engine. A subscriber works without it, but only peers it connects to can download from it | | 6969, TCP (`INDEX_SWARM_TRACKER_PORT`) | Publishers of torrents | The closed tracker, where peers find each other. Published on every node, but only a publisher listens | Outbound, the sidecar needs HTTPS to each publisher's gateway, and the torrent engine needs to reach peers anywhere on the internet on any port. If your host filters outbound traffic, allow both. The sidecar publishes the tracker port, 6969, whenever it runs, HTTP only included. The torrent engine publishes 6881 when it runs. Either container won't start if another program already holds its port. Move them with `--engine-port` and `INDEX_SWARM_TRACKER_PORT`. Keep the engine port below 32768: Linux uses 32768–60999 for outgoing connections, and one of those can hold the port when the engine restarts. Some hosting providers restrict BitTorrent traffic in their terms of service. Check yours before turning the engine on. ### Disk - **The budget is a ceiling, not a reservation.** `INDEX_SWARM_MAX_DISK_BYTES` stops the sidecar taking more than that, but nothing sets the space aside, so keep at least the budget free on the filesystem. It counts installed bands, downloads in progress and every torrent download twice (the download and the copy that installs), so with that much free, the budget is reached before the disk fills. A band that would go over is skipped, and `index-swarm-status` says so. - **One filesystem.** Everything under `data/indexes` must be on one filesystem: an install is a rename and a publisher seeds from hard links. To put it on another disk, move the whole directory and set `INDEX_SWARM_DATA_PATH`; the gateway mounts the same path. - **SSD or spinning disk.** An installed band answers a lookup in about 2 ms on SSD and tens of milliseconds on a busy spinning disk, still far faster than asking the network. Before installing, the sidecar reads each band once to check it, about a minute for a 7 GB band on a spinning disk. If the same disk also serves your gateway's data, cap the download rate with `INDEX_SWARM_DOWNLOAD_RATE_LIMIT_BYTES_PER_SEC`. ### Bandwidth A first pull of the index turbo-gateway.com publishes is about 21 GB. After that, the tip band is rebuilt daily and the next band every few days. Upload is bounded by the [upload limits](#bounding-upload): at the defaults, up to 100 GB a day, about 3 TB a month. On a host that charges for bandwidth, lower them. ## Firewalls and Docker Ports The torrent engine publishes only its peer port, `INDEX_SWARM_ENGINE_PORT` (6881, TCP and UDP). The sidecar also publishes the tracker port, `INDEX_SWARM_TRACKER_PORT` (6969, TCP), on every node, but only a node that publishes torrents listens on it; a subscriber needn't open it. The engine's Web API is never published. The engine is kept away from the rest of the gateway, because the peers and trackers it talks to are chosen by other gateways. It runs on its own Docker network, shared only with the sidecar, so it can't reach the gateway, the observer, ClickHouse or anything else on the node's network. Its IP filter refuses private, loopback, link-local and carrier-grade NAT addresses for peers, trackers and WebSeeds. It can write only its own download and configuration directories; the bands your gateway serves are mounted read-only. Ports that Docker publishes are forwarded before the host's `INPUT` chain sees them, so a host firewall such as ufw neither blocks nor protects them. Open them wherever traffic reaches the host (a cloud security group or router, for example). To restrict them on the host, filter where Docker forwards. With Docker's default iptables backend, that is the `DOCKER-USER` chain. Its nftables backend has no `DOCKER-USER`, so use a chain of your own table on the `forward` hook, at a priority before Docker's. ## Bounding Upload Seeding is free to peers but not to you: every byte is your upload, and a peer can fetch the bands again and again. Every node that runs the engine seeds, subscribers included, so two limits apply to every node, with defaults: | Variable | Default | Effect | | --- | --- | --- | | `INDEX_SWARM_UPLOAD_LIMIT_BYTES_PER_SEC` | `10000000` (10 MB/s) | Caps the engine's upload rate. `0` is unlimited | | `INDEX_SWARM_UPLOAD_DAILY_LIMIT_BYTES` | `100000000000` (100 GB) | Caps upload per UTC day. Once spent, seeding is throttled to 1 KiB/s until the next UTC day. Downloads and the HTTP routes are unaffected. `0` is no budget | A large publisher should raise both. The engine's memory also grows with the bytes it seeds: it maps the files, and mapped pages count against a container memory limit. ## Tuning the Torrent Engine The defaults suit most gateways. These are the settings worth changing, all in `.env`, applied by restarting the sidecar and engine: | Variable | Default | When to change it | | --- | --- | --- | | `INDEX_SWARM_ENGINE_PORT` | `6881` | Another program holds the port, or your network throttles it. Also `--engine-port` | | `INDEX_SWARM_UPLOAD_LIMIT_BYTES_PER_SEC`, `INDEX_SWARM_UPLOAD_DAILY_LIMIT_BYTES` | 10 MB/s, 100 GB | Lower them on metered bandwidth; raise them on a publisher. See [Bounding Upload](#bounding-upload) | | `INDEX_SWARM_DOWNLOAD_RATE_LIMIT_BYTES_PER_SEC` | no cap | A spinning disk that also serves your gateway | | `INDEX_SWARM_DOWNLOAD_CONCURRENCY` | `4` | Files fetched at once within one band. Lower it on a slow disk or link | | `INDEX_SWARM_WEBSEED_AFTER_SECONDS` | `120` | How long a torrent may stall before the publisher's WebSeed is added. The WebSeed is metered like the HTTP routes | | `INDEX_SWARM_TORRENT_TIMEOUT_SECONDS` | `3600` | How long a torrent may go without progress before the band is fetched over HTTP instead | | `INDEX_SWARM_MAX_DISK_BYTES` | 50 GiB (set by the setup script) | Subscribing to more publishers or to larger indexes. Also `--max-disk-gib` | | `INDEX_SWARM_ENGINE_PUBLIC_HOST` | host of the tracker URL | A publisher whose engine is reached at a different address from its tracker, such as behind a load balancer | ### Sharing Between Your Own Gateways If you run several gateways on one private network, they can seed bands to each other over it instead of each pulling from the internet. By default the engine refuses private addresses, so that other gateways can't use it to reach your network. For a private swarm, set `INDEX_SWARM_ENGINE_BLOCK_PRIVATE=false` on each node and list your own tracker's announce URL in `INDEX_SWARM_ALLOWED_TRACKERS`. Do this only on a network you control. ## Publishing from a Fleet A large gateway is often several nodes behind an HTTP load balancer, and only one of them holds the observer key and signs. ```mermaid graph LR CLIENTS[HTTP clients] PEERS[Subscribers and peers] subgraph Fleet ["Your fleet"] LB[Load balancerand cache] subgraph Signer ["Signing node"] C1[Core Service] S1[Index sidecarpublisher · tracker] E1[Torrent engine :6881] end subgraph Other ["Other nodes"] C2[Core Service] S2[Index sidecarsubscriber, HTTP only] end end CLIENTS --> LB PEERS -->|/ar-io/indexes| LB LB -->|/ar-io/indexes| C1 LB -->|everything else| C2 S2 -->|url: signing node :4000,allowlisted| C1 PEERS |pieces, to the node'sown public address| E1 PEERS -->|announce| S1 classDef service fill:#10b981,stroke:#059669,stroke-width:2px,color:#fff classDef proxy fill:#f59e0b,stroke:#d97706,stroke-width:2px,color:#fff classDef external fill:#8b5cf6,stroke:#7c3aed,stroke-width:2px,color:#fff class C1,C2,S1,S2,E1 service class LB proxy class CLIENTS,PEERS external ``` For BitTorrent: 1. **One node publishes and seeds.** The signing node, the one `/ar-io/indexes` is sent to, runs the engine and the tracker. The other nodes need neither. 2. **The peer port reaches that node directly.** BitTorrent is not HTTP, so the load balancer can't carry it. Publish `INDEX_SWARM_ENGINE_PORT` (TCP and UDP) on the node's own public address, make sure the internet reaches it there (Docker-published ports bypass the host firewall; see [Firewalls and Docker ports](#firewalls-and-docker-ports) to restrict them), and set `INDEX_SWARM_ENGINE_PUBLIC_HOST` (`--public-host`) to that address. Otherwise the tracker lists the engine under the host of its tracker URL, which for a fleet is the load balancer. 3. **The tracker, one of two ways:** - Directly: publish `INDEX_SWARM_TRACKER_PORT` on the same address and announce to `http://:6969/announce` in `INDEX_SWARM_TRACKERS`. - Through the load balancer: route `/announce` to the signing node's tracker port, uncached, with `proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;`, and list the proxies' addresses in `INDEX_SWARM_TRACKER_TRUSTED_PROXIES`. Otherwise every peer appears at the proxy's address, the tracker's per-address limits throttle them together, and it hands out an address nobody can connect to. 4. **The `.torrent` and WebSeed routes** are under `/ar-io/indexes`, so the [NGINX block](#behind-nginx) for that prefix already covers them. ### Rate Limiting and x402 in a Fleet Nothing extra to configure. The rate limiter and x402 apply to the byte routes (files by name, by digest, and the WebSeed) the same way they apply to data. The publication and the `.torrent` files are free. Peer-to-peer transfer and tracker announces never reach the gateway at all; the upload budget bounds them instead. Because `/ar-io/indexes` goes to one node, all metering happens there, so per-address limits stay consistent even if your nodes don't share a rate limiter. ### Advertising From the Other Nodes **Release Requirement**: `INDEXES_ADVERTISE_FROM_URL` needs Release 85, which is not released yet. `/ar-io/info` describes the node that answered, so leave it on every node. A node that doesn't sign has no `indexes` block of its own. Set `INDEXES_ADVERTISE_FROM_URL` on each of those nodes to the signing node's gateway URL. The node then fetches the publication every minute and advertises the same `indexes` block, as long as the signature verifies and the publisher is its own `AR_IO_WALLET`. ### Giving the Other Nodes the Index If your other nodes answer lookups from their own disk, they need the bands installed too. Subscribe each one over HTTP to your own publication, pointed at the publishing node directly: ```bash INDEX_SWARM_SUBSCRIBE='[{"publisher":"","name":"root-tx-index","url":"http://:4000"}]' ``` `url` only changes where the publication and files are fetched from. The signature is still checked against your registered observer key, so an internal address is safe. Then add those nodes' addresses to `RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST` on the publishing node, so they aren't rate-limited or asked to pay (the allowlist applies when the rate limiter is on). They don't need a torrent engine: between nodes on one network, HTTP is simpler. Subscribers behind NAT still work: they reach the publisher's engine, and a reachable subscriber can be reached back. ## Behind NGINX Give `/ar-io/indexes` its own uncached location, with no trailing slash, that forwards the client IP. A cache breaks range reads and per-subscriber rate limits. [Advanced NGINX Caching](/build/run-a-gateway/manage/nginx-caching#index-sharing) has the location block. ## Monitoring `./tools/index-swarm-status` covers the day-to-day checks. The sidecar also serves Prometheus metrics on port `9101` inside its container. The ones worth watching: | Metric | What it tells you | | --- | --- | | `index_subscription_manifest_age_seconds` | How old each publisher's latest publication is. **Alarm if it climbs past a day**: the publisher has gone quiet. | | `index_swarm_installed_bands` | How many bands are live | | `index_subscription_total{result}` | Outcomes of each poll. `signature_failed` should always be zero. `transport_fallback` means a torrent was not used and the band came over HTTP | | `index_subscription_bytes_total{transport}` | Bytes fetched over `http` or `torrent`: how much the swarm carries | | `index_swarm_engine_available` | `1` while the torrent engine answers. Absent when none is configured | | `index_swarm_upload_today_bytes`, `index_swarm_upload_throttled` | Seeding today against the daily budget; `1` means it is spent and seeding is throttled until the next UTC day | | `index_publish_total{result}` | On a publisher: `published`, `unchanged` or `failed` | | `index_publish_seeding_bands` | On a publisher: bands handed to the engine. Below `index_publish_bands` means some are offered over HTTP only | | `index_swarm_tracker_seeding_hosts`, `index_swarm_tracker_seeders` | On a publisher running the tracker: who shares your index. Seeding hosts are the other hosts seeding at least one of your bands, roughly how many gateways share it; seeders are the peers holding a whole band, per band. Both are as peers report them, and peers found only through DHT or peer exchange aren't counted. From Release 85 | On the gateway, `indexes_requests_total{route,status}` shows who is downloading from you, by route (`publication`, `file`, `blob`, `torrent`, `webseed`), including `402` and `429` from your rate limits. **Release Requirement**: `index-export` and its metrics need Release 85, which is not released yet. `index-export` serves its own metrics on port `9102` inside its container: | Metric | What it tells you | | --- | --- | | `index_export_runs_total{index,kind,result}` | Band builds by outcome: `published`, `unchanged`, `skipped`, `couldnt_check` (retried by itself) or `rejected` (waits for you). **Alarm on `rejected`** | | `index_export_last_success_timestamp_seconds{index,kind}` | When a kind of band last published or was found unchanged. **Alarm if the daily band (`kind="d"`) is more than two days old** | ## Troubleshooting Run `./tools/index-swarm-status` first: it names most problems and their fix. **Nothing installs, and the logs show `402` or `429`.** The publisher is rate-limiting you. It is expected on a first download over HTTP: each poll picks up where the last stopped. Turn on BitTorrent, ask the publisher to raise your limits, or wait. **Bands installed, but lookups still go to the network.** Check `ROOT_TX_LOOKUP_ORDER` puts `cdb` right after `db`, and that `CDB64_ROOT_TX_INDEX_SOURCES` starts with `data/indexes/installed/root-tx-index`. Both need a gateway restart; `./tools/index-swarm-setup --restart` does it. **No peer has connected in.** Port 6881 is closed somewhere between the internet and the engine. Bands still arrive, from peers and over HTTP, and the peers you connect to can download from you, but peers that can't reach you can't. Open or forward the port if you can; for a subscriber it is optional. **`Bind for 0.0.0.0:6969 failed: port is already allocated`** (or `6881`). Another program on the host holds the port, so the sidecar (6969) or the engine (6881) can't start. Move the engine with `--engine-port`, or the tracker with `INDEX_SWARM_TRACKER_PORT`, and restart. **The engine refuses the sidecar.** Check `INDEX_SWARM_ENGINE_AUTH`. The sidecar does not retry a failed login, because the engine bans an address after five. **`Publication was signed by an unregistered key`.** The publication you fetched was not signed by the observer key registered for that wallet. Check the publisher's registry record; if it just changed keys, it can take up to an hour to reach your gateway's registry view. **`Set OBSERVER_KEYPAIR_PATH or OBSERVER_PRIVATE_KEY, not both`.** On a publisher, `OBSERVER_PRIVATE_KEY` and `INDEX_SWARM_OBSERVER_KEYPAIR_FILE` are both set (inside the sidecar, `INDEX_SWARM_OBSERVER_KEYPAIR_FILE` becomes `OBSERVER_KEYPAIR_PATH`). Keep one. **`Gateway is too old to load installed index bands`.** Upgrade the gateway to Release 84 or later. The sidecar notices the upgrade by itself. ## Turning It Off Stopping the sidecar changes nothing your gateway serves: installed bands stay loaded, and published bands stay served from the last publication written. ```bash docker compose --profile index-swarm stop index-swarm docker compose --profile index-swarm rm -f index-swarm ``` With the torrent engine, also stop and remove `index-swarm-engine` and `index-swarm-engine-init` (profile `index-swarm-torrent`), then delete `data/indexes/swarm/`, `data/indexes/torrents/` and `data/index-swarm-engine/`. Remove `INDEX_SWARM_ENGINE_AUTH` from `.env` too: while it is set, the sidecar expects the engine and warns that it is not answering. - **On a subscriber**, restore your previous `CDB64_ROOT_TX_INDEX_SOURCES` and restart the gateway, then delete `data/indexes/installed/`. In that order, so the gateway is no longer holding the files open. - **On a publisher**, delete `data/indexes/published/publication.json`. The gateway stops serving the routes on its next request, without a restart. The band directories can then go too. From Release 85, also stop and remove `index-export` (profile `index-export`); it writes its bands as root, so removing them takes `sudo`. ## Related } /> } /> } /> } /> # Importing SQLite Database Snapshots (/build/run-a-gateway/manage/index-snapshots) ## Overview A new gateway builds its index of the Arweave network block by block. That can take days or weeks, depending on your hardware and on how much data you index. You can skip most of it in one of two ways: - [Import a SQLite snapshot](#import-a-sqlite-snapshot). The 2025-04-23 snapshot holds transactions and the data items uploaded through ArDrive products, including Turbo, up to block 1645229. - [Bootstrap from L1 bands](#bootstrap-from-l1-bands) (Release 85). Your gateway imports the Arweave base layer (blocks, transactions, tags and owners) from signed bands that another gateway publishes, up to the current tip. Data items are not included. Looking to resolve data items faster without a full database? [Index Sharing](/build/run-a-gateway/manage/index-sharing) lets your gateway subscribe to another gateway's root transaction indexes, about 21 GB today and kept current, instead of importing a snapshot. The below instructions are designed to be used in a linux environment. Windows and MacOS users must modify the instructions to use the appropriate package manager/ command syntax for their platform. Unless otherwise specified, all commands should be run from the root directory of the gateway. ## Import a SQLite Snapshot SQLite snapshots are large and hard to update in steps, so ar.io distributes them over BitTorrent. Importing a snapshot deletes your existing database and replaces it with the snapshot. ### Install a Torrent Client Any BitTorrent client works. For example, [transmission-cli](https://github.com/transmission/transmission): ```bash # Ubuntu/Debian sudo apt-get install transmission-cli # CentOS/RHEL sudo yum install transmission-cli # macOS brew install transmission-cli ``` ### Download the Snapshot ```bash transmission-cli "magnet:?xt=urn:btih:62ca6e05248e6df59fac9e38252e9c71951294ed&dn=2025-04-23-sqlite.tar.gz&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=http%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=https%3A%2F%2Ftracker.bt4g.com%3A443%2Fannounce" ``` This downloads `2025-04-23-sqlite.tar.gz`, about 42.8 GB. Check that it arrived whole with `ls -lh 2025-04-23-sqlite.tar.gz`. Seeding the torrent after the download is not required, but it keeps the snapshot available for other operators. ### Extract the Archive ```bash tar -xzf 2025-04-23-sqlite.tar.gz ls -la 2025-04-23-sqlite/ ``` This creates a directory named after the file, without `.tar.gz`, that holds the SQLite database files. If you use a different snapshot, replace the file name. ### Stop the Gateway ```bash docker compose down ``` If you start your gateway with your own `-f` files, pass the same ones to every `docker compose` command on this page. ### Replace the Database Back up your existing database (optional), then move the snapshot into `data/sqlite`: ```bash mkdir sqlite-backup mv data/sqlite/* sqlite-backup/ mv 2025-04-23-sqlite/* data/sqlite/ ``` If you skip the backup, delete the old files with `rm data/sqlite/*` before the last command. ### Start the Gateway ```bash docker compose up -d ``` ### Verify the Import Watch the gateway logs for the block height being imported: ```bash docker compose logs -f core ``` The 2025-04-23 snapshot was taken at block `1645229`. If the import worked, the gateway imports blocks from 1645230 upward. The [Grafana extension](/build/extensions/grafana) also shows the last block imported. ## Bootstrap from L1 Bands **Release Requirement**: L1 bands, and the `ar-io-node` CLI that imports them, arrive in gateway Release 85, which is not released yet. Your gateway and the publisher you subscribe to must both run it. An L1 band (`parquet-l1`) holds the Arweave base layer for a range of block heights, as Parquet files: blocks, transactions, tags and owners. Gateways that publish them sign every file, and your gateway checks each one before it installs the band. `index-l1-import` then fills your gateway's `core.db` from the installed bands, so it starts near the tip instead of at block 0. L1 bands carry no data items. Your gateway still unbundles the bundles your filters select, and it does that for imported history only if you also set `BACKFILL_BUNDLE_RECORDS`. ### Prerequisites - A gateway on Release 85. The core service migrates its database when it starts, and the import refuses a `core.db` that has not been migrated - A `core.db` with no unstable blocks near the tip (the `new_*` tables). The import refuses one that holds them, so run it on a new gateway before its block importer has indexed the tip - Room on disk: about 15 GB for the bands, and about 120 GB for a `core.db` that covers the whole chain. The import refuses to start without the space it needs - Time: a full chain takes many hours. The import logs its progress once a minute ### Subscribe to L1 Bands L1 bands arrive through [Index Sharing](/build/run-a-gateway/manage/index-sharing). A subscription takes them only when it names them, so add `parquet-l1` to the `name` list in `INDEX_SWARM_SUBSCRIBE`: ```bash INDEX_SWARM_SUBSCRIBE='[{"publisher":"34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5","name":["root-tx-index","parquet-l1"]}]' ``` Then recreate the `index-swarm` sidecar and wait until `./tools/index-swarm-status` shows the bands installed under `data/indexes/installed/parquet-l1`. ### Stop the Gateway The import refuses a `core.db` that another process is writing to. Stop only the core service, with the same `-f` files your gateway was started with: ```bash docker compose stop core ``` ### Import the Bands Run it from the gateway's directory. The tool runs in the core image, so the host needs only Docker: ```bash ./tools/ar-io-node index-l1-import \ --bands-dir data/indexes/installed/parquet-l1 \ --core-db data/sqlite/core.db \ --cache-mib 8192 ``` Bands import in height order, lowest first. `--cache-mib` sets SQLite's page cache for the run (default 1024 MiB); give it what your machine can spare, since a larger cache keeps a long import fast. An interrupted run is safe: run the command again and it carries on from the last band that landed. `--max-bands` splits the work across several runs. If your gateway started above block 0 (`START_HEIGHT`), add `--from 0 --to ` to fill in the history underneath. Check that the result's `holes` field is absent or empty before you start the gateway. ### Start the Gateway ```bash docker compose up -d --no-deps core ``` The block importer continues from the highest height imported. To check a set of bands, or your gateway's own `core.db`, against the chain, use `./tools/ar-io-node index-l1-verify`. Every option, and the JSON each command prints, is in the [`ar-io-node` CLI reference](https://github.com/ar-io/ar-io-node/blob/develop/docs/cli.md#index-l1-import). # Manage your Gateway (/build/run-a-gateway/manage) import { Database, RotateCcw, TrendingUp, Zap, Globe, Shield, Settings, Wrench, CreditCard, Search, Bot, Share2, } from "lucide-react"; Guides for configuring and running your ar.io gateway, from performance tuning to content moderation. ## Gateway Management }> Learn how to import pre-synchronized database snapshots to quickly bootstrap your gateway and reduce initial sync time from weeks to hours. }> Step-by-step guide to safely upgrade your ar.io Gateway to the latest version without losing data or progress. }> Authorize an operations address to update your gateway's metadata and claim the ArNS gateway operator discount, so your operator wallet can stay offline. ## Monitoring & Analytics } > Deploy and configure Grafana to monitor your gateway's health and performance. ## Performance Optimization }> Configure advanced filters to efficiently process and index only the data you need, optimizing performance and resource usage. }> Configure the CDB64 index for O(1) data item lookups. Three indexes are enabled by default, covering close to 3.8 billion records, so historical data items resolve locally. }> Subscribe to another gateway's signed indexes so your gateway finds data items locally, or publish your own. } > Customize your gateway's root domain to serve custom content, project information, or documentation instead of default network info. }> Configure the x402 payment protocol so clients can pay in cryptocurrency for data beyond their free allowance. ## Content Management } > Implement content moderation policies using blocklisting and filtering to control what content your gateway serves. ## Configuration Reference } > Reference for every ar.io gateway environment variable, organized by service component. ## AI-Assisted Operations } > The `ar-io-node` repo ships a Claude Code skill at `.claude/skills/ar-io-gateway-operator/` with a one-screen `scripts/health-check` snapshot and an operational runbook covering the ANS-104 pipeline, ClickHouse, ArNS resolution, observer/cranker, and common pitfalls. Claude Code auto-loads it when run from the repo root. Readable as plain Markdown if you don't use Claude Code. ## Support & Troubleshooting } > Troubleshooting guide and FAQ for common gateway issues, failed epoch guidance, and frequently asked questions. # Advanced NGINX Caching (/build/run-a-gateway/manage/nginx-caching) The [quick-start guide](/build/run-a-gateway/quick-start) covers basic NGINX reverse proxy setup for SSL termination and header forwarding. This guide covers adding a caching layer in front of your ar.io node for high-traffic gateways, based on production configurations running at scale. NGINX caching is optional. The ar.io node has its own internal caching and serves data correctly without it. Add NGINX caching when you need to reduce load on the node process or serve high volumes of traffic for popular content. ## Prerequisites - A running ar.io gateway with NGINX already configured as a reverse proxy (see the [quick-start guide](/build/run-a-gateway/quick-start)) - Root or sudo access on the host running NGINX If you are running the default Docker Compose setup, NGINX runs on the host and proxies to **envoy** on port 3000, which in turn proxies to the ar.io node core on port 4000. The `proxy_pass http://127.0.0.1:3000` directives in this guide target envoy, which is the correct entry point. The cache directories described below are on the host filesystem. If you already have an NGINX configuration (e.g., `/etc/nginx/sites-available/default` from the quick-start), you will be adding cache directives to it. The `http`-block directives (cache zones, maps) go outside your existing `server` block, and the location blocks replace or extend the ones in your existing `server` block. ## Why Cache at NGINX Arweave data is immutable - once a transaction is confirmed, its content never changes. This makes it an ideal candidate for aggressive edge caching: - **Reduce node load** - Serve repeated requests for the same content directly from NGINX's disk cache without hitting the node process. - **Thundering herd protection** - When many clients request the same uncached content simultaneously, NGINX ensures only one request reaches the node while others wait for the cached result. - **Stale serving during failures** - If the node is temporarily unavailable, NGINX can serve stale cached content rather than returning errors. - **Faster response times** - Cached responses skip the node entirely, reducing latency. **Rate limiter interaction**: Once NGINX caches a significant portion of traffic, fewer requests reach the ar.io node's rate limiter. This effectively increases per-IP limits for cached content. This is generally beneficial but means abusive clients can hammer cached endpoints without triggering rate limits. Consider this when tuning rate limit values. ## http Block Configuration All `proxy_cache_path` and `map` directives must be placed in the `http` block of your NGINX config, **outside** the `server` block. On Debian/Ubuntu, files in `/etc/nginx/sites-enabled/` are included inside the `http` block via `nginx.conf`, so you can place these directives at the top of your site config file (before the `server` block). ### Create Cache Directories Create the directories and set ownership before reloading NGINX. NGINX will fail to start if these directories don't exist. ```bash sudo mkdir -p /var/lib/nginx/cache/api sudo mkdir -p /var/lib/nginx/cache/block sudo mkdir -p /var/lib/nginx/cache/tx-and-chunk sudo mkdir -p /var/lib/nginx/cache/data sudo mkdir -p /var/lib/nginx/cache/arns sudo chown -R www-data:www-data /var/lib/nginx/cache ``` ```bash sudo mkdir -p /var/lib/nginx/cache/api sudo mkdir -p /var/lib/nginx/cache/block sudo mkdir -p /var/lib/nginx/cache/tx-and-chunk sudo mkdir -p /var/lib/nginx/cache/data sudo mkdir -p /var/lib/nginx/cache/arns sudo chown -R nginx:nginx /var/lib/nginx/cache ``` If your cache directories are on a different filesystem than NGINX's temp path, cached file writes will use slow cross-filesystem copies instead of fast renames. For best performance, keep cache directories on the same filesystem as NGINX's default temp path. Also avoid placing the cache on the same disk as your ar.io node's SQLite/ClickHouse databases if I/O is constrained. ### Tiered Cache Zones Different types of gateway content have different caching characteristics. A tiered approach uses separate cache zones sized and configured for each content type: ```nginx # --- Tiered cache zones (http block, outside server) --- # High churn - small short-TTL API responses proxy_cache_path /var/lib/nginx/cache/api levels=1:2 keys_zone=api_cache:10m max_size=1g inactive=10m use_temp_path=off; # Block metadata - highly cacheable, rarely changes proxy_cache_path /var/lib/nginx/cache/block levels=1:2 keys_zone=block_cache:10m max_size=10g inactive=365d use_temp_path=off; # TX + chunk metadata - tens of millions of items proxy_cache_path /var/lib/nginx/cache/tx-and-chunk levels=1:2 keys_zone=tx_and_chunk_cache:30m max_size=30g inactive=90d use_temp_path=off manager_threshold=500ms loader_files=1000 loader_threshold=500ms; # Data - immutable Arweave content, bulk of disk usage # Set max_size based on your available disk space proxy_cache_path /var/lib/nginx/cache/data levels=1:2 keys_zone=data_cache:200m max_size=500g inactive=90d use_temp_path=off manager_threshold=500ms loader_files=1000 loader_threshold=500ms; # ArNS - web app content served via ArNS subdomains proxy_cache_path /var/lib/nginx/cache/arns levels=1:2 keys_zone=arns_cache:10m max_size=50g inactive=30d use_temp_path=off; ``` Size the `data_cache` zone based on your actual available disk space. The data cache will be the largest zone by far. Leave headroom for the node's own data storage, databases, and OS needs. The `keys_zone` memory size determines how many entries can be tracked — 1MB holds approximately 8,000 keys. | Parameter | Purpose | |-----------|---------| | `levels=1:2` | Two-level directory structure for cache files. Prevents any single directory from containing too many files. | | `keys_zone=name:size` | Shared memory zone for cache keys. Size based on expected number of cached items. | | `max_size` | Maximum disk space for this cache zone. NGINX evicts least-recently-used entries when exceeded. | | `inactive` | Remove entries not accessed within this period, even if not expired. | | `use_temp_path=off` | Write cache files directly to the cache directory (avoids cross-filesystem copies). | | `manager_threshold` | Maximum time the cache manager spends per cleanup iteration. Prevents disk I/O spikes on large caches. | | `loader_files` / `loader_threshold` | Controls how NGINX loads cache metadata on startup. Important for large caches to avoid slow restarts. | ### Map Directives All `map` directives must also be in the `http` block. These must be defined **before** the `server` block because the server references the variables they create. ```nginx # --- Map directives (http block, outside server) --- # WebSocket upgrade support (used by /graphql and catch-all) map $http_upgrade $connection_upgrade { default upgrade; '' close; } # Route ArNS subdomains to arns_cache, everything else to api_cache. # Excludes sandbox subdomains (52-char base32) which are not ArNS. map $host $catch_all_cache { "~^[a-z2-7]{52}\." api_cache; "~^[^.]+\..+\..+" arns_cache; default api_cache; } # Never cache 429 responses — prevent rate-limit errors from # overwriting valid cache entries. map $upstream_status $no_cache_429 { 429 1; default 0; } # Never cache responses with Cache-Control: no-store — prevents # caching 402 Payment Required and other uncacheable responses. map $upstream_http_cache_control $no_cache_no_store { "~no-store" 1; default 0; } ``` ## Cache Key Design For Arweave content, query strings are not meaningful - the content is identified by its transaction ID in the URL path. The cache key excludes query strings to prevent cache fragmentation: ```nginx proxy_cache_key "$scheme://$host$uri"; ``` The `$host` component is important because ArNS subdomains serve different content at the same path. ## Server Block Configuration The following directives go inside your `server` block. If you have an existing server block from the quick-start guide, merge these directives into it. ### Shared Defaults These directives apply as defaults across all locations. Individual locations override them as needed: ```nginx server { # ... your existing SSL, server_name, listen directives ... # Default cache zone (overridden per-location) proxy_cache api_cache; proxy_cache_key "$scheme://$host$uri"; # Thundering herd protection: first request fetches from origin, # others wait up to lock_timeout, then go to origin if lock_age exceeded proxy_cache_lock on; proxy_cache_lock_age 60s; proxy_cache_lock_timeout 60s; # Serve stale entries during errors, timeouts, or background updates proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504; proxy_cache_revalidate on; proxy_cache_background_update on; # Never cache 429 or no-store responses proxy_no_cache $no_cache_429 $no_cache_no_store; proxy_cache_bypass $no_cache_429 $no_cache_no_store; # Error TTLs shared across all locations proxy_cache_valid 400 60s; proxy_cache_valid 403 10s; proxy_cache_valid 451 30d; proxy_cache_valid 500 502 503 504 10s; # Prevent cache fragmentation from Vary headers. # Trade-off: this drops ALL Vary values, not just Vary: Origin. # For Arweave gateways this is safe because the node does not use # Vary: Accept-Encoding for content negotiation. proxy_ignore_headers Vary; # Proxy buffer settings proxy_buffering on; proxy_buffer_size 32k; proxy_buffers 64 32k; proxy_max_temp_file_size 8192m; # Default timeouts proxy_read_timeout 120s; proxy_send_timeout 30s; # ... location blocks follow ... } ``` ### Route-Specific Caching #### Immutable Data (30-day TTL) Transaction data and raw content are immutable on Arweave. Cache aggressively with extended timeouts for large file downloads: ```nginx # Raw data endpoint location ^~ /raw/ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_read_timeout 600s; proxy_send_timeout 600s; # Lock holder gets 10 minutes for large file downloads; # waiters give up after 5s and go to origin directly proxy_cache_lock_age 600s; proxy_cache_lock_timeout 5s; proxy_cache data_cache; # Disable background refresh — avoids expensive multi-GB origin fetches proxy_cache_background_update off; proxy_ignore_headers Set-Cookie Vary; proxy_hide_header Set-Cookie; proxy_cache_valid 200 30d; proxy_cache_valid 404 60s; } # Transaction data (43-char base64url IDs) location ~ "^/[a-zA-Z0-9_-]{43}(/|$)" { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_read_timeout 600s; proxy_send_timeout 600s; proxy_cache_lock_age 600s; proxy_cache_lock_timeout 5s; proxy_cache data_cache; proxy_cache_background_update off; proxy_ignore_headers Set-Cookie Vary; proxy_hide_header Set-Cookie; proxy_cache_valid 200 30d; proxy_cache_valid 404 60s; } ``` Note that `/raw/TX_ID` and `/TX_ID` serve the same content but are cached as separate entries because the cache key includes the URI path. For most gateways this duplication is acceptable. If disk space is tight, you can normalize the cache key with a `map` — see the [Troubleshooting](#troubleshooting) section. #### Chunks (24-hour TTL) Arweave chunks are immutable but accessed less frequently than full transactions: ```nginx location ^~ /chunk/ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache tx_and_chunk_cache; proxy_cache_background_update off; proxy_ignore_headers Cache-Control Expires Set-Cookie Vary; proxy_hide_header Set-Cookie; proxy_cache_valid 200 24h; proxy_cache_valid 404 30s; } ``` #### Volatile Metadata (Short TTLs) API endpoints and metadata change frequently and need short TTLs: ```nginx # Health, info, metrics, tx_anchor (2-minute TTL) location ~ ^/(health|info|metrics|tx_anchor)$ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache_valid 200 120s; proxy_cache_valid 404 30s; } # Height and time (20-second TTL - changes every block) location ~ ^/(height|time)$ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache_valid 200 20s; proxy_cache_valid 404 30s; } # Peers, current_block (30-second TTL) location ~ ^/(peers|current_block)$ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache_valid 200 30s; proxy_cache_valid 404 15s; } # Wallet, price, unconfirmed TX lookups (30-second TTL) location ~ ^/(wallet|price|unconfirmed_tx)/ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache_valid 200 30s; proxy_cache_valid 404 15s; } # AR.IO API (30-second TTL) location ^~ /ar-io/ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache_valid 200 30s; proxy_cache_valid 404 5s; } # Block metadata location ^~ /block/ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache block_cache; proxy_cache_valid 404 30s; } # TX metadata location /tx/ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache tx_and_chunk_cache; proxy_cache_valid 404 30s; } # TX status (30-second TTL) location ~ ^/tx/[A-Za-z0-9_-]+/status$ { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache tx_and_chunk_cache; proxy_cache_valid 200 30s; proxy_cache_valid 404 30s; } # Pending transactions (20-second TTL) location = /tx/pending { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache tx_and_chunk_cache; proxy_cache_valid 200 20s; proxy_cache_valid 404 30s; } # Current block (2-minute TTL) location = /block/current { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; add_header X-Cache-Status $upstream_cache_status always; proxy_cache_valid 200 120s; proxy_cache_valid 404 30s; } ``` #### Endpoints to Never Cache Some endpoints must never be cached because they handle writes, WebSocket connections, or state-changing operations. These locations use `proxy_cache off` and explicitly set `Cache-Control: no-store` (this is a response header injected by NGINX, distinct from the upstream `no-store` bypass in the map directives): ```nginx # GraphQL - supports WebSocket upgrades and POST mutations location = /graphql { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_cache off; proxy_hide_header Cache-Control; add_header Cache-Control "no-store" always; } # Transaction submission (POST endpoint) location = /tx { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_cache off; proxy_hide_header Cache-Control; add_header Cache-Control "no-store" always; } # Chunk upload (POST endpoint) location = /chunk { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_cache off; proxy_hide_header Cache-Control; add_header Cache-Control "no-store" always; } ``` #### Catch-All with ArNS Routing The catch-all location handles remaining requests including ArNS subdomain content: ```nginx location / { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; add_header X-Cache-Status $upstream_cache_status always; proxy_cache $catch_all_cache; proxy_cache_valid 200 300s; proxy_cache_valid 404 30s; } ``` ## Index Sharing If your gateway [shares indexes](/build/run-a-gateway/manage/index-sharing), give `/ar-io/indexes` its own location block, **uncached**. There are two reasons, and the second is easy to miss. The gateway already marks each response correctly (errors are `no-store`, and files are `private` when you meter them), but an `/ar-io/` block with its own `proxy_cache_valid` or stale-serving rules could otherwise keep a publication a subscriber should not see. **A cache zone on this prefix also breaks range requests.** When a location has a cache zone, NGINX strips the client's `Range` from the upstream request and fetches the whole object so it can store it. It does that before it has seen any `Cache-Control`, so no response header prevents it. `proxy_buffering off` on its own does not fix it. A `proxy_cache_bypass` keyed on `$upstream_http_cache_control` cannot help either, because that variable is empty when the bypass is evaluated. Only `proxy_cache off` works. What the client sees depends on whether the response was cacheable, so the symptom differs between gateways: | Your gateway's `Cache-Control` for a file by name | Through a cache zone | | --- | --- | | `public, no-cache` (you do not meter) | NGINX stores it and answers from cache, so the client does get a `206` | | `private, no-cache` (you meter) | Nothing can be stored, so NGINX has no cached object to answer a range from and returns `200` with the whole body, on **every** request | Measured against a controlled 5 MB upstream: a cacheable response answered a 100-byte range with `206` and 100 bytes, while an uncacheable one returned `200` and all 5,000,000 bytes, twice, never caching. On a metering gateway serving a 541 MB band file the same request returned all 541,753,181 bytes, and the next one returned `402` because that transfer had spent the allowance. **A cache zone is wrong even on the branch that returns `206`.** The whole file still crosses from your gateway on the first range request, so the latency and the egress are paid either way. Band files reach 1.35 GB in a zone sized for small objects. If you meter, it is worse than wrong: it breaks [BitTorrent WebSeeds](#torrent-routes) and [querying a dataset in place](/build/advanced/index-publications#query-a-parquet-dataset). It also fails quietly: a `200` carrying more bytes than were asked for looks like success to anything that does not check the status code. Keep the prefix without a trailing slash, as below. NGINX answers a request for a `proxy_pass` location whose prefix ends in `/` with a `301` to the slashed form, and `/ar-io/indexes` is the publication every subscriber polls. HTTPSig signs `@path`, so a redirect moves the signed path. Band sizes depend on the index: a `cdb64-root-tx` partition is 7 to 30 MB, while a `parquet-l1` band reaches 1.35 GB with a single table file over 500 MB. Turn off buffering so a large file streams instead of being staged: ```nginx # Longer prefix than any /ar-io/ block, so it wins. location ^~ /ar-io/indexes { proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_cache off; proxy_buffering off; proxy_max_temp_file_size 0; proxy_read_timeout 300s; } ``` Check it after reloading, at every layer that serves the gateway: ```bash BAND=/ar-io/indexes/parquet-l1//transactions.parquet for u in http://localhost:4000 http://localhost:3000 https://; do echo "$u -> $(curl -s -o /dev/null -w '%{http_code} %{size_download}' -r 0-99 "$u$BAND")" done ``` Every layer should print `206 100`. The gateway and Envoy are always correct, so a layer that prints `200` and the full size is the one to fix. Two things cost time here. A TLS listener and an internal cache listener often live in **different files**, so fixing one can leave the other broken. And `nginx -s reload` retires old workers gracefully, so a request made right after a reload can still be answered by the old configuration. Re-test a few seconds later before concluding a change did not work. **More than one node?** Only the node that holds your observer key signs publications. Point this block's `proxy_pass` at that node on every node, so subscribers never reach one that doesn't publish. ### Torrent Routes A gateway that publishes over BitTorrent serves two more routes. Both live under `/ar-io/indexes`, so the block above already covers them, and so do any pin or cache rules you have for that prefix: | Route | What it serves | `Cache-Control` on success | | --- | --- | --- | | `/ar-io/indexes/torrents/.torrent` | A band's `.torrent` file. Not metered | `public, max-age=86400` | | `/ar-io/indexes/webseed//` | A band file for torrent clients (the WebSeed). Metered like the blob route | As the blob route: `private` when metered, else `public`, `max-age=31536000, immutable` | Errors from both are `no-store`, as for every index route. The tracker is not a gateway route: it runs in the sidecar, on its own port (6969). To serve it through NGINX or a load balancer, see [Publishing from a Fleet](/build/run-a-gateway/manage/index-sharing#publishing-from-a-fleet). ## TTL Summary | Route | Cache Zone | 200 TTL | 404 TTL | Notes | |-------|-----------|---------|---------|-------| | `/raw/`, `/[TX_ID]` | data_cache | 30d | 60s | Immutable content, extended timeouts | | `/chunk/` | tx_and_chunk_cache | 24h | 30s | Immutable chunks | | `/block/` | block_cache | origin | 30s | Origin-controlled TTL | | `/block/current` | api_cache | 120s | 30s | Changes every block | | `/tx/` | tx_and_chunk_cache | origin | 30s | Origin-controlled TTL | | `/tx/[ID]/status` | tx_and_chunk_cache | 30s | 30s | Status can change | | `/tx/pending` | tx_and_chunk_cache | 20s | 30s | Changes frequently | | `/ar-io/` | api_cache | 30s | 5s | Short-lived API data | | `/health`, `/info` | api_cache | 120s | 30s | Metadata endpoints | | `/height`, `/time` | api_cache | 20s | 30s | Changes every block | | `/peers`, `/current_block` | api_cache | 30s | 15s | Network state | | `/wallet/`, `/price/` | api_cache | 30s | 15s | Volatile data | | `/` (catch-all) | dynamic | 300s | 30s | ArNS or API cache | | `/graphql`, `/tx` POST, `/chunk` POST | none | - | - | Never cached | | `/ar-io/indexes` | none | - | - | [Index Sharing](#index-sharing); the gateway sets `Cache-Control` | ## Content Moderation & Cache Purging Caching immutable content for 30 days creates a compliance risk: if the ar.io node blocks a transaction via [content moderation](/build/run-a-gateway/manage/content-moderation), NGINX may continue serving the cached copy for the remainder of the TTL. This is a compliance concern for production gateways. If you use content moderation filters, you need a way to force-refresh cached entries after blocking. ### Cache Bypass Header Add a `map` that checks a secret bypass header value, and use it with `proxy_cache_bypass`. When triggered, NGINX skips the cache and fetches fresh from the ar.io node — which now returns the blocked response — and overwrites the stale cache entry. In the `http` block (with your other maps), add: ```nginx # Cache bypass for content moderation purging. # Only bypass when the header matches your secret — prevents abuse. map $http_x_cache_bypass $purge_allowed { "your-secret-here" 1; default 0; } ``` In each data-serving location (`/raw/`, `/[TX_ID]`, catch-all), add: ```nginx proxy_cache_bypass $purge_allowed; ``` Then force-refresh a blocked transaction: ```bash # Replace "your-secret-here" and domain with your actual values curl -s -o /dev/null -H "X-Cache-Bypass: your-secret-here" \ https://your-domain.example/TX_ID_HERE curl -s -o /dev/null -H "X-Cache-Bypass: your-secret-here" \ https://your-domain.example/raw/TX_ID_HERE ``` The node returns its blocked/404 response, NGINX caches that instead, and subsequent requests get the blocked response. If you run a content scanning sidecar, fire these bypass requests automatically after each block event to complete the purge without any extra infrastructure. ## Monitoring Cache Performance Add the `X-Cache-Status` header to expose cache behavior on every response: ```nginx add_header X-Cache-Status $upstream_cache_status always; ``` Check cache performance: ```bash curl -s -D - -o /dev/null https://your-gateway.example/TX_ID 2>&1 | grep -i x-cache ``` | Status | Meaning | |--------|---------| | `HIT` | Served from cache | | `MISS` | Fetched from origin, now cached | | `EXPIRED` | Cache entry expired, fetched fresh from origin | | `UPDATING` | Stale entry served while background update runs | | `STALE` | Stale entry served due to origin error | | `BYPASS` | Cache was bypassed (e.g., 429 or no-store response) | | (empty) | Response status not covered by any `proxy_cache_valid` directive for this location | ## Validate and Reload Always test your configuration before reloading NGINX: ```bash sudo nginx -t && sudo systemctl reload nginx ``` If `nginx -t` reports errors, fix them before reloading. A bad reload with `systemctl reload` is safe (NGINX keeps the old config running), but `systemctl restart` with a broken config will take NGINX offline. ## Additional Configuration ### Load Balancer Real IP If your gateway is behind a load balancer, configure NGINX to trust the `X-Forwarded-For` header from the load balancer's IP: ```nginx # Replace with your load balancer's actual IP set_real_ip_from 10.0.0.1/32; real_ip_header X-Forwarded-For; real_ip_recursive on; ``` Without this, all requests appear to come from the load balancer's IP. ## Troubleshooting ### Duplicate Cache Entries for /raw/ and / `/raw/TX_ID` and `/TX_ID` serve the same bytes but are cached as separate entries because the cache key includes the full URI path. If disk space is a concern, normalize the key with a map in the `http` block: ```nginx map $uri $normalized_cache_uri { "~^/raw/(.+)$" "/$1"; default $uri; } ``` Then use `proxy_cache_key "$scheme://$host$normalized_cache_uri";` in both the `/raw/` and TX data locations. This causes both endpoints to share one cache entry. ### Cache Directories Not Writable If NGINX logs show write errors, check ownership matches the NGINX worker user (`www-data` on Debian/Ubuntu, `nginx` on RHEL/CentOS): ```bash ls -la /var/lib/nginx/cache/ ``` ### Shared Memory Exhaustion If `keys_zone` is too small for the number of cached items, NGINX silently evicts entries. Monitor with `stub_status` and increase `keys_zone` size if your hit rate drops unexpectedly despite having disk space available. ## Related } /> } /> } /> } /> # Delegate gateway management (/build/run-a-gateway/manage/operations-address) An operations address is a second Solana address that you, the gateway operator, authorize to handle routine gateway tasks. It can update your gateway's metadata and claim the ArNS gateway operator discount. It can't touch your stake, your delegation settings, or who manages the gateway. Use one to keep your operator wallet offline. The operator wallet controls your stake, so it's the key you least want on a server or in a script. ## What an operations address can do The following table shows which wallet can sign each gateway action: | Action | Operator wallet | Operations address | |---|---|---| | Update metadata: label, FQDN, port, protocol, properties, note | Yes | Yes | | Claim the ArNS gateway operator discount | Yes | Yes | | Change stake or delegation settings, or leave the network | Yes | No | | Change the observer address | Yes | No | | Set or revoke the operations address | Yes | No | A gateway has one operations address at a time. Until you set one, the operations address is your operator address, so nothing changes until you choose to delegate. Anyone who holds the operations keypair can change your gateway's metadata, including its FQDN, until you revoke it. Store the operations keypair as carefully as any other key that can change what your gateway advertises. ## Before you begin You need the following: - A joined gateway. A gateway that's leaving the network can't change its operations address. - The ar.io CLI, version 4.4.0 or later: `npm install -g @ar.io/sdk`. - Your gateway's operator keypair file. - The [Solana CLI](https://solana.com/docs/intro/installation), to create a keypair for the operations address. ## Set an operations address ### Create a keypair for the operations address ```bash solana-keygen new --outfile operations-keypair.json solana-keygen pubkey operations-keypair.json ``` The second command prints the operations address. You need it in step 3. ### Fund the operations address Send a small amount of SOL to the operations address. It pays the transaction fee for every change it signs, and 0.01 SOL covers many metadata updates. ### Authorize the operations address Sign with your operator keypair: ```bash ar.io update-operations-address \ --operations-address OPERATIONS_ADDRESS \ --wallet-file OPERATOR_KEYPAIR_FILE ``` Replace the following: - `OPERATIONS_ADDRESS`: the address that step 1 printed. - `OPERATOR_KEYPAIR_FILE`: the path to your operator keypair file. The CLI shows the gateway and the operations address, and asks you to confirm. ### Check the result ```bash ar.io get-gateway --address OPERATOR_ADDRESS ``` Replace `OPERATOR_ADDRESS` with your gateway's operator address. The `operationsAddress` field shows the address you authorized. ## Update gateway metadata with the operations address To update metadata, sign with the operations keypair and name your gateway by its operator address: ```bash ar.io update-gateway-metadata \ --gateway-address OPERATOR_ADDRESS \ --fqdn GATEWAY_FQDN \ --wallet-file OPERATIONS_KEYPAIR_FILE ``` Replace the following: - `OPERATOR_ADDRESS`: your gateway's operator address. The signer isn't the operator, so this tells the network which gateway to update. - `GATEWAY_FQDN`: the domain your gateway serves, such as `turbo-gateway.com`. - `OPERATIONS_KEYPAIR_FILE`: the path to the operations keypair file. The command also accepts `--label`, `--port`, `--protocol`, `--properties`, and `--note`. Include only the fields you want to change. ## Claim the ArNS gateway operator discount When the operations address pays for an ArNS purchase, add `--discount-gateway-address` with your operator address. To check the price first: ```bash ar.io get-cost-details \ --intent Extend-Lease \ --name ARNS_NAME \ --years 1 \ --address OPERATIONS_ADDRESS \ --discount-gateway-address OPERATOR_ADDRESS ``` To make the purchase: ```bash ar.io extend-lease \ --name ARNS_NAME \ --years 1 \ --discount-gateway-address OPERATOR_ADDRESS \ --wallet-file OPERATIONS_KEYPAIR_FILE ``` Replace `ARNS_NAME` with the name you're extending. The same flag works with `buy-record`, `increase-undername-limit`, and `upgrade-record`. The discount applies only when your gateway qualifies. For the rules, see [Gateway operator ArNS discount](/learn/arns/pricing-model#gateway-operator-arns-discount). If you name a gateway that doesn't qualify, the CLI stops and prints the reason instead of charging full price. ## Revoke or replace the operations address To revoke, set the operations address back to your operator address: ```bash ar.io update-operations-address \ --operations-address OPERATOR_ADDRESS \ --wallet-file OPERATOR_KEYPAIR_FILE ``` To replace it, run the command with a different operations address. Either change takes effect with its transaction, and the network refuses the previous operations address from then on. ## Troubleshooting The following table lists the errors the CLI prints for an operations address, and what each one means: | Error contains | Cause | Fix | |---|---|---| | `is neither the operator nor the operations address of gateway` | The signing keypair isn't authorized for that gateway. The operations address was revoked or replaced, `--gateway-address` names another gateway, or `--wallet-file` points at the wrong keypair. | Run `ar.io get-gateway --address OPERATOR_ADDRESS` and compare `operationsAddress` with the signing address. | | `does not qualify for the operator discount` | The gateway fails a discount rule. The rest of the message names the rule. | Check the gateway against the [discount rules](/learn/arns/pricing-model#gateway-operator-arns-discount), or make the purchase without `--discount-gateway-address`. | | `No gateway found for operator` | `--discount-gateway-address` isn't a gateway's operator address. | Use the operator address, not the operations address. | For the SDK methods behind these commands, see [Gateways in the ar.io SDK reference](/sdks/ar-io-sdk/gateways). # Setting Apex Domain Content (/build/run-a-gateway/manage/setting-apex-domain) Configure your ar.io Gateway to serve custom content from the apex domain instead of the default Arweave network information. This allows you to customize your gateway's root domain with useful information, project details, or any content you wish to share. ## Overview Prior to gateway Release 28, the apex domain of a gateway would only display information about the Arweave network. Release 28 introduced two new environment variables that allow a gateway to serve custom content from the apex domain: - `APEX_TX_ID`: Set to serve content from a specific transaction ID - `APEX_ARNS_NAME`: Set to serve content from an ArNS name These variables enable gateway operators to customize their gateway's apex domain with useful information, details about the operator or associated projects, or any other content they wish to share. ## Quick Start ### Choose Your Content Source Decide how you want to serve your content: **Option 1: Direct Transaction ID** - Upload your content to Arweave - Use the transaction ID directly **Option 2: ArNS Name (Recommended)** - Upload your content to Arweave - Assign your content's transaction ID to an ArNS name - Use the ArNS name for easier management ### Upload Your Content Upload your dApp, website, or other content to Arweave using your preferred method: - **ArDrive** - For simple file uploads - **Turbo** - For application bundles - **Direct upload** - For advanced users ### Configure Environment Variable Add one of these variables to your `.env` file: ```bash # Option 1: Direct transaction ID APEX_TX_ID=your-transaction-id # Option 2: ArNS name (recommended) APEX_ARNS_NAME=your-arns-name ``` You cannot set both variables simultaneously. Providing both variables will result in an error. ### Restart Your Gateway Restart your gateway to apply the changes: ```bash docker compose down docker compose up -d ``` ### Verify Configuration Visit your gateway's apex domain to confirm the custom content is being served correctly. ## Configuration Methods ### Using Direct Transaction ID ### Upload Content Upload your content to Arweave and note the transaction ID: ```bash # Example: Upload using ArDrive CLI ardrive upload-file --file-path ./my-website.html # Note the returned transaction ID # Example: abc123...def789 ``` ### Set Environment Variable Add the transaction ID to your `.env` file: ```bash APEX_TX_ID=abc123...def789 ``` ### Restart Gateway Restart your gateway to apply the configuration: ```bash docker compose down docker compose up -d ``` ### Update Content To update your content: 1. Upload new content to Arweave 2. Update `APEX_TX_ID` with the new transaction ID 3. Restart your gateway **Advantages:** - Direct control over content - No additional ArNS setup required - Simple for one-time content **Disadvantages:** - Requires gateway restart for updates - Less flexible for content management ### Using ArNS Name (Recommended) ### Upload Content Upload your content to Arweave: ```bash # Upload your website or dApp ardrive upload-file --file-path ./my-dapp.html # Note the transaction ID: xyz789...abc123 ``` ### Register ArNS Name Register an ArNS name pointing to your content: 1. Visit [ArNS App](https://arns.ar.io) 2. Connect your wallet 3. Choose your desired name (e.g., `my-gateway-content`) 4. Set the transaction ID: `xyz789...abc123` 5. Pay the registration fee ### Configure Environment Variable Add the ArNS name to your `.env` file: ```bash APEX_ARNS_NAME=my-gateway-content ``` ### Restart Gateway Restart your gateway to apply the configuration: ```bash docker compose down docker compose up -d ``` ### Update Content To update your content: 1. Upload new content to Arweave 2. Update the ArNS name to point to the new transaction ID 3. **No gateway restart required!** **Advantages:** - No restart required for content updates - Easy content management - Professional domain naming - Can be updated independently **Disadvantages:** - Requires ArNS setup - Additional cost for ArNS registration ### Advanced Setup Options ### Custom Content Types Configure different types of content: **Static Website:** ```bash # Upload HTML/CSS/JS files APEX_ARNS_NAME=my-gateway-website ``` **Single Page Application:** ```bash # Upload SPA bundle APEX_ARNS_NAME=my-dapp ``` **Documentation Site:** ```bash # Upload documentation APEX_ARNS_NAME=my-gateway-docs ``` ### Content Management Workflow Implement a content management workflow: 1. **Development** - Test content locally 2. **Upload** - Deploy to Arweave 3. **Register** - Create/update ArNS name 4. **Verify** - Check content on gateway 5. **Monitor** - Track performance and usage ## Use Cases and Examples ### Display Gateway Service Information To present your gateway service: **Content Ideas:** - Gateway operator information - Service capabilities and features - Contact information - Status and uptime statistics - Network participation details **Example Structure:** ```html My ar.io Gateway ar.io Gateway Service Reliable gateway infrastructure for the permanent web High availability Fast response times Global CDN Contact: operator@example.com ``` ### Showcase Associated Projects Highlight your projects and services: **Content Ideas:** - Project portfolio - Service offerings - Recent updates and news - Links to other projects - Integration examples **Example Structure:** ```html My Projects - ar.io Gateway My Projects Project Alpha Description of project and its features Visit Project Project Beta Another project description Visit Project ``` ### Host Documentation Provide documentation: **Content Ideas:** - Gateway setup guides - API documentation - Integration tutorials - Troubleshooting guides - FAQ sections **Example Structure:** ```html Gateway Documentation .nav { float: left; width: 200px; } .content { margin-left: 220px; } Navigation Setup Guide API Reference Troubleshooting Gateway Documentation Setup Guide Step-by-step setup instructions... ``` ### Real-World Examples Several gateway operators have implemented this feature: **arweave.tech** - Serves a custom landing page with gateway service information - Professional presentation of capabilities **arlink.xyz** - Serves the permaDapp for the Arlink project - Demonstrates integration with existing projects **frostor.xyz / love4src.com** - Serves information about the Memetic Block Software Guild - Showcases community and project information **vilenarios.com** - Serves personalized portfolio/link tree information - Personal branding and contact information These examples demonstrate the flexibility of the apex domain feature and how different operators use it to create unique, personalized experiences for their users. ## Troubleshooting ### Fix Configuration Problems ### Check Environment Variables Verify your `.env` file configuration: ```bash # Check if variables are set correctly grep -E "APEX_(TX_ID|ARNS_NAME)" .env # Should show only one of: # APEX_TX_ID=your-transaction-id # APEX_ARNS_NAME=your-arns-name ``` Ensure you have only ONE of the APEX variables set, not both. ### Verify Gateway Restart Ensure your gateway has been restarted after configuration changes: ```bash # Check if gateway is running docker compose ps # Restart if needed docker compose down docker compose up -d ``` ### Check Gateway Logs Review logs for any error messages: ```bash docker compose logs ar-io-core | grep -i apex ``` ### Resolve Content Issues ### Verify Content Accessibility Test if your content is accessible: ```bash # Test transaction ID directly curl -I https://turbo-gateway.com/your-transaction-id # Test ArNS name resolution curl -I https://your-arns-name.ar.io ``` ### Check Content Format Ensure your content is properly formatted: - **HTML content** should have proper DOCTYPE - **Text content** should be UTF-8 encoded - **Binary content** should have appropriate Content-Type headers ### Test Content Rendering Verify content renders correctly in different browsers: 1. Test in Chrome, Firefox, Safari 2. Check mobile responsiveness 3. Verify all links work correctly 4. Test with different screen sizes ### Fix ArNS Problems ### Verify ArNS Resolution Check if your ArNS name resolves correctly: ```bash # Test ArNS resolution nslookup your-arns-name.ar.io # Check if it points to the correct transaction curl -s https://your-arns-name.ar.io | head -10 ``` ### Update ArNS Record If ArNS name points to wrong content: 1. Go to [ArNS App](https://arns.ar.io) 2. Find your ArNS name 3. Update the transaction ID 4. Wait for propagation (usually immediate) ### Check ArNS Status Verify ArNS name is active and not expired: 1. Visit the ArNS app 2. Check your name's status 3. Ensure it's not expired 4. Verify payment is up to date ## Best Practices ### Content Design ### Optimize for Performance - Keep file sizes reasonable - Use efficient HTML/CSS - Optimize images and assets - Minimize external dependencies ### Ensure Accessibility - Use semantic HTML - Include alt text for images - Ensure good color contrast - Test with screen readers ### Mobile Responsiveness - Design for mobile-first - Use responsive CSS - Test on various devices - Ensure touch-friendly interfaces ### Content Management ### Version Control - Keep content in version control - Document changes and updates - Test changes before deployment - Maintain backup copies ### Regular Updates - Keep information current - Update contact details - Refresh project information - Monitor for broken links ### Backup Strategy - Backup content regularly - Keep multiple copies - Document restoration procedures - Test backup recovery ## Next Steps **Ready to customize your gateway?** Start with the Quick Start section above, then explore the different configuration methods and use cases to find what works best for your needs. ### Additional Resources - **ArNS Documentation** - Learn more about ArNS names and management - **Content Upload Guides** - Best practices for uploading content to Arweave - **Gateway Configuration** - Advanced gateway configuration options - **Community Examples** - See how other operators use this feature ### Getting Help If you encounter issues: 1. Check the troubleshooting section above 2. Verify your configuration is correct 3. Test content accessibility independently 4. Consult the [ar.io Discord](https://discord.com/invite/HGG52EtTc2) for community support # Solana Migration for Operators (/build/run-a-gateway/manage/solana-migration) ## Overview The ar.io network has migrated protocol execution from AO to Solana. If you're an existing gateway operator, this guide covers everything you need to change. **Your gateway will continue serving Arweave data uninterrupted** — the changes affect how your node interacts with the protocol layer (staking, observations, rewards). Complete these steps before the cutover date to ensure uninterrupted reward eligibility. Gateways that fail 30 consecutive epochs will be [pruned](/learn/oip/pruning) with 100% of minimum stake slashed. ## What Changed | Before (AO) | After (Solana) | |-------------|----------------| | Arweave wallet (RSA JWK) | Solana keypair (Ed25519) | | AO Compute Unit for state reads | Solana RPC for state reads | | `AO_CU_URL`, `AO_MU_URL`, etc. | `SOLANA_RPC_URL` | | Observations submitted via AO messages | Observations submitted as Solana transactions | | Observer wallet pays in AR/Turbo credits | Observer wallet pays in SOL | | Rewards distributed atomically | Rewards distributed via [cranker pipeline](/learn/oip/epoch-pipeline) | | Observer address not unique | Observer address must be unique per gateway | | Minimum stake = 10,000 ARIO | Minimum stake = 20,000 ARIO | ## What Didn't Change - Your gateway still serves Arweave data (caching, indexing, ArNS resolution) - Docker-based deployment workflow is the same - Epoch duration remains 24 hours - Staking, delegation, and reward mechanics are functionally equivalent ## Step-by-Step Migration ### Map Your Address Before the cutover, register your Solana wallet address using the ar.io claim app. This maps your existing Arweave address to your new Solana address so your stake, delegation, and gateway registration carry over automatically. ### Generate a Solana Keypair Create a new Solana keypair for your gateway and observer: ```bash # Install Solana CLI tools sh -c "$(curl -sSfL https://release.anza.xyz/stable/install)" # Generate gateway operator keypair solana-keygen new --outfile operator-keypair.json # Generate observer keypair (must be unique — not shared with any other gateway) solana-keygen new --outfile observer-keypair.json ``` Your observer address must be **unique across all gateways**. If another gateway is already using the same observer address, your registration will fail. This is enforced onchain by the ObserverLookup PDA. ### Fund Your Wallets with SOL Your observer wallet needs SOL for submitting observation transactions. Typical costs are less than 0.01 SOL per transaction. ```bash # Check your observer address solana-keygen pubkey observer-keypair.json # Send SOL to it from an exchange or existing wallet ``` We recommend keeping at least **0.5 SOL** in the observer wallet to cover several months of observation submissions. ### Update Environment Variables Edit your `.env` file with the new Solana configuration: ```bash # Operator + observer addresses (Solana pubkeys, base58) AR_IO_WALLET= OBSERVER_WALLET= # Solana RPC endpoint (see Callout below — use a premium provider in production) SOLANA_RPC_URL=https://your-rpc-provider.com # Keypair file paths inside the container (mounted from ./wallets/) SOLANA_KEYPAIR_PATH=/app/wallets/operator-keypair.json OBSERVER_KEYPAIR_PATH=/app/wallets/observer-keypair.json # The four ar.io Solana programs (canonical mainnet) ARIO_CORE_PROGRAM_ID=73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh ARIO_GAR_PROGRAM_ID=89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf ARIO_ARNS_PROGRAM_ID=2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ ARIO_ANT_PROGRAM_ID=2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 ``` **Remove the old AO variables** (they are no longer used): ```bash # DELETE THESE from your .env: # AO_CU_URL=... # NETWORK_AO_CU_URL=... # ANT_AO_CU_URL=... # AO_MU_URL=... # AO_GATEWAY_URL=... # AO_GRAPHQL_URL=... # HTTPSIG_UPLOAD_ATTESTATION=... # WALLETS_PATH=... ``` The default public Solana RPC (`api.mainnet-beta.solana.com`) is rate-limited. Under any real load your observer and cranker will start dropping transactions, and the resulting timeouts cause most observation cycles to miss their submission window. Use a dedicated RPC provider in production: [Helius](https://helius.dev), [Triton](https://triton.one), or [QuickNode](https://quicknode.com). ` (or `SOLANA_PRIVATE_KEY=`) instead of the `*_KEYPAIR_PATH` env. When using an inline observer key, also set `SOLANA_UPLOAD_PRIVATE_KEY=` so observer report uploads to Turbo use the same signer. Setting both forms for the same role is rejected at startup. ### Place Keypair Files Copy your operator and observer Solana keypair JSON files into the `wallets/` directory in your ar-io-node checkout. Docker Compose mounts this directory into `/app/wallets/` inside the container, which is where the `*_KEYPAIR_PATH` env vars point. ```bash cp operator-keypair.json wallets/operator-keypair.json cp observer-keypair.json wallets/observer-keypair.json chmod 600 wallets/*.json ``` Env vars must use the **in-container** path (`/app/wallets/...`), not the host path (`./wallets/...`). Docker Compose bind-mounts `${WALLETS_PATH:-./wallets}` to `/app/wallets` inside the container. | Wrong (host path) | Right (container path) | |---|---| | `SOLANA_KEYPAIR_PATH=./wallets/operator-keypair.json` | `SOLANA_KEYPAIR_PATH=/app/wallets/operator-keypair.json` | Skip this step entirely if you set `OBSERVER_PRIVATE_KEY` / `SOLANA_PRIVATE_KEY` env vars (base58 strings) instead. ### Update ar-io-node Pull the latest Solana-compatible release and recreate the containers: ```bash cd ar-io-node git pull docker compose pull docker compose up -d --force-recreate core envoy observer ``` Avoid `docker compose down -v` here — the `-v` flag deletes named volumes, which on most setups wipes the sqlite index and chunk data the gateway has spent days/weeks building. Use `docker compose up -d --force-recreate ` instead, which restarts containers in place without touching volumes. Check your release version at `https:///ar-io/info`. ### Optional: Enable Epoch Cranking Your observer can optionally help drive the epoch reward pipeline. This is completely permissionless and costs minimal SOL (~0.000155 SOL per epoch): ```bash # Add to .env ENABLE_EPOCH_CRANKING=true ``` See the [epoch pipeline](/learn/oip/epoch-pipeline) docs for details. Running the cranker creates redundancy so the network is never dependent on a single bot. ### Verify After restarting, verify your gateway is operating correctly: 1. **Release + program IDs (cross-check config is what you intended):** ```bash curl -s https:///ar-io/info | jq '{release, wallet, programIds}' ``` `programIds.core/gar/arns/ant` should match the network you're targeting. If they don't, the gateway is still pointed at the old program set, which is usually why a migrated gateway looks inert. 2. **Gateway registration is live on the new network:** ```bash ar.io get-gateway -t solana \ --rpc-url https://api.mainnet-beta.solana.com \ --core-program-id 73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh \ --gar-program-id 89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf \ --arns-program-id 2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ \ --ant-program-id 2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 \ --address ``` Should return `"status": "joined"` with the FQDN/stake/settings you set at join time. 3. **ArNS resolution works end-to-end:** ```bash curl -I -H "Host: ardrive." https:/// ``` Expect HTTP 200 (or a manifest redirect). 404 here means the cache hasn't hydrated yet — give it a minute and re-check. 4. **Observer reports are flowing:** `https:///ar-io/observer/reports/current` 5. **Gateway listed in the portal:** [gateways.ar.io](https://gateways.ar.io) 6. **No accumulating failed epochs** in the portal's gateway view. ## Wallet Roles and Configuration Patterns The gateway uses up to four distinct wallet roles. Understanding these helps you pick the right configuration for your setup. | Role | What it signs | Env vars | Fallback | |---|---|---|---| | **Operator** (+ cranker) | `join_network`, `update_gateway_settings`, permissionless cranker instructions | `SOLANA_KEYPAIR_PATH` or `SOLANA_PRIVATE_KEY` | — (required) | | **Observer** | `save_observations` transactions | `OBSERVER_KEYPAIR_PATH` or `OBSERVER_PRIVATE_KEY` | Falls back to operator key | | **Upload** | Observer report bundles sent to Turbo | See [upload precedence](#upload-signing-precedence) below | Falls back to observer → operator Solana key, but explicit upload env is recommended | | **HTTPSIG signer** | RFC 9421 response headers | Uses observer Solana key when set | Auto-generated standalone Ed25519 key | Setting both the file-path and inline forms for the same role (e.g. `SOLANA_KEYPAIR_PATH` **and** `SOLANA_PRIVATE_KEY`) is rejected at startup as ambiguous. Pick one. ### Supported Configurations These are the five supported wallet setups. **Pattern 1 is the recommended default** — one key does everything. Pattern 2 is the most common migration path for operators who already have an Arweave JWK. | # | Operator | Observer | Upload | Required envs | |---|---|---|---|---| | **1** | Solana | = operator | = operator (Solana) | `SOLANA_KEYPAIR_PATH` | | **2** | Solana | = operator | Arweave JWK | `SOLANA_KEYPAIR_PATH` + `ARWEAVE_UPLOAD_KEY_FILE` | | **3** | Solana A | Solana B | Solana C | `SOLANA_KEYPAIR_PATH` + `OBSERVER_KEYPAIR_PATH` + `SOLANA_UPLOAD_KEYPAIR_PATH` | | **4** | Solana A | Solana B | Arweave JWK | `SOLANA_KEYPAIR_PATH` + `OBSERVER_KEYPAIR_PATH` + `ARWEAVE_UPLOAD_KEY_FILE` | | **5** | Solana A | Solana B | Ethereum | `SOLANA_KEYPAIR_PATH` + `OBSERVER_KEYPAIR_PATH` + `ETHEREUM_UPLOAD_PRIVATE_KEY_FILE` | #### Pattern 1 — Single Solana keypair (recommended) ```bash # One key for operator + observer + uploads SOLANA_KEYPAIR_PATH=/app/wallets/operator-keypair.json SOLANA_UPLOAD_KEYPAIR_PATH=/app/wallets/operator-keypair.json SOLANA_RPC_URL= AR_IO_WALLET= OBSERVER_WALLET= ENABLE_EPOCH_CRANKING=false # flip to true when ready ``` If you use a base58 private key instead of a keypair file, explicitly set the upload key too: ```bash OBSERVER_PRIVATE_KEY= SOLANA_UPLOAD_PRIVATE_KEY= ``` If you publish indexes with [Index Sharing](/build/run-a-gateway/manage/index-sharing#set-the-signing-key), the `index-swarm` sidecar does not see `SOLANA_KEYPAIR_PATH` or the gateway's wallets directory. Set `INDEX_SWARM_OBSERVER_KEYPAIR_FILE` to the host path of the same keypair file, or use `OBSERVER_PRIVATE_KEY`, not both. #### Pattern 2 — Keep existing Arweave JWK for uploads The most common path for operators migrating from a pre-Solana setup. Your existing Arweave JWK continues signing report bundles while the Solana keypair handles protocol interactions. ```bash SOLANA_KEYPAIR_PATH=/app/wallets/operator-keypair.json ARWEAVE_UPLOAD_KEY_FILE=/app/wallets/.json SOLANA_RPC_URL= AR_IO_WALLET= OBSERVER_WALLET= ENABLE_EPOCH_CRANKING=false ``` ### Upload Signing Precedence The gateway picks the first matching upload signer from this list: ``` 1. ARWEAVE_UPLOAD_KEY_FILE (file) → ArweaveSigner 2. ARWEAVE_UPLOAD_JWK (inline) → ArweaveSigner 3. ETHEREUM_UPLOAD_PRIVATE_KEY_FILE (file) → EthereumSigner 4. ETHEREUM_UPLOAD_PRIVATE_KEY (inline) → EthereumSigner 5. SOLANA_UPLOAD_KEYPAIR_PATH (explicit file) → SolanaSigner 6. SOLANA_UPLOAD_PRIVATE_KEY (explicit inline) → SolanaSigner 7. Fallback: OBSERVER_KEYPAIR_PATH ?? SOLANA_KEYPAIR_PATH → SolanaSigner ``` Setting upload envs from more than one chain at once (e.g. `ARWEAVE_UPLOAD_KEY_FILE` **plus** `ETHEREUM_UPLOAD_PRIVATE_KEY`) raises a startup error listing every conflicting env. Pick exactly one upload chain. If observer logs warn that `TurboReportSink` is not configured, the observer does not have an upload signer for report data. Add `SOLANA_UPLOAD_PRIVATE_KEY=` or `SOLANA_UPLOAD_KEYPAIR_PATH=`, then restart the observer and check the next epoch logs. ### Key Formats Solana keypairs come in two common formats. Both encode the same 64-byte secret (`seed(32) || pubkey(32)`): | Format | Example | Source | |---|---|---| | **JSON array** (Solana CLI standard) | `[12,34,56,...]` — 64 uint8 integers | `solana-keygen new --outfile keypair.json` | | **base58 secret** | 87–88 character base58 string | Phantom " Use the JSON file with `*_KEYPAIR_PATH` env vars, or the base58 string with `*_PRIVATE_KEY` env vars — never both for the same role. ## Troubleshooting ### "Observer is restart-looping with 'Epoch 0 PDA not found'" If the observer logs repeatedly: ``` error: Continuous observer start() rejected — attempting auto-restart ...error: "Epoch 0 PDA not found at — has prescribe_epoch run yet?" ``` …the network you're targeting hasn't had its first epoch initialized. The observer needs entropy from `epoch[N].prescribed_observers` to bootstrap, which doesn't exist until someone calls `create_epoch` (typically a cranker). Verify the epoch state directly: ```bash ar.io get-current-epoch -t solana \ --rpc-url https://api.mainnet-beta.solana.com \ --core-program-id 73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh \ --gar-program-id 89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf \ --arns-program-id 2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ \ --ant-program-id 2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 ``` For staging-devnet or local devnet, replace the RPC URL and program IDs with that environment's values. If this returns `"Epoch 0 not found"`, the network is configured but inactive. This is a network-operations state, not a gateway misconfiguration. Wait for an active cranker (yours or another operator's) to bootstrap epoch 0. ### "Cranker started but never submits any instructions" The cranker silently bails out if `EpochSettings.enabled` is `false`. This is intentional: operators shouldn't pay SOL fees attempting cranker instructions against a paused network. Confirm with `ar.io get-epoch-settings` and check whether epochs are enabled on your target network. If `enabled` is false, no cranker activity is expected. ### "ArNS names return 404 but the on-chain record exists" The gateway hydrates an ArNS names cache at boot, paginating through the on-chain registry. If your SDK pin is significantly older than the deployed `ario-arns` program, the paginated response shape may not match and the cache hydrates incompletely. Symptoms: - Cache hydration logs `Successfully hydrated ArNS names cache` quickly (e.g. ~4s for thousands of records) - Resolver logs report `Base name not found in ArNS names cache` for names that demonstrably exist via the SDK CLI's `get-arns-record --name ` Fix: bump `@ar.io/sdk` in your gateway's image (`package.json`) to the latest `^4.0.0-solana.*` and rebuild. ### Wallet Configuration Startup Errors The gateway validates wallet configuration at startup. Errors are loud and name the offending env: | Error pattern | Cause | Fix | |---|---|---| | `multiple chain groups configured for upload role` | Upload envs from more than one chain are set (e.g. `ARWEAVE_UPLOAD_*` and `ETHEREUM_UPLOAD_*`) | Pick one upload chain and remove the others | | `ambiguous: both ... set for role` | File-path and inline forms for the same role are both set (e.g. `SOLANA_KEYPAIR_PATH` + `SOLANA_PRIVATE_KEY`) | Use one form per role | | `material at SOLANA_KEYPAIR_PATH does not look like a Solana keypair` | An Arweave JWK or other JSON was placed at the Solana keypair path | Check you copied the right file — Solana keypairs are a JSON array of 64 integers, not a JWK object | | `material at ARWEAVE_UPLOAD_KEY_FILE does not look like an Arweave JWK` | A Solana keypair (JSON array) was placed at the Arweave upload slot | Swap the file for your Arweave JWK | | `SOLANA_KEYPAIR_PATH not set in Solana mode` | The operator key is missing entirely | Set `SOLANA_KEYPAIR_PATH` or `SOLANA_PRIVATE_KEY` | | `OBSERVER_KEYPAIR_PATH does not match on-chain Gateway.observer_address` | The observer key doesn't match what was registered at `join_network` | Update the key to match, or call `update_observer_address` on-chain | ## New Risks to Be Aware Of ### Gateway Pruning Gateways that fail **30 consecutive epochs** are automatically pruned from the network. When pruned: - 100% of minimum stake (20,000 ARIO) is slashed to the protocol balance - Excess operator stake enters the standard 30-day withdrawal queue - Delegated stakes enter the standard 30-day withdrawal queue (delegators are not slashed) See [Gateway Pruning](/learn/oip/pruning) for full details. ### SOL Balance Monitoring Keep your observer wallet funded with SOL. If it runs out, your observer cannot submit observations, which leads to failed epochs and eventually pruning. Set up monitoring/alerts for your observer wallet balance. ## FAQ **Do I need to re-register my gateway?** No. If you mapped your address before cutover, your gateway registration, stake, and delegations are migrated automatically. **What happens to my delegators?** Delegations are migrated as-is. Delegators who mapped their addresses will see their stakes in their Solana wallet. Unmapped delegations are held in escrow for claiming. **Can I still use my Arweave wallet for data uploads?** Yes. Arweave wallets are still used for uploading data to Arweave via Turbo. The Solana wallet is only for protocol interactions (staking, observations, ArNS). **How do I check my observer's SOL balance?** ```bash solana balance --url mainnet-beta ``` # Automating SSL Certificate Renewal (/build/run-a-gateway/manage/ssl-certs) Secure your ar.io Gateway with automated SSL certificate renewal using Certbot and DNS challenge validation. This guide covers setup for different DNS providers to automatically renew certificates without manual intervention. ## Overview Using DNS challenge validation with Certbot allows you to: - Automatically renew SSL certificates - Support wildcard certificates - Avoid manual certificate management - Ensure continuous gateway security ## Prerequisites - A running ar.io Gateway - Domain name configured with your DNS provider - Administrative access to your server - API access to your DNS provider ## DNS Provider Setup ### Cloudflare Configuration ### Create Cloudflare API Token Navigate to **Cloudflare → My Profile → API Tokens → Create Token** Configure the token with these permissions: - **Zone → Zone → Read** - **Zone → DNS → Edit** ![Cloudflare API Token Configuration](https://turbo-gateway.com/GMzqNXNCQMSLqyt7SV7FrGOgCuGBeaO5qjRWibFkVBE) ### Install Certbot and Cloudflare Plugin ```bash apt update apt install certbot python3-certbot-dns-cloudflare -y ``` ### Configure API Credentials Create the credentials file: ```bash nano /etc/letsencrypt/cloudflare.ini ``` Add your API token: ```ini dns_cloudflare_api_token = your_api_token_here ``` Secure the file: ```bash chmod 600 /etc/letsencrypt/cloudflare.ini ``` ### Generate SSL Certificate Request the certificate with wildcard support: ```bash certbot certonly --dns-cloudflare \ --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \ -d example.com -d *.example.com ``` **Expected output:** ```bash Successfully received certificate. Certificate is saved at: /etc/letsencrypt/live/example.com/fullchain.pem Key is saved at: /etc/letsencrypt/live/example.com/privkey.pem ``` ### Test Automatic Renewal Perform a dry run to validate the renewal process: ```bash certbot renew --dry-run ``` **Expected output:** ```bash Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/example.com.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Account registered. Simulating renewal of an existing certificate for example.com and *.example.com Waiting 10 seconds for DNS changes to propagate - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Congratulations, all simulated renewals succeeded: /etc/letsencrypt/live/example.com/fullchain.pem (success) ``` ### Verify Automatic Renewal Timer Check that the certbot timer is active: ```bash systemctl list-timers | grep certbot ``` **Expected output:** ```bash Tue 2024-11-05 02:22:10 UTC 3h 21min Mon 2024-11-04 17:16:51 UTC 5h 43min ago certbot.timer certbot.service ``` ### Namecheap Configuration **API Requirements:** Namecheap requires specific conditions to create API keys: - At least 20 domains under your account - Minimum $50 account balance - At least $50 spent within the last 2 years If you don't meet these requirements, contact Namecheap support for a waiver. ### Create Namecheap API Key Navigate to **Namecheap → Profile → Tools → Manage API Access Keys** Create your API credentials and note: - Your username - Your API key ### Install Certbot and Dependencies ```bash apt update apt install certbot python3-pip -y ``` Install the Namecheap DNS plugin: ```bash pip install certbot-dns-namecheap ``` ### Configure API Credentials Create the credentials file: ```bash nano /etc/letsencrypt/namecheap.ini ``` Add your API credentials: ```ini dns_namecheap_username = your_username dns_namecheap_api_key = your_api_key ``` Secure the file: ```bash chmod 600 /etc/letsencrypt/namecheap.ini ``` ### Generate SSL Certificate Request the certificate with wildcard support: ```bash certbot certonly --dns-namecheap \ --dns-namecheap-credentials /etc/letsencrypt/namecheap.ini \ -d example.com -d *.example.com ``` **Expected output:** ```bash Successfully received certificate. Certificate is saved at: /etc/letsencrypt/live/example.com/fullchain.pem Key is saved at: /etc/letsencrypt/live/example.com/privkey.pem ``` ### Test Automatic Renewal Perform a dry run to validate the renewal process: ```bash certbot renew --dry-run ``` **Expected output:** ```bash Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/example.com.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Account registered. Simulating renewal of an existing certificate for example.com and *.example.com Waiting 10 seconds for DNS changes to propagate - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Congratulations, all simulated renewals succeeded: /etc/letsencrypt/live/example.com/fullchain.pem (success) ``` ### Verify Automatic Renewal Timer Check that the certbot timer is active: ```bash systemctl list-timers | grep certbot ``` **Expected output:** ```bash Tue 2024-11-05 02:22:10 UTC 3h 21min Mon 2024-11-04 17:16:51 UTC 5h 43min ago certbot.timer certbot.service ``` ## Post-Installation Steps After successfully setting up automatic SSL renewal: ### Update Gateway Configuration Configure your ar.io Gateway to use the new certificates. Update your gateway's SSL configuration to point to: - **Certificate:** `/etc/letsencrypt/live/your-domain.com/fullchain.pem` - **Private Key:** `/etc/letsencrypt/live/your-domain.com/privkey.pem` ### Reload Web Server (Optional) If you're using nginx or another web server, reload it to apply the new certificates: ```bash systemctl reload nginx ``` ### Monitor Renewal Process Certbot automatically sets up a systemd timer for renewal. Certificates will be renewed when they have 30 days or less remaining. To manually check renewal status: ```bash certbot certificates ``` ## Troubleshooting ### Common Issues - **DNS propagation delays:** Wait 5-10 minutes for DNS changes to propagate - **API rate limits:** Check your DNS provider's API rate limits - **Permission errors:** Ensure credential files have correct permissions (600) ### Logs and Debugging Check certbot logs for detailed error information: ```bash tail -f /var/log/letsencrypt/letsencrypt.log ``` ## Next Steps With SSL certificates automated, consider: - [Setting up monitoring](/build/extensions/grafana) to track certificate expiration - [Configuring gateway filters](/build/run-a-gateway/manage/filters) for optimal performance - [Implementing content moderation](/build/run-a-gateway/manage/content-moderation) policies # Troubleshooting (/build/run-a-gateway/manage/troubleshooting) This page collects common gateway problems and their fixes. Start with the quick lookup table, then read the matching section below. ## Quick Lookup Below is a quick summary of what you should check when troubleshooting your gateway. Find more detailed information in the sections below. | Issue | What to Check | | ------------------------------------------------------------ | ------------------------------------------------------------------------- | | My release number is wrong | Pull the latest github updates and make sure you are on the `main` branch | | Gateway appears offline on Viewblock or https://gateways.ar.io | Probably fine, but verify that your gateway is still running. | | '/ar-io/observer/reports/current' just says "report pending" | Normal behavior, wait for the report to complete. | | Observer error "Cannot read properties of undefined" | Normal behavior, Observer is checking for data not implemented yet. | | Observing my gateway shows failures | Check `AR_IO_WALLET` and `ARNS_ROOT_HOST` settings. | | Updated .env settings not reflected on gateway | Rebuild your gateway after editing .env file. | | Out of disk space error | Check for inode exhaustion and delete files if necessary. | | Can't load ArNS names | Check `ARNS_ROOT_HOST` setting in .env file, and DNS records. | | "Your connection is not private" error | Generate or renew SSL certificates. | | 404/Nginx error when accessing domain | Check Nginx settings and restart Nginx if necessary. | | 502 error from Nginx | Check for errors in your gateway. | | Trouble generating SSL certificates | Ensure TXT records have propagated and follow certbot instructions. | ## General Troubleshooting ### My Gateway Seems to be Running but... If your release number when you go to `/ar-io/info` is lower than the current release, you simply need to upgrade your gateway in order to reach the latest release. If your release number includes the suffix "-pre" it means you are running your gateway from the development branch of the github repository, instead of the main branch. The development branch is used for staging work that the engineering team is in the middle of. Because of this, it can be much less stable than the main branch used for production and can cause significant issues. Ensure that you are running the latest release, from the main branch, by running the below commands in your terminal: ```console sudo docker compose down --rmi all git checkout main git pull sudo docker compose up -d ``` If this doesn't resolve the issue, you can also try a more extreme method of clearing out the incorrect docker images: ```console sudo docker compose down sudo docker system prune -a sudo docker compose up -d ``` Viewblock and https://gateways.ar.io use a very simple ping method for determining if a gateway is "up". This ping can fail while the gateway works normally, so showing as down is not cause for concern. Just verify that your gateway is still running, and wait. Your gateway will show as up again soon. /ar-io/observer/reports/current just says 'report pending'"> This is normal. Your Observer is working to generate a report and that report will be displayed once it is complete. This is not an issue with your observer. The short explanation is that your Observer is looking for tasks assigned to it by the ar.io network contract, but there isnt anything there. You can safely ignore this error message. When observing a gateway, there are two main pass/fail tests. "Ownership" and "ArNS Assessment" - Ownership: This tests to see if the value set in your gateway `AR_IO_WALLET` value (in .env) matches the wallet used to join ar.io. If they don't match, update the value in your .env file and restart your gateway. - ArNS Assessment: This tests to see if a gateway is able to resolve ArNS names correctly. The first thing you should check is if you have the `ARNS_ROOT_HOST` value set in your .env file. If not, set the value and restart your gateway. If this value is set, check to make sure you have current DNS records and SSL certificates for wildcard subdomains on your gateway. **Release Requirement**: From gateway Release 85, which is not released yet, observers also run offset (chunk-proof) checks. Each epoch, every observer picks the same share of gateways (20% by default). A sampled gateway must serve `/chunk/` with a Merkle proof that validates against the chain. It fails only if all of its sampled offsets (4 by default) fail, and the report lists each failure in `offsetAssessments` with a `failureCategory`. Once you edit your .env file, you need to "rebuild" your gateway for the changes to take effect. As of release 3, every time you start your gateway with `docker compose` it is automatically rebuilt. So all you need to do is shut your gateway down and restart it. The most likely cause of this is inode exhaustion. Test this by running the command: ``` df -i ``` If one of the lines in the output says 100%, you have run out of inodes and so your filesystem is not capable of creating new files, even if you have available space. The solution is to delete files from your `data` folder in order to free up inodes. This was a common issue prior to release #3, when Redis caching was introduced to reduce the number of small files created. If you are using an older version of the gateway, consider upgrading to mitigate the risk of inode exhaustion. The first thing you should check if your gateway is not resolving ArNS names is that you have `ARNS_ROOT_HOST` set in your .env file. If not, set it to your domain name used for the gateway. For example, `ARNS_ROOT_HOST=turbo-gateway.com`. Once this value is set, restart your gateway for the changes to take effect. If that doesn't resolve the issue, check your dns records. You need to have a wildcard subdomain ( \*. ) set with your domain registrar so that ArNS names will actually point at your gateway. You can set this record, and generate an SSL certificate for it, in the same way you set the records for your primary domain. This error message means that your SSL certificates have expired. Expired certificates are also the most common reason a working gateway suddenly stops. Renew them by running the same certbot command you used when you set up your gateway: ``` sudo certbot certonly --manual --preferred-challenges dns -d .com -d '*..com' ``` Certbot certificates expire after 90 days, so you need to rerun this command every time, unless you set up [automatic renewal](/build/run-a-gateway/manage/ssl-certs). After renewing, restart NGINX (`sudo service nginx restart`): it can keep serving the old certificates until it restarts. A 404 error from Nginx (the reverse proxy in the setup guide) means your domain points at the machine running your gateway. Either your Nginx settings are not configured correctly, or your gateway is not running. The "Set Up Networking" step of the [production setup](/build/run-a-gateway/quick-start#production-setup-with-custom-domain) has detailed instructions on configuring your Nginx server. If all else fails, try restarting Nginx, that usually clears any issues with the server clinging to old configurations. ``` sudo service nginx restart ``` A 502 error from Nginx means that Nginx is working correctly, but it is receiving an error from your gateway when it tries to forward traffic. When using the manual certbot command from the setup guide, you need to be sure that you are waiting after creating your TXT records for them to completely propagate. You can check propagation using a tool like [dnschecker.org](https://dnschecker.org). If you continue to have issues, you can check the [official certbot instructions guide](https://certbot.eff.org/instructions). - ar.io gateways recover from most temporary errors without affecting normal operation. Be concerned only if the error repeats, or your gateway stops working properly. - Observers generate and submit their reports at specific times throughout the epoch. This is to ensure a healthy network throughout the entire epoch, not just at the start. - Your observer wallet must match the observer wallet associated with your gateway in the ar.io contract. You can check this by navigating to your gateway in https://gateways.ar.io. - This happens when a request to a CU fails, and your gateway receives an html failure message instead of the expected JSON response. This will normally clear up on its own after congestion on that CU dies down, but if it is persistent try switching to a different CU. - This is normal. It means you have reached the current Arweave block and need to wait for more before you can index them. - This is normal. If a gateway fails to resolve an arns name within 3 seconds, it will fall back to a trusted gateway (turbo-gateway.com by default) to help resolve the name. - There are many reasons a gateway could fail an epoch. Following these steps is usually enough to identify and correct the issue: - Try to visit your gateway in a browser and see if your SSL certs are bad - Try to resolve an ArNS name on your gateway. If it fails to resolve, check the console and your gateway logs for errors - Look at the observation reports that failed your gateway, they will list the reason for failure ## Troubleshooting Failed Epochs ### Overview The ARIO Network provides several tools to help troubleshoot problems with a gateway. The most useful is the [Observer](/learn/oip). The Observer, which is a component of every gateway joined to the ARIO Network, checks all gateways in the network to ensure that they are functioning properly, and returning the correct data. The Observer then creates a report of the results of these checks, including the reasons why a gateway might have failed the checks. If a gateway fails the checks from more than half of the prescribed observers, the gateway is marked as failed for the epoch, and does not receive any rewards for that epoch. First, try to resolve data on the failed gateway in a browser. If that does not make the issue clear, use the Observer report to diagnose the problem. ### Manual Observation Manual observations may be run on a gateway at any time buy using the [Network Portal](https://gateways.ar.io). This allows operators (or anyone with an interest in the gateway's performance) to check the gateway's performance at any time. To run a manual observation: 1. Navigate to the [Network Portal](https://gateways.ar.io) 2. Select the gateway you are interested in from the list of gateways 3. Click on the "Observe" button in the top right corner of the page. ![Diagram](https://turbo-gateway.com/0G52dTWe65abQ6qDGvI99ERAaGU7DHR9srimJXnYRGA) 4. Click on the "Run Observation" button in the bottom right corner of the page. ![Diagram](https://turbo-gateway.com/A_B_58rufQ0Pj4ri0AKuC0DJn61u5ayO5ONWpkMerQw) Two randomly selected ArNS names will be entered automatically in the "ArNS names" field to the left of the "Run Observation" button. These can be changed, or additional ArNS names can be added to the list before running the observation. The Manual observation will run the same checks as the observer, and will display the results on the right side of the page. ![Diagram](https://turbo-gateway.com/vgRXfbx4fa47qGDpjndq128VCHl1wajKaq464KeA0Qg) ### Accessing the Observer Report The simplest way to access an observer report is via the [Network Portal](https://gateways.ar.io), following the steps below: 1. Navigate to the [Network Portal](https://gateways.ar.io) 2. Select the gateway you are interested in from the list of gateways 3. In the Observation window, select the epoch you are interested in. This will display a list of the observers that failed the gateway for that epoch. 4. Click on the "View Report" button to the right any observer on that list. This will display the entire report that observer generated. ![Diagram](https://turbo-gateway.com/ynbxYU_8xLRaU1D6a_LMoUq00roWwsMKgr-xrsDE0Sk) 5. Locate the gateway you are interested in in the report, and click on that row. This will display the report for that gateway. ### Understanding the Observer Report The observer report will display a list of checked ArNS names, and a reason if the gateway failed to return the correct data for that name. There are several reasons why a gateway might fail to return the correct data for an ArNS name. Below is a list of the most common reasons, and how to resolve them. #### Timeout awaiting 'socket', or Timeout awaiting 'connect' ![Diagram](https://turbo-gateway.com/_GupbMa-EW_wiCD201MuOkDQrLT0MXTfxDXhLSDmyh4) ![Diagram](https://turbo-gateway.com/0WkzxdyN-9hJfv0pSiTs0Ozg_wqFvE3-OWlgzYPimtU) This failure means that the observer was unable to connect to the gateway when it tried to check the ArNS name. There are lots of reasons why this might happen, many of them unrelated to the gateway itself. If an observer report has a small number of these failures, among a larger number of successful checks, it is unlikely to be an issue with the gateway. If this failure occurs persistently for a large number, or all ArNS names checked, it likely means that the observer is having trouble connecting to the gateway at all. You can verify this by: - Attempting to connect to the gateway in a browser - Running manual observations on the gateway using the [Network Portal](https://gateways.ar.io) - Using tools like `curl` or `ping` to check the gateway's connectivity If these methods consistently fail to connect to the gateway, it is likely that the gateway is not properly configured or powered on. If this is the case: - Check Docker and the gateway's logs to see if the gateway is on. - Ensure that the SSL certificates are valid for the gateway's domain. - Check DNS records for the gateway's domain, misconfigured or conflicting DNS records can cause connectivity issues. Some gateway operators who run their gateways on their personal home networks have also reported issues with their ISP blocking, throttling, or otherwise delaying traffic to a gateway. If none of the above steps resolve the issue, it may be worth checking with your ISP to see if they are blocking or throttling traffic to the gateway. Using [Grafana](/build/extensions/grafana) can also provide a visual representation of the gateway's ArNS resolution times. If this is consistently high (above 10 seconds), it is likely that the gateway is not properly configured to resolve ArNS names. Ensure that the gateway is operating on the latest Release. #### Cert has expired This failure means that the gateway's SSL certificate has expired. Obtaining a new SSL certificate and updating the gateway's reverse proxy (nginx, etc) configuration to use the new certificate is the only solution to this issue. #### dataHashDigest mismatch ![Diagram](https://turbo-gateway.com/xXe0bHne--0JJv-HRf5HT9R1V1UbzaOh2AxvAdQZhjg) This failure means that the gateway did respond to a resolution request, but the data it returned did not match the data that was expected. This could be due to a number of reasons, including: - Cached data was returned by the gateway that doesnt match the most current data on the network. - The gateway is configured to operate on testnet or devnet. Gateways joined to the ARIO Network MUST operate on mainnet in order to pass observation checks. - The gateway is intentionally returning fraudulent data. A gateway will not return fraudulent data unless that operator intentionally rewrote the gateway's code to do so, and a major purpose of the Observation and Incentive Protocol is to catch and prevent this behavior. A gateway may return mistaken data on occasion, usually due to a cache mismatch between the gateway and the observer's authority (usually arweavae.net). This is a relatively rare occurrence, and should only be considered an issue if it occurs persistently. If most or all of the ArNS names checked are failing for this reason, it is likely that the gateway is not operating on mainnet. #### Response code 502 (Bad Gateway) ![Diagram](https://turbo-gateway.com/NBQsYUKP6IZt_rYg77QXgzwUUPvimFGXCQqtesbW1_I) This failure means that the observer was able to connect to the gateway's network, but the reverse proxy returned a 502 error. This is almost always a reverse proxy issue. Ensure that the gateway's reverse proxy is running, and that it is configured to forward requests to the gateway. Test the reverse proxy's configuration file (`sudo nginx -t` on Nginx) for more information. If the file has no problems, reloading the proxy (`sudo nginx -s reload`) often resolves the issue. It is also possible that the gateway itself is not running at all. Check Docker and the gateway's logs to see if the gateway is on. #### Response code 503 (Service Unavailable) ![Diagram](https://turbo-gateway.com/7eFKSm-cs81-aJ_H4xkolR2nSlxl5tYWXJdFTei8Dbs) This failure means that the observer was able to connect to the gateway's network, but the reverse proxy was unable to forward the request to the gateway. It differs from the 502 error in that the reverse proxy is likely able to see that the gateway is running, but is unable to communicate with it. This is often a temporary issue, caused by the gateway not being able to handle a heavy load of requests, or the gateway being in the process of restarting. If this failure occurs once or twice in a report, it is likely a temporary issue and should not be considered an issue with the gateway. However, when this failure occurs persistently, particularly for every ArNS name checked on the report, it is likely that the gateway may have crashed. Manually restarting the gateway can likely resolve the issue. #### connect EHOSTUNREACH ![Diagram](https://turbo-gateway.com/O-uG-yGm5bNxjw2ADH_yBjOcGo-ZEiFym8GeFZZNueY) This failure means that the observer was unable to connect to the gateway at all. The connection was either refused, or the gateway was not able to find a target based on the domain name's DNS records. This is almost always an issue with DNS records or local network configuration. Ensure that the gateway domain has correct DNS records, and that the local network is set up to allow connections. Checking logs from the local network's reverse proxy (nginx, etc) may provide more information about the issue. #### getaddrinfo ENOTFOUND ![Diagram](https://turbo-gateway.com/WJDwW0NM29uKC-9puvhXK_n75vgFXLpa6VKFVMhRsLQ) This is another DNS related issue. Likely, the gateway does not have a valid DNS record either for the top level domain or the required wildcard subdomain. Having this failure occur once or twice in a report could mean that the DNS server being used by the observer is having temporary issues and should not be considered an issue with the gateway. However, when this failure occurs persistently, particularly for every ArNS name checked on the report, it is likely that the gateway's DNS records are not set, or are misconfigured. #### Hostname/IP does not match certificate's altnames: Host: \. is not in the cert's altnames: DNS:\ ![Diagram](https://turbo-gateway.com/HfOfpAYm811dWFPNQC7bANEvjGVK4ch3kO7K7qMN9qs) This failure means that the observer's SSL certificate does not match the gateway's domain name. This is almost always an issue with the gateway's SSL certificate. This most likely occurred because the gateway's operator did not update the gateway's SSL certificate when the gateway's domain name was changed. Obtaining a new SSL certificate and updating the gateway's reverse proxy configuration to use the new certificate is the only solution to this issue. #### write EPROTO \:error:\:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:\:SSL alert number 112 ![Diagram](https://turbo-gateway.com/Hbip_ZmqmFN8-uXijw1aylyYp1YllwgyZTNAcsPCxSg) This failure almost always means that the gateway operator did not properly obtain SSL certificates for the gateway's wildcard subdomain. Obtaining a new SSL certificate and updating the gateway's reverse proxy configuration to use the new certificate is the only solution to this issue. ## FAQ - Gateway protocol rewards are calculated as 0.1% of the protocol balance (0.05% after August 2025) split between all gateways in the network. A change in the protocol balance or the number of gateways in the network between epochs will result in the reward for an individual gateway changing. - The Observer rewards are separate from protocol rewards, and if your gateway is selected as an observer for an epoch, assuming it performs its duties well, it will receive additional rewards The observer selection process uses a weighted random selection method that considers multiple factors beyond just stake: - **Stake Weight (SW)**: Ratio of your total staked ARIO tokens (including delegated stake) to the network minimum - **Tenure Weight (TW)**: How long your gateway has been part of the network (capped at 4 after 2 years) - **Gateway Performance Ratio Weight (GPRW)**: Ratio of epochs where you correctly resolved names vs total participation - **Observer Performance Ratio Weight (OPRW)**: Ratio of epochs where you successfully submitted reports vs total observer periods A composite weight (CW) is calculated as: CW = SW × TW × GPRW × OPRW Up to 50 gateways are chosen as observers per epoch. If there are more than 50 gateways, selection is randomized based on these normalized weights. Even with a high stake, other factors like performance and tenure affect your chances of being selected. - There is a 30 day locking period when withdrawing delegated stake or excess operator stake from your gateway. This locking period can be skipped, for a fee. The fee starts at 50% of the withdrawal amount, and goes down over time. If you chose to withdraw without waiting, you paid the fee to skip the locking period. The required network-join stake follows the separate network leave process. - The minimum operator stake for gateways (20,000 ARIO) cannot be withdrawn while the gateway is active. It can only be returned through the network leave process, and it is unrecoverable if the gateway is pruned for sustained poor performance. - If possible, leave your original server running while you prepare the new one - Set up the new server following the same steps you used to set up the original server - This includes setting up SSL certificates for the new server - You must use the same gateway wallet when setting up the new server - The observer wallet may be changed at any point, but requires extra steps. It is recommended you use the original observer wallet as well - Once the new server is set up, change your DNS A records to point at the new server - After your DNS records are set and you have verified your gateway is operating correctly, shut down the original server - No changes need to be made in the network contract or on https://gateways.ar.io - Yes - Configure your new domain to point at your gateway, including setting up SSL certificates - Update your NGINX (or other reverse proxy) server to recognize the new domain. This usually requires a restart of NGINX - Update the `ARNS_ROOT_HOST` variable in your `.env` and restart the gateway - Using https://gateways.ar.io, update your gateway settings to change the FQDN in the contract - Your gateway is now using the new domain name for normal operation. ## Getting Help If you encounter any issues during the troubleshooting process, please seek assistance from the [ar.io community](https://discord.com/invite/HGG52EtTc2). **Ready to get back to building?** Once your gateway is running smoothly, check out [Manage your Gateway](/build/run-a-gateway/manage) for guides on optimization, monitoring, and more. # Upgrading your Gateway (/build/run-a-gateway/manage/upgrading-a-gateway) Upgrade your ar.io gateway to each new release to get its fixes and security updates. Indexed data lives outside Docker, so neither upgrading the gateway nor pruning Docker erases your data or progress. ## Prerequisites - Your Gateway should have been cloned using git. If you haven't, follow the [installation instructions](/build/run-a-gateway/quick-start). ## Checking your Release Number Effective with release 3, you can view the currently implemented release on any gateway by visiting `https:///ar-io/info` in a browser. Be sure to replace `` with the domain of the gateway you are checking. If the release number displayed includes `-pre` it means that your gateway is using the `develop` branch of the github repo for the gateway code. Follow steps in our [troubleshooting guide](/build/run-a-gateway/manage/troubleshooting) to switch over to the more stable `main` branch. Announcements will be made in our [discord server](https://discord.com/invite/HGG52EtTc2) showing each new release. ## Upgrade Process ### Full Upgrade Process ### Pull Latest Changes Navigate to your cloned repository directory and execute: ```bash git pull ``` ### Shut Down Docker Stop your gateway: ```bash sudo docker compose down -v ``` ```bash docker compose down -v ``` ### Check for New Environment Variables Read the update release change logs and community announcements to see if the new version includes any new environmental variables that you should set before restarting your gateway. ### Restart the Gateway Start your gateway with the new version: ```bash sudo docker compose up -d ``` ```bash docker compose up -d ``` You don't need the `--build` flag: Docker uses the image named in `docker-compose.yaml`. ### Docker Pruning (Optional) It's a good practice to clean up unused Docker resources after shutting down your gateway. This will erase all inactive docker containers on your machine. If you use docker for anything beyond running a gateway be extremely careful using this command. ### Shut Down Gateway First, stop your gateway: ```bash sudo docker compose down -v ``` ```bash docker compose down -v ``` ### Prune Docker System Clean up unused Docker resources: ```bash sudo docker system prune ``` ```bash docker system prune ``` ### Restart Gateway Start your gateway: ```bash sudo docker compose up -d ``` ```bash docker compose up -d ``` ### Checking for New Environment Variables New gateway releases may introduce new environment variables that you need to configure. ### Review Release Notes Check the release notes and community announcements for any new environment variables: - Review the [GitHub releases](https://github.com/ar-io/ar-io-node/releases) - Check the [ar.io Discord](https://discord.com/invite/HGG52EtTc2) for announcements - Look for changes in the `.env.example` file ### Update Your .env File Add any new environment variables to your `.env` file: ```bash # Example: Add new environment variables NEW_FEATURE_ENABLED=true NEW_CONFIG_VALUE=default_value ``` ### Restart Gateway Restart your gateway to apply the new environment variables: ```bash sudo docker compose up -d ``` ```bash docker compose up -d ``` That's it! Your ar.io Gateway is now upgraded to the latest version. Ensure to test and verify that everything is functioning as expected. If you encounter any issues, reach out to the [ar.io community](https://discord.com/invite/HGG52EtTc2) for assistance. # Verification & Trust Headers (/build/run-a-gateway/manage/verification-headers) ar.io gateways attach HTTP response headers to every data response. They say whether the data has been verified against the Arweave base layer, whether it has reached finality, and whether the source is trusted. Clients can use them to decide how far to trust a response without relying only on the operator's reputation. ## Trust & Verification Headers Every data response from an ar.io gateway includes the following headers: | Header | Values | Description | |--------|--------|-------------| | `X-AR-IO-Verified` | `true` / `false` | `true` only when data is served from local cache **and** has been verified against the Arweave base layer. Data streamed from network sources is marked `false` even if the gateway's database indicates prior verification, because the hash cannot be confirmed in-flight during streaming. | | `X-AR-IO-Stable` | `true` / `false` | `true` when the data exists beyond Arweave's maximum fork depth, indicating finality. Once stable, the data cannot be reorganized out of the chain. | | `X-AR-IO-Trusted` | `true` / `false` | `true` when the data was retrieved from a source the operator has configured as trusted (via `TRUSTED_GATEWAY_URL` or `TRUSTED_GATEWAYS_URLS`). | | `Content-Digest` | `sha-256=:base64:` | SHA-256 hash of the response body in [RFC 9530](https://www.rfc-editor.org/rfc/rfc9530.html) format. Set on cache hits and HEAD requests. An uncached response of known size up to 2 MiB is buffered and hashed so it carries one too. Larger uncached responses stream without it. | | `X-AR-IO-Digest` | base64url string | The raw SHA-256 hash of the content in base64url encoding. Set alongside `Content-Digest`. | | `ETag` | `"hash"` | Content hash for HTTP conditional requests (`If-None-Match`), so a client can get a `304 Not Modified`. Treat it as opaque, and read the digest from `X-AR-IO-Digest`. See the note below for responses served through an ArNS name. | | `X-AR-IO-Data-Id` | base64url string | The Arweave transaction ID or data item ID of the content being served. | | `X-AR-IO-Hops` | number | The number of inter-gateway hops the request traversed before reaching this gateway. | | `X-Cache` | `HIT` / `MISS` | Whether the data was served from the gateway's local cache or fetched from the network. | **Release Requirement**: From gateway Release 85, which is not released yet, a response served through an ArNS name carries the `ETag` `"."`. A `304` then means the same bytes and the same resolution, so a cache does not keep an old resolution after the name is re-pointed. `/raw`, `/` and the index routes keep the plain digest. ### Why X-AR-IO-Verified Can Be False for Verified Data A common source of confusion: data may be verified in the gateway's database but still return `X-AR-IO-Verified: false`. This happens when the data is streamed from a network source rather than served from local cache. During streaming, the gateway cannot compute a hash of the data in-flight to confirm it matches the verified hash. Only when the data is served from local cache can the gateway guarantee that the bytes being sent match what was previously verified. ``` # Cache hit - verified data served from local storage X-AR-IO-Verified: true X-Cache: HIT Content-Digest: sha-256=:4ROTs2lTPAKbr8Y41WrjHu+2q+7S+m+yTuO7fAUzZI4=: # Network fetch of a large file - same data, but the hash can't be confirmed in-flight X-AR-IO-Verified: false X-Cache: MISS # Content-Digest is NOT set (the body is streamed, not hashed) ``` ## ANS-104 Bundle Headers For data items served from [ANS-104 bundles](https://specs.ar.io/?tx=fFJEsPOLDJKRfNgaeaRPlfSvcNtHp0Y3JVxD-HHzmwA), gateways include additional headers in two categories. ### Position Headers These headers describe the data item's location within its parent bundle, allowing clients to independently verify inclusion: | Header | Description | |--------|-------------| | `X-AR-IO-Root-Transaction-Id` | The root Arweave transaction containing the bundle | | `X-AR-IO-Data-Item-Offset` | Byte offset of this data item within the bundle | | `X-AR-IO-Data-Item-Size` | Size of the data item in bytes | | `X-AR-IO-Data-Item-Data-Offset` | Byte offset of the data payload within the data item | | `X-AR-IO-Root-Item-Offset` | Offset of the root item in the bundle | | `X-AR-IO-Root-Item-Size` | Size of the root item | ### Metadata Headers ANS-104 fields are exposed at the HTTP layer so clients can read data item tags and owner information directly from response headers without parsing the binary format: | Header | Description | |--------|-------------| | `X-Arweave-Owner` | The raw owner public key of the data item | | `X-Arweave-Owner-Address` | The data item signer's wallet address (derived from the owner key) | | `X-Arweave-Tag-*` | One header per tag, with the tag name as the suffix (e.g., `X-Arweave-Tag-Content-Type`) | | `X-Arweave-Tags-Truncated` | Set when the tag set exceeds a configurable byte budget, indicating partial tag exposure | ## Client-Side Verification Clients can verify gateway responses at multiple levels, from lightweight header checks to full cryptographic verification: ### Body Integrity Compare the `Content-Digest` header (when present) against a locally computed SHA-256 hash of the response body. For Arweave content, the `X-AR-IO-Data-Id` is itself a content hash - clients can walk the Arweave or ANS-104 signature chain from this ID to verify the content. ```javascript // Verify Content-Digest in a browser const response = await fetch('https://gateway.example/raw/TX_ID'); const body = await response.arrayBuffer(); const hash = await crypto.subtle.digest('SHA-256', body); const base64Hash = btoa(String.fromCharCode(...new Uint8Array(hash))); const digestHeader = response.headers.get('Content-Digest'); // Header format: sha-256=:base64hash=: // Extract the hash between the colons const match = digestHeader?.match(/sha-256=:(.+?):/); const verified = match?.[1] === base64Hash; console.log('Integrity check:', verified); ``` ### Cross-Gateway Comparison Query the same data from multiple gateways and compare their response headers. If `X-AR-IO-Verified: true` and `Content-Digest` values match across independent gateways, the data is authentic with high confidence. ### Wayfinder Verification The [Wayfinder](/learn/wayfinder) protocol provides automated client-side routing and verification across the gateway network. It supports configurable strategies including balanced (random), fastest ping, and static gateway selection, with built-in response verification. ## HTTP Message Signatures (RFC 9421) Gateways sign response headers with [RFC 9421 HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html). This is on by default. An Ed25519 key signs the trust-relevant headers of each response, producing a `Signature` and `Signature-Input` header pair. The signature proves which gateway operator produced a response. ### Turning HTTP Signatures Off Signing is on unless you turn it off in your gateway's `.env` file: ```bash HTTPSIG_ENABLED=false ``` #### Signing Key Selection When `OBSERVER_KEYPAIR_PATH` or `OBSERVER_PRIVATE_KEY` is set, the gateway uses the observer's Solana keypair directly as the HTTPSIG signing key. Verifiers derive the Solana address from the public key in the `keyId` and look it up in the on-chain Gateway Registry — no separate attestation is needed. When neither observer env is set, the gateway auto-generates a standalone Ed25519 key at `data/keys/httpsig.pem`. Responses are still signed, but the signer can't be tied back to the on-chain registry. If you use a single Solana key for both operator and observer (Pattern 1 in the [migration guide](/build/run-a-gateway/manage/solana-migration#supported-configurations)), HTTPSIG falls back to the auto-generated key because the gateway only piggybacks on `OBSERVER_KEYPAIR_PATH`/`OBSERVER_PRIVATE_KEY` when they're **explicitly set**. To get on-chain-verifiable HTTPSIG with a single key, explicitly set `OBSERVER_KEYPAIR_PATH` (or `OBSERVER_PRIVATE_KEY`) to the same value as your operator key. ### Configuration Reference | Variable | Default | Description | |----------|---------|-------------| | `HTTPSIG_ENABLED` | `true` | RFC 9421 response signing | | `HTTPSIG_KEY_FILE` | `data/keys/httpsig.pem` | Path to standalone Ed25519 private key (auto-generated if missing). Ignored when `OBSERVER_KEYPAIR_PATH` or `OBSERVER_PRIVATE_KEY` is set | | `HTTPSIG_BIND_REQUEST` | `true` | Include request method and path in signature (prevents replay) | | `OBSERVER_KEYPAIR_PATH` | - | Path to a 64-byte Solana keypair JSON. When set, used as the HTTPSIG signing key — verifiable against the on-chain GAR | | `OBSERVER_PRIVATE_KEY` | - | Alternative: base58-encoded 64-byte secret. Mutually exclusive with the file form | ### Verifying It's Working Check the `/ar-io/info` endpoint to confirm: ```bash curl -s https://your-gateway.example/ar-io/info | jq '.httpsig' ``` The response includes the public key, key ID, Solana address, and attestation details (if configured). Signed responses will include `Signature` and `Signature-Input` headers on all trust-relevant data responses. ### What Gets Signed The gateway signs all trust-relevant response headers but **not the response body**. Body integrity is achieved through the signed `X-AR-IO-Data-Id` (which is a content hash) and the `Content-Digest` header. Header-only signing preserves streaming performance for large responses. **Signed headers** (when present on the response): - Trust headers: `X-AR-IO-Data-Id`, `X-AR-IO-Verified`, `X-AR-IO-Stable`, `X-AR-IO-Trusted` - Bundle headers: `X-AR-IO-Root-Transaction-Id`, `X-Arweave-Owner-Address`, `X-Arweave-Tags-Truncated` - ArNS headers: `X-ArNS-Name`, `X-ArNS-Resolved-Id`, `X-ArNS-TTL-Seconds`, `X-ArNS-ANT-Program-Id`, `X-ArNS-ANT-Id`, and with them `X-ArNS-Basename`, `X-ArNS-Record`, `X-ArNS-Resolved-At`, `X-ArNS-Undername-Limit`, `X-ArNS-Record-Index` - Chunk headers: `X-Arweave-Chunk-Data-Root`, `X-Arweave-Chunk-Tx-Id`, `X-AR-IO-Chunk-Source-Type` - Index Sharing headers: `X-AR-IO-Index-Publication` on the publication at `/ar-io/indexes`, and `X-AR-IO-Index-File` on band files - Content headers: `Content-Type`, `Content-Digest`, `Repr-Digest` - Root offset headers: `X-AR-IO-Root-Item-Offset`, `X-AR-IO-Root-Item-Size`, `X-AR-IO-Root-Path` and the older `X-AR-IO-Root-Data-Item-Offset`, `X-AR-IO-Root-Data-Offset` - All `X-Arweave-Tag-*` headers (dynamically matched) - Request components (when `HTTPSIG_BIND_REQUEST=true`): `@method`, `@path`, `@status` Only responses containing at least one trust-relevant header are signed. Non-data responses (health checks, info endpoint, errors) are not signed. A range request for a band file answers `206` with `Repr-Digest`, the digest of the whole file, instead of a `Content-Digest` of the bytes sent. The signature covers it, so a signed range is bound to the file it came from. **Release Requirement**: From gateway Release 85, which is not released yet, `GET /ar-io/offsets/:id` answers are signed too. They carry `X-AR-IO-Root-Transaction-Id` and a `Content-Digest` that binds the body. ### Verification Chain Clients can verify signed responses through the following chain: 1. **Signature verification** — The public key is embedded in the `Signature-Input` header's `keyid` parameter and is verifiable via the Web Crypto API in modern browsers. 2. **Identity verification** — When the observer Solana key is used for signing, derive the Solana address from the public key and look it up in the on-chain Gateway Registry (GAR). A match confirms the signer is a registered gateway operator. 3. **Body integrity** — Compare the `Content-Digest` header against a locally computed hash, or walk the Arweave signature chain from the signed `X-AR-IO-Data-Id`. ## Related } /> } /> } /> } /> # x402 Payment Setup (/build/run-a-gateway/manage/x402-setup) Learn how to set up x402 payment protocol on your ar.io Gateway to monetize data egress. For what x402 is and how gateways use it, see our [x402 Payments guide](/learn/gateways/x402-payments). **Release Requirement**: x402 payments first became available with gateway Release 54. Ensure your gateway is running Release 54 or later to use this feature. ## Prerequisites - Running ar.io Gateway - Administrative access to your server - USDC wallet for receiving payments - API access to x402 payment facilitator **Important**: x402 payments require the rate limiter to be enabled. There is no "payments only" configuration - both systems must work together. ## Quick Start ### Enable Rate Limiter Configure basic rate limiting in your `.env` file: ```bash # Enable rate limiter (required for x402) ENABLE_RATE_LIMITER=true RATE_LIMITER_TYPE=redis RATE_LIMITER_IP_TOKENS_PER_BUCKET=100000 RATE_LIMITER_IP_REFILL_PER_SEC=20 RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET=1000000 RATE_LIMITER_RESOURCE_REFILL_PER_SEC=100 ``` ### Configure x402 Payments Add x402 configuration to your `.env` file: ```bash # Enable x402 payments ENABLE_X_402_USDC_DATA_EGRESS=true X_402_USDC_NETWORK=base-sepolia # or base for production X_402_USDC_WALLET_ADDRESS=0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb X_402_USDC_FACILITATOR_URL=https://x402.org/facilitator X_402_USDC_PER_BYTE_PRICE=0.0000000001 X_402_USDC_DATA_EGRESS_MIN_PRICE=0.001 X_402_USDC_DATA_EGRESS_MAX_PRICE=1.00 X_402_RATE_LIMIT_CAPACITY_MULTIPLIER=10 ``` ### Restart Gateway Apply the configuration changes: ```bash docker compose down docker compose up -d ``` ### Test Configuration Verify x402 is working by checking gateway logs: ```bash docker compose logs -f core | grep -i "x402\|payment" ``` ## Detailed Configuration ### Testnet Configuration For development and testing with free testnet USDC: ### Configure Rate Limiter Set up memory-based rate limiting for single-node deployment: ```bash # Rate limiter configuration ENABLE_RATE_LIMITER=true RATE_LIMITER_TYPE=memory RATE_LIMITER_IP_TOKENS_PER_BUCKET=100000 RATE_LIMITER_IP_REFILL_PER_SEC=20 RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET=1000000 RATE_LIMITER_RESOURCE_REFILL_PER_SEC=100 ``` ### Configure x402 Testnet Set up x402 with Base Sepolia testnet: ```bash # x402 testnet configuration ENABLE_X_402_USDC_DATA_EGRESS=true X_402_USDC_NETWORK=base-sepolia X_402_USDC_WALLET_ADDRESS=0x742d35Cc6634C0532925a3b844Bc9e7595f0bEb X_402_USDC_FACILITATOR_URL=https://x402.org/facilitator X_402_USDC_PER_BYTE_PRICE=0.0000000001 X_402_USDC_DATA_EGRESS_MIN_PRICE=0.001 X_402_USDC_DATA_EGRESS_MAX_PRICE=1.00 X_402_RATE_LIMIT_CAPACITY_MULTIPLIER=10 ``` ### Get Testnet USDC Obtain free testnet USDC from a faucet: 1. Visit [Base Sepolia Faucet](https://www.coinbase.com/faucets/base-ethereum-sepolia-faucet) 2. Connect your wallet 3. Request testnet USDC 4. Use the same wallet address in your configuration ### Test Payment Flow Test the payment system: ```bash # Make multiple requests to trigger rate limiting for i in {1..200}; do curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3000/TX_ID done # Check for 402 Payment Required responses curl -v http://localhost:3000/TX_ID ``` ### Mainnet Configuration Production setup with real USDC payments: ### Configure Redis Rate Limiter Set up distributed rate limiting for multi-node deployment: ```bash # Redis-based rate limiter ENABLE_RATE_LIMITER=true RATE_LIMITER_TYPE=redis RATE_LIMITER_REDIS_ENDPOINT=redis://redis:6379 RATE_LIMITER_IP_TOKENS_PER_BUCKET=100000 RATE_LIMITER_IP_REFILL_PER_SEC=20 RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET=1000000 RATE_LIMITER_RESOURCE_REFILL_PER_SEC=100 ``` ### Configure x402 Mainnet Set up x402 with Base mainnet: ```bash # x402 mainnet configuration ENABLE_X_402_USDC_DATA_EGRESS=true X_402_USDC_NETWORK=base X_402_USDC_WALLET_ADDRESS=0xYOUR_MAINNET_WALLET X_402_USDC_FACILITATOR_URL=https://facilitator.x402.rs X_402_USDC_PER_BYTE_PRICE=0.0000000001 X_402_USDC_DATA_EGRESS_MIN_PRICE=0.001 X_402_USDC_DATA_EGRESS_MAX_PRICE=1.00 X_402_RATE_LIMIT_CAPACITY_MULTIPLIER=10 ``` ### Set Up CDP Integration (Onramp) Configure Coinbase Developer Platform for browser paywall with easy USDC purchases: **What are CDP keys for?** These keys enable Coinbase Onramp integration in the browser paywall, allowing users to easily purchase USDC without leaving your gateway. They are **optional for testnet** but **required for mainnet** when using Coinbase facilitators. **Security Note**: - `X_402_CDP_CLIENT_KEY` is **PUBLIC** (safe for client-side use) - `CDP_API_KEY_ID` and `CDP_API_KEY_SECRET_FILE` are **SENSITIVE SECRETS** - CDP keys are for Onramp integration, NOT facilitator authentication ```bash # CDP configuration for Coinbase Onramp (browser paywall) CDP_API_KEY_ID=YOUR_API_KEY_ID CDP_API_KEY_SECRET_FILE=/app/secrets/cdp_secret_key X_402_CDP_CLIENT_KEY=YOUR_PUBLIC_CLIENT_KEY ``` Create the secret file: ```bash mkdir -p ./secrets chmod 700 ./secrets echo "YOUR_CDP_SECRET_KEY" > ./secrets/cdp_secret_key chmod 600 ./secrets/cdp_secret_key ``` **Note**: If both `CDP_API_KEY_SECRET_FILE` and `CDP_API_KEY_SECRET` are set, the file version takes precedence. ### Configure Redis Persistence Ensure paid tokens persist across restarts: ```bash # Redis persistence for paid tokens EXTRA_REDIS_FLAGS=--save 300 10 --appendonly yes --appendfsync everysec ``` ### Advanced Setup Options Customize your x402 implementation: ### Configure Paywall Customization Customize the payment interface: ```bash # Paywall customization X_402_APP_NAME=My ar.io Gateway X_402_APP_LOGO=https://example.com/logo.png ``` ### Set Up Chunk Pricing Configure fixed-size pricing for chunk requests: ```bash # Chunk pricing configuration CHUNK_GET_BASE64_SIZE_BYTES=368640 ``` ### Configure Allowlists Exempt specific IPs or ArNS names from rate limiting: ```bash # IP/CIDR allowlist (comma-separated) RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST=192.168.1.100,10.0.0.50 # ArNS name allowlist (comma-separated) RATE_LIMITER_ARNS_ALLOWLIST=trusted-app,premium-service ``` An allowlisted client skips both rate limits and payment. Only the client's own address counts, never other addresses in its request headers. ### Tell the Gateway Which Proxies to Trust Rate limits, payment credit and the IP allowlist all use the client's address. A client can write anything in `X-Forwarded-For`, so the gateway believes that header (and `X-Real-IP`) only from proxies in `TRUSTED_PROXIES`. Behind one, the client is the nearest `X-Forwarded-For` address that is not itself a trusted proxy, reading from the right: the address the proxy recorded, never one the client wrote. The default trusts loopback, private, carrier-grade NAT and link-local addresses. That covers Envoy on the Docker network and NGINX or an internal load balancer on the same host or private network, so most gateways need to change nothing. **Release Requirement**: `TRUSTED_PROXIES` arrives with Release 84. Upgrade the Envoy image together with core: from that release Envoy appends the address that connected to it, so a client reaching port 3000 directly cannot choose its address. Change it when something in front is on a public address: ```bash # Behind a CDN (Cloudflare, say) or an internet-facing load balancer: add its # published ranges, keeping the defaults if NGINX sits in between TRUSTED_PROXIES=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10,169.254.0.0/16,::1/128,fc00::/7,fe80::/10, # Only when clients reach core directly, with no Envoy or NGINX in front. # Every request through Envoy (the standard path, port 3000) would otherwise # carry Envoy's address and share one bucket TRUSTED_PROXIES=none ``` Without the CDN's ranges, the CDN's own address is taken as every visitor's, and they all share one rate-limit bucket. A proxy on another public host needs its address added the same way. A malformed entry stops the gateway at startup. ### Set Up Monitoring Configure monitoring and metrics: ```bash # Enable detailed logging LOG_LEVEL=debug # Monitor payment metrics # Access metrics at: http://localhost:3000/ar-io/__gateway_metrics ``` ## Environment Variables Reference ### Required Variables | Variable | Description | Example | | ------------------------------- | ---------------------------------------- | ------------------------------ | | `ENABLE_RATE_LIMITER` | Enable rate limiting (required for x402) | `true` | | `ENABLE_X_402_USDC_DATA_EGRESS` | Enable x402 payments | `true` | | `X_402_USDC_NETWORK` | Base network (base-sepolia or base) | `base-sepolia` | | `X_402_USDC_WALLET_ADDRESS` | Your USDC wallet address | `0x742d35Cc...` | | `X_402_USDC_FACILITATOR_URL` | Payment facilitator URL | `https://x402.org/facilitator` | ### Pricing Configuration | Variable | Description | Default | | -------------------------------------- | ---------------------- | -------------- | | `X_402_USDC_PER_BYTE_PRICE` | Price per byte in USDC | `0.0000000001` | | `X_402_USDC_DATA_EGRESS_MIN_PRICE` | Minimum payment amount | `0.001` | | `X_402_USDC_DATA_EGRESS_MAX_PRICE` | Maximum payment amount | `1.00` | | `X_402_RATE_LIMIT_CAPACITY_MULTIPLIER` | Paid token multiplier | `10` | ### Rate Limiter Configuration | Variable | Description | Default | | ----------------------------------------- | ----------------------------------- | --------- | | `RATE_LIMITER_TYPE` | Rate limiter type (memory or redis) | `redis` (docker-compose), `memory` (standalone) | | `RATE_LIMITER_IP_TOKENS_PER_BUCKET` | IP bucket size | `100000` | | `RATE_LIMITER_IP_REFILL_PER_SEC` | IP refill rate | `20` | | `RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET` | Resource bucket size | `1000000` | | `RATE_LIMITER_RESOURCE_REFILL_PER_SEC` | Resource refill rate | `100` | ## Testing Your Configuration ### Verify Rate Limiting Test that rate limiting is working: ```bash # Make requests to trigger rate limits for i in {1..150}; do curl -s -o /dev/null -w "%{http_code}\n" http://localhost:3000/TX_ID done # Should see 402 responses after rate limit is exceeded ``` ### Test Payment Response Check the 402 payment response: ```bash # Get detailed payment information curl -v http://localhost:3000/TX_ID # Look for X-Payment header in response ``` ### Monitor Gateway Logs Watch for payment-related activity: ```bash # Monitor x402 and payment logs docker compose logs -f core | grep -i "x402\|payment\|rate limit" ``` ### Check Metrics View payment and rate limiting metrics: ```bash # Access gateway metrics curl -s http://localhost:3000/ar-io/__gateway_metrics | grep rate_limit # Check token consumption curl -s http://localhost:3000/ar-io/__gateway_metrics | grep tokens_consumed ``` ## Troubleshooting ### Common Issues **Rate Limiter Not Working**: Ensure `ENABLE_RATE_LIMITER=true` is set and gateway has been restarted. **No 402 Responses**: Check that rate limits are being exceeded and x402 is properly configured. **Payment Verification Fails**: Verify wallet address and facilitator URL are correct. ### Debug Steps 1. **Check Configuration**: ```bash # Verify environment variables docker compose exec core env | grep -E "RATE_LIMITER|X_402" ``` 2. **Monitor Logs**: ```bash # Watch for errors docker compose logs -f core | grep -i error ``` 3. **Test Rate Limits**: ```bash # Check if rate limiting is active curl -s http://localhost:3000/ar-io/__gateway_metrics | grep rate_limit ``` 4. **Verify Network Connectivity**: ```bash # Test facilitator connectivity curl -I https://x402.org/facilitator/supported ``` ## Next Steps Once x402 payments are configured: } > Use Grafana to track payment metrics, rate limiting statistics, and gateway performance analytics. } > Adjust rate limits and pricing based on usage patterns, and configure gateway filters for optimal performance. } > Consider Redis for distributed deployments and review environment variables for advanced configuration options. } > Configure paywall branding, messaging, and customize your gateway's apex domain for a professional appearance. # Installation & Setup (/build/run-a-gateway/quick-start) New to ar.io gateways? Learn more about what they are and how they work at [ar.io Gateways](/learn/gateways). Get your ar.io gateway running in **30 seconds** with Docker. No configuration needed - just run and test. ## Quickstart ```bash # Start ar.io gateway with Docker docker run -p 4000:4000 ghcr.io/ar-io/ar-io-core:latest ``` **Test it's working:** ```bash # Fetch a test transaction curl localhost:4000/4jBV3ofWh41KhuTs2pFvj-KBZWUkbrbCYlJH0vLA6LM # Expected output: test ``` That's it! Your gateway is now serving Arweave data at `localhost:4000`. ## Production Setup with Custom Domain To run a gateway with your own domain name and SSL certificates, follow these steps. ### System Requirements **Minimum requirements:** - 4 core CPU - 4 GB RAM - 500 GB storage (SSD recommended) - Stable 50 Mbps internet connection **Recommended:** - 12 core CPU - 32 GB RAM - 2 TB SSD storage - Stable 1 Gbps internet connection External storage devices should be formatted as ext4. ### Install Required Packages **Quick install all packages:** ```bash sudo apt update -y && sudo apt upgrade -y && sudo apt install -y curl openssh-server git certbot nginx sqlite3 build-essential && sudo systemctl enable ssh && curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash && source ~/.bashrc && sudo ufw allow 22 80 443 && sudo ufw enable ``` **Install Docker:** ```bash # Add Docker's official GPG key sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update # Install Docker sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin ``` **Install Node.js and Yarn:** ```bash nvm install 20.19.6 && nvm use 20.19.6 && npm install -g yarn@1.22.22 ``` ### Install the Node **Clone the repository:** ```bash git clone -b main https://github.com/ar-io/ar-io-node cd ar-io-node ``` **Note:** Your indexing databases will be created in the project directory unless otherwise specified in your .env file, not your Docker environment. So, if you are using an external hard drive, you should install the node directly to that external drive. **Create environment file:** ```bash nano .env ``` **Add configuration:** ```bash GRAPHQL_HOST=turbo-gateway.com GRAPHQL_PORT=443 START_HEIGHT=1000000 RUN_OBSERVER=true ARNS_ROOT_HOST= AR_IO_WALLET= OBSERVER_WALLET= OBSERVER_PRIVATE_KEY= SOLANA_UPLOAD_PRIVATE_KEY= ``` `AR_IO_WALLET` and `OBSERVER_WALLET` must be Solana public keys (base58 Ed25519). The observer address must be unique — no other gateway can use the same observer. **Both wallets need SOL** — the observer pays for `save_observations` instructions, and the operator pays for cranker instructions when `ENABLE_EPOCH_CRANKING=true` (`tally_weights`, `prescribe_epoch`, `distribute_epoch`, `close_epoch`). Budget ~0.5 SOL each to cover several months of activity. **Supply Observer Wallet Keyfile:** Save your Solana keypair JSON file as `.json` in the `wallets` directory. If you use a keypair file instead of inline base58 keys, remove `OBSERVER_PRIVATE_KEY` and `SOLANA_UPLOAD_PRIVATE_KEY` from `.env`, then set both paths explicitly: ```bash OBSERVER_KEYPAIR_PATH=/app/wallets/.json SOLANA_UPLOAD_KEYPAIR_PATH=/app/wallets/.json ``` By default, the Observer will use [Turbo Credits](/build/upload/turbo-credits) to pay for uploading reports to Arweave. This allows reports under 100kb to be uploaded for free, but larger reports will fail if the Observer wallet does not contain Credits. Set `SOLANA_UPLOAD_PRIVATE_KEY` to the same base58 secret as `OBSERVER_PRIVATE_KEY` so Turbo has an explicit upload signer. If logs warn that `TurboReportSink` is not configured, this upload key is usually missing. Including `REPORT_DATA_SINK=arweave` in your `.env` file will configure the Observer to use AR tokens instead of Turbo Credits, without any free limit. **Start the Docker container:** ```bash sudo docker compose up -d ``` ### Set Up Networking **Register a Domain Name:** Choose a domain registrar (e.g., [Namecheap](https://namecheap.com)) to register a domain name. **Point Domain at Your Home Network:** - Get your public IP address: `curl ifconfig.me` - Create A records for your domain and wildcard subdomains (`*.yourdomain.com`) **Set up Port Forwarding:** - Get local IP: `ip addr show | grep -w inet | awk '{print $2}' | awk -F'/' '{print $1}'` - Configure router to forward ports 80 and 443 to your local machine **Create SSL Certificates:** ```bash sudo certbot certonly --manual --preferred-challenges dns -d .com -d '*..com' ``` Previous versions of these instructions advised providing an email address to Certbot. As of June 2025, LetsEncrypt (the certificate authority used by Certbot) no longer supports email notifications. **Important:** Wild card subdomain (*.<your-domain>.com) cannot auto renew without obtaining an API key from your domain registrar. Not all registrars offer this. Certbot certificates expire every 90 days. Be sure to consult with your chosen registrar to see if they offer an API for this purpose, or run the above command again to renew your certificates. ### Configure nginx **Open nginx configuration:** ```bash sudo nano /etc/nginx/sites-available/default ``` **Replace with this configuration:** ```nginx # Force redirects from HTTP to HTTPS server { listen 80; listen [::]:80; server_name .com *..com; location / { return 301 https://$host$request_uri; } } # Forward traffic to your node and provide SSL certificates server { listen 443 ssl; listen [::]:443 ssl; server_name .com *..com; ssl_certificate /etc/letsencrypt/live/.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/.com/privkey.pem; location / { proxy_pass http://localhost:3000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; # Forward ar.io headers if present in the request proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; } } ``` **Test and restart nginx:** ```bash sudo nginx -t sudo service nginx restart ``` **Note:** Previous versions of these instructions advised checking a gateway's ability to fetch content using `localhost`. Subsequent security updates prevent this without first unsetting `ARNS_ROOT_HOST` in your `.env`. ### Test Your Gateway **Verify it's working:** ```bash curl https:///3lyxgbgEvqNSvJrTX2J7CfRychUD5KClFhhVLyTPNCQ ``` **Expected output:** ``` 1984 ``` If you see `1984`, your gateway is working. ### Install Required Software **Install Docker Desktop:** - Download from [Docker Desktop for Windows](https://www.docker.com/products/docker-desktop/) - Run installer and follow prompts - Select WSL (Windows Subsystem for Linux) during installation - Restart your PC - Update WSL: ```cmd wsl --update wsl --shutdown ``` - Restart Docker Desktop **Install Git:** - Download from [Git for Windows](https://git-scm.com/download/win) - Run installer with default settings ### Clone the Repository **Open Command Prompt:** - Press `Windows Key + R` - Type `cmd` and press `Enter` **Navigate to desired directory:** ```cmd cd Documents ``` **Clone the repository:** ```cmd git clone -b main https://github.com/ar-io/ar-io-node ``` **Note:** Your database will be created in the project directory, not Docker. If using an external hard drive, install directly to that drive. ### Create Environment File **Open a text editor (e.g., Notepad):** - Press `Windows Key` and search for "Notepad" **Create .env file with this content:** ```bash GRAPHQL_HOST=turbo-gateway.com GRAPHQL_PORT=443 START_HEIGHT=0 RUN_OBSERVER=true ARNS_ROOT_HOST= AR_IO_WALLET= OBSERVER_WALLET= OBSERVER_PRIVATE_KEY= SOLANA_UPLOAD_PRIVATE_KEY= ``` **Save as `.env`** (select "All Files" as file type) **Supply Observer Wallet Keyfile:** Save your wallet keyfile as `.json` in the `wallets` directory. If you use a keypair file instead of inline base58 keys, remove `OBSERVER_PRIVATE_KEY` and `SOLANA_UPLOAD_PRIVATE_KEY` from `.env`, then set both paths explicitly: ```bash OBSERVER_KEYPAIR_PATH=/app/wallets/.json SOLANA_UPLOAD_KEYPAIR_PATH=/app/wallets/.json ``` ### Start Docker Containers **Navigate to project directory:** ```cmd cd Documents\ar-io-node ``` **Start the container:** ```cmd docker compose up -d ``` **Explanation of flags:** - `up`: Start the Docker containers - `-d`: Run containers as background processes (detached mode) **Shutdown command:** ```cmd docker compose down ``` ### Set Up Router Port Forwarding **Obtain a Domain Name:** Choose a domain registrar (e.g., [Namecheap](https://namecheap.com)) and purchase a domain name. **Point Domain at Your Home Network:** - Visit https://www.whatsmyip.org/ to get your public IP address - Access your domain registrar's settings - Create A records for your domain and wildcard subdomains (`*.yourdomain.com`) **Get Local IP Address:** ```cmd ipconfig ``` Look for IPv4 Address (format: `192.168.X.X` or `10.X.X.X`) **Set Up Router Port Forwarding:** - Access router settings (usually `192.168.0.1`) - Navigate to port forwarding settings - Forward ports 80 and 443 to your local machine's IP address ### Install and Configure NGINX Docker **Clone NGINX Docker repository:** ```cmd cd Documents git clone -b main https://github.com/bobinstein/dockerized-nginx ``` **Follow the repository instructions** for setting up NGINX Docker. **Important:** When configuring your nginx setup, ensure that your nginx configuration includes the following ar.io headers in the proxy configuration: ```nginx # Forward ar.io headers if present in the request proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; ``` These headers are essential for proper ar.io network functionality. ### Test Your Gateway **Verify it's working:** Visit `https:///3lyxgbgEvqNSvJrTX2J7CfRychUD5KClFhhVLyTPNCQ` in your browser. **Expected output:** ``` 1984 ``` If you see `1984`, your gateway is working. ## Useful Docker Commands Monitor and manage your ar.io gateway with these commands: ```bash # View all running services docker ps # Run services in background daemon docker compose up -d # Turn off all services docker compose down # Pull latest images docker compose pull # Follow the logs of the core service docker compose logs core -f -n 10 # Follow the logs of the observer docker compose logs core observer -f -n 10 ``` ## What's Next? Your gateway is running! Now you can: } /> } /> } /> } /> **Need more context?** Learn [What is an ar.io Gateway](/learn/gateways) to understand the full capabilities. # Admin UI (/build/run-wayfinder-router/admin-ui) The admin UI is a built-in web dashboard that runs on a **separate port** (default 3001) from the public router (default 3000). This ensures admin endpoints are never exposed to public traffic. ``` http://localhost:3001 ``` ## Dashboard Pages ### Status Live dashboard showing: - **Uptime** - How long the router has been running - **Operating Mode** - Current mode (proxy/route) - **Verification Status** - Whether verification is enabled and working - **Gateway Health Bar** - Visual indicator of healthy vs unhealthy gateways - **Cache Utilization** - Content cache size and hit rate - **Ping Service Stats** - Background ping status and results ### Gateways Sortable table of all ar.io network gateways with: - **Health** - Current gateway status (healthy/unhealthy/unknown) - **Temperature Score** - Performance score for routing decisions - **Latency** - Average response time - **Success Rate** - Percentage of successful requests - **Traffic Stats** - Requests served and bytes transferred ### Telemetry Time-ranged metrics with - **Time Ranges** - 1 hour, 6 hours, 24 hours, 7 days - **Request Totals** - Total requests, success rate, error breakdown - **Bytes Served** - Total data transferred - **Per-Gateway Performance** - Table with CSV Content moderation management: - **Block/Unblock** - Add or remove ArNS names and transaction IDs - **View Blocklist** - See all currently blocked content - **Enable Moderation** - If not yet configured, shows setup instructions ### Settings View current configuration grouped by category: - Server, Mode, Routing, Verification - Cache, Telemetry, Rate Limiting - HTTP, Shutdown, Admin ## Setup Wizard On first run (when `BASE_DOMAIN=localhost`), the admin UI shows a guided setup wizard that walks you through domain, routing, and verification configuration. See [Quick Start](/build/run-wayfinder-router/quick-start#setup-wizard) for details. ## Security Model | | Public Port (3000) | Admin Port (3001) | |---|---|---| | **Default bind** | `0.0.0.0` (all interfaces) | `127.0.0.1` (localhost only) | | **Admin UI** | Not available (404) | Full access | | **Content serving** | Normal operation | N/A | Key security features: - Admin is **never** exposed on the public port - Default localhost binding means only local access - `ADMIN_PORT` must differ from `PORT` (validated at startup) If you set `ADMIN_HOST=0.0.0.0` to expose the admin UI over the network, you **must** set `ADMIN_TOKEN` to protect it. ## Remote Access To access the admin UI from a remote machine while keeping it secure: ### Option 1: SSH Tunnel (Recommended) ```bash # On your local machine ssh -L 3001:localhost:3001 your-server # Then open in your browser http://localhost:3001 ``` This forwards port 3001 on your local machine to the server's localhost:3001, keeping the admin UI secure. ### Option 2: Token Authentication If you need direct remote access: ```bash # In .env ADMIN_HOST=0.0.0.0 ADMIN_TOKEN=your-secure-random-token ``` Generate a secure token: ```bash openssl rand -base64 32 ``` The UI will prompt for the token when you load it. ## Admin API Endpoints The admin UI is backed by JSON API endpoints: | Endpoint | Method | Description | |----------|--------|-------------| | `/api/status` | GET | Aggregated status data | | `/api/gateways` | GET | Gateway list with health and scores | | `/api/telemetry` | GET | Time-ranged telemetry stats | | `/api/config` | GET | Current configuration (sanitized) | | `/api/moderation` | GET | Moderation status and blocklist | | `/api/config/save` | POST | Save .env file | | `/api/restart` | POST | Validate config and restart router | All endpoints require the `Authorization: Bearer ` header when `ADMIN_TOKEN` is set. ## Restart from Admin UI The admin UI can restart the router after configuration changes: 1. Make changes in the Settings page 2. Click "Save Configuration" 3. Click "Restart Router" to apply changes The restart endpoint validates the new configuration before restarting, preventing configuration errors from taking down the router. # Configuration (/build/run-wayfinder-router/configuration) All configuration is via environment variables. Create a `.env` file or set them directly in your environment. The admin UI setup wizard at `http://localhost:3001` can generate a `.env` file for you. ## Server | Variable | Default | Description | |----------|---------|-------------| | `PORT` | `3000` | Public server port | | `HOST` | `0.0.0.0` | Public server bind address | | `BASE_DOMAIN` | `localhost` | Base domain for ArNS subdomain routing | | `ROOT_HOST_CONTENT` | _(empty)_ | ArNS name or txId to serve at root domain | | `RESTRICT_TO_ROOT_HOST` | `false` | Only serve root domain content (404 for subdomains/txIds) | | `GRAPHQL_PROXY_URL` | _(empty)_ | Upstream GraphQL endpoint for `/graphql` proxy | ### Root Host Configuration `ROOT_HOST_CONTENT` accepts either an ArNS name or a transaction ID (auto-detected by format): ```bash # ArNS name ROOT_HOST_CONTENT=wayfinder # Transaction ID (43-char base64url) ROOT_HOST_CONTENT=bNbA3TEQVL60xlgCcqdz4ZPHFZ711cZ3hmkpGttDt_U ``` When set, the root domain serves this content instead of the info page. The info page moves to `/wayfinder/info`. ## Operating Mode | Variable | Default | Description | |----------|---------|-------------| | `DEFAULT_MODE` | `proxy` | Operating mode: `proxy` or `route` | | `ALLOW_MODE_OVERRIDE` | `true` | Allow `?mode=` query parameter to override | **Modes:** - **proxy** - Fetch, verify, and serve content through the router - **route** - Redirect clients directly to a gateway URL ## Routing Controls where content is fetched from. | Variable | Default | Description | |----------|---------|-------------| | `ROUTING_STRATEGY` | `fastest` | Gateway selection strategy | | `ROUTING_GATEWAY_SOURCE` | `network` | Where to get the gateway list | | `ROUTING_STATIC_GATEWAYS` | _(see below)_ | Comma-separated gateway URLs (when source=`static`) | | `TRUSTED_PEER_GATEWAY` | `https://turbo-gateway.com` | Gateway for peer list (when source=`trusted-peers`) | | `TRUSTED_ARIO_GATEWAYS` | _(empty)_ | Trusted gateways (when source=`trusted-ario`) | ### Routing Strategies | Strategy | Behavior | Best For | |----------|----------|----------| | `fastest` | Concurrent ping, use first responder | Lowest latency | | `random` | Random selection from healthy gateways | Load distribution | | `round-robin` | Sequential rotation through gateways | Predictable distribution | | `temperature` | Weighted selection based on recent performance | Production (recommended) | The `temperature` strategy tracks per-gateway latency and success rate. Better-performing gateways get more traffic, but slower gateways still receive some requests to detect improvements. ### Gateway Sources | Source | Behavior | |--------|----------| | `network` | All ar.io gateways from the onchain registry (recommended) | | `trusted-peers` | Peer list from `TRUSTED_PEER_GATEWAY` | | `trusted-ario` | Specific gateways from `TRUSTED_ARIO_GATEWAYS` | | `static` | Manual list from `ROUTING_STATIC_GATEWAYS` | ## Verification Controls content integrity verification. | Variable | Default | Description | |----------|---------|-------------| | `VERIFICATION_ENABLED` | `true` | Enable content hash verification | | `VERIFICATION_GATEWAY_SOURCE` | `top-staked` | Source for trusted gateways | | `VERIFICATION_GATEWAY_COUNT` | `3` | Number of gateways to query for verification | | `VERIFICATION_STATIC_GATEWAYS` | _(see below)_ | Comma-separated URLs (when source=`static`) | | `VERIFICATION_RETRY_ATTEMPTS` | `3` | Gateways to try before failing | | `ARNS_CONSENSUS_THRESHOLD` | `2` | Minimum gateways that must agree on ArNS resolution | ### Verification Sources | Source | Behavior | |--------|----------| | `top-staked` | Top N gateways by stake (economic security, recommended) | | `static` | Manual list from `VERIFICATION_STATIC_GATEWAYS` | ## Cache | Variable | Default | Description | |----------|---------|-------------| | `CONTENT_CACHE_ENABLED` | `true` | Enable verified content cache | | `CONTENT_CACHE_MAX_SIZE_BYTES` | `53687091200` (50GB) | Maximum total cache size | | `CONTENT_CACHE_MAX_ITEM_SIZE_BYTES` | `2147483648` (2GB) | Maximum single item size | | `CONTENT_CACHE_PATH` | _(empty)_ | Disk path for persistence (empty = in-memory only) | | `ARNS_CACHE_TTL_MS` | `300000` (5min) | ArNS resolution cache TTL | ### Disk-Backed Cache For production, enable disk persistence: ```bash CONTENT_CACHE_PATH=./data/content-cache ``` When set: - LRU holds metadata only (low memory footprint) - Content stored as files on disk (`.bin` + `.meta.json`) - Cache survives restarts - Atomic writes via temp file + rename for crash safety ## Admin UI The admin UI runs on a **separate port** for security. | Variable | Default | Description | |----------|---------|-------------| | `ADMIN_UI_ENABLED` | `true` | Enable admin UI server | | `ADMIN_PORT` | `3001` | Admin server port (must differ from `PORT`) | | `ADMIN_HOST` | `127.0.0.1` | Admin bind address | | `ADMIN_TOKEN` | _(empty)_ | Bearer token for auth (required when not localhost) | | `ADMIN_OPEN_BROWSER` | `true` | Auto-open browser on startup | When `ADMIN_HOST=0.0.0.0`, you **must** set `ADMIN_TOKEN` to protect the admin UI. ## Resilience | Variable | Default | Description | |----------|---------|-------------| | `RETRY_ATTEMPTS` | `3` | Retry attempts for failed requests | | `RETRY_DELAY_MS` | `100` | Delay between retries | | `CIRCUIT_BREAKER_THRESHOLD` | `3` | Failures before marking gateway unavailable | | `CIRCUIT_BREAKER_RESET_MS` | `60000` (1min) | Time before retrying a broken gateway | | `GATEWAY_HEALTH_TTL_MS` | `300000` (5min) | Health status cache TTL | | `STREAM_TIMEOUT_MS` | `120000` (2min) | Per-chunk stream read timeout | ## Telemetry | Variable | Default | Description | |----------|---------|-------------| | `TELEMETRY_ENABLED` | `true` | Enable telemetry collection | | `TELEMETRY_ROUTER_ID` | `router-{timestamp}` | Instance identifier | | `TELEMETRY_DB_PATH` | `./data/telemetry.db` | SQLite database path | | `TELEMETRY_RETENTION_DAYS` | `30` | Data retention period | | `TELEMETRY_SAMPLE_SUCCESS` | `0.1` | Sampling rate for successful requests (0.0-1.0) | | `TELEMETRY_SAMPLE_ERRORS` | `1.0` | Sampling rate for errors | ## Rate Limiting | Variable | Default | Description | |----------|---------|-------------| | `RATE_LIMIT_ENABLED` | `false` | Enable per-IP rate limiting | | `RATE_LIMIT_WINDOW_MS` | `60000` (1min) | Rate limit window | | `RATE_LIMIT_MAX_REQUESTS` | `1000` | Max requests per IP per window | ## Content Moderation | Variable | Default | Description | |----------|---------|-------------| | `MODERATION_ENABLED` | `false` | Enable content moderation | | `MODERATION_BLOCKLIST_PATH` | `./data/blocklist.json` | Blocklist file (auto-created, hot-reloaded) | | `MODERATION_ADMIN_TOKEN` | _(empty)_ | Bearer token for moderation API | ## Arweave HTTP API Proxy Proxy Arweave node API endpoints (`/info`, `/tx/{id}`, `/block/height/{h}`, etc.). | Variable | Default | Description | |----------|---------|-------------| | `ARWEAVE_API_ENABLED` | `false` | Enable Arweave API proxy | | `ARWEAVE_READ_NODES` | _(Arweave tip nodes)_ | Nodes for GET requests | | `ARWEAVE_WRITE_NODES` | _(falls back to read)_ | Nodes for POST requests | | `ARWEAVE_API_CACHE_ENABLED` | `true` | Cache API responses | ## Gateway Ping Service Background latency probing for the `temperature` routing strategy. | Variable | Default | Description | |----------|---------|-------------| | `PING_ENABLED` | `true` | Enable ping service | | `PING_INTERVAL_HOURS` | `4` | How often to refresh ping data | | `PING_GATEWAY_COUNT` | `50` | Gateways to ping each round | | `PING_TIMEOUT_MS` | `5000` | Timeout per ping | | `PING_CONCURRENCY` | `10` | Maximum concurrent pings | ## Shutdown | Variable | Default | Description | |----------|---------|-------------| | `SHUTDOWN_DRAIN_TIMEOUT_MS` | `15000` (15s) | Grace period for in-flight requests | | `SHUTDOWN_TIMEOUT_MS` | `30000` (30s) | Total shutdown timeout | ## Logging | Variable | Default | Description | |----------|---------|-------------| | `LOG_LEVEL` | `info` | Log level: `trace`, `debug`, `info`, `warn`, `error`, `fatal` | ## Example Configuration ```bash # .env - Production example # Server PORT=3000 HOST=0.0.0.0 BASE_DOMAIN=yourdomain.com ROOT_HOST_CONTENT=your-arns-name # Mode DEFAULT_MODE=proxy ALLOW_MODE_OVERRIDE=false # Routing ROUTING_STRATEGY=temperature ROUTING_GATEWAY_SOURCE=network # Verification VERIFICATION_ENABLED=true VERIFICATION_GATEWAY_SOURCE=top-staked VERIFICATION_GATEWAY_COUNT=3 ARNS_CONSENSUS_THRESHOLD=2 # Cache CONTENT_CACHE_ENABLED=true CONTENT_CACHE_PATH=./data/content-cache CONTENT_CACHE_MAX_SIZE_BYTES=53687091200 # Admin ADMIN_UI_ENABLED=true ADMIN_PORT=3001 ADMIN_HOST=127.0.0.1 # Telemetry TELEMETRY_ENABLED=true TELEMETRY_DB_PATH=./data/telemetry.db # Rate Limiting RATE_LIMIT_ENABLED=true RATE_LIMIT_MAX_REQUESTS=1000 ``` # Wayfinder Router (/build/run-wayfinder-router) Wayfinder Router is a lightweight proxy service that provides a **single trusted endpoint** for accessing Arweave data through the decentralized ar.io gateway network. It fetches content from multiple gateways, verifies integrity via cryptographic hash checking, and serves verified data to your users. ``` Your Users | v +------------------+ | Wayfinder Router | |------------------| | - Verification | | - Caching | | - Telemetry | +------------------+ | +--------------+--------------+ | | | v v v +------+ +------+ +------+ | GW1 | | GW2 | | GW3 | +------+ +------+ +------+ \___________+___________/ | ar.io Gateway Network ``` ## Why Use Wayfinder Router? } title="Content Verification" description="Cryptographically verify all content against trusted gateways before serving" /> } title="Single Endpoint" description="Give your users one reliable URL instead of managing gateway failover in client code" /> } title="Built-in Caching" description="LRU cache with optional disk persistence for fast repeat access" /> } title="Admin Dashboard" description="Web UI for configuration, monitoring, and content moderation" /> ## Two Operating Modes | Mode | Behavior | Use Case | |------|----------|----------| | **Proxy** | Fetch, verify, and serve content through the router | Full verification, caching, single domain | | **Route** | Redirect clients directly to a gateway URL | Lower latency, client-side verification | ## When to Use Wayfinder Router **Use Wayfinder Router when you need:** - A single trusted endpoint for your application - Server-side content verification - Centralized caching and rate limiting - Content moderation capabilities - Monitoring and telemetry **Use the SDKs directly when you need:** - Client-side verification in browsers - Custom routing logic in your application - Integration with existing infrastructure ## Architecture Overview The router separates **routing** (where to fetch data) from **verification** (who to trust): - **Routing Gateways** - Where content is fetched from (all ar.io gateways, trusted peers, or static list) - **Verification Gateways** - Who to trust for hash verification (top-staked gateways or static list) This means even if a routing gateway is compromised, tampered content is detected and rejected. When verification fails, the router returns an error to the client rather than serving potentially malicious content. ## Get Started } title="Quick Start" description="Get running in 30 seconds with a standalone binary or Docker" href="/build/run-wayfinder-router/quick-start" /> } title="Configuration" description="Environment variables for routing, verification, caching, and more" href="/build/run-wayfinder-router/configuration" /> } title="Admin UI" description="Web dashboard for monitoring, configuration, and moderation" href="/build/run-wayfinder-router/admin-ui" /> } title="Operations" description="Monitoring, troubleshooting, and production deployment" href="/build/run-wayfinder-router/operations" /> # Operations (/build/run-wayfinder-router/operations) ## Monitoring ### Health and Readiness ```bash # Health check (is the process running?) curl http://localhost:3000/wayfinder/health # Readiness check (is the router ready to serve traffic?) curl http://localhost:3000/wayfinder/ready # API Guard compatible health check curl http://localhost:3000/ar-io/healthcheck ``` Use these endpoints for load balancer health checks and orchestration systems. ### Prometheus Metrics ```bash curl http://localhost:3000/wayfinder/metrics ``` Exposes standard metrics for scraping by Prometheus. Configure your Prometheus scrape targets to point at `/wayfinder/metrics`. ### Gateway Statistics ```bash # Summary statistics curl http://localhost:3000/wayfinder/stats/gateways # List all tracked gateways curl http://localhost:3000/wayfinder/stats/gateways/list # Detailed stats for a specific gateway curl http://localhost:3000/wayfinder/stats/gateways/:gateway # Export telemetry data curl http://localhost:3000/wayfinder/stats/ ### Router Info ```bash curl http://localhost:3000/wayfinder/info ``` Returns current configuration, version, uptime, and operating mode. ### Telemetry Storage Telemetry is stored in SQLite at `TELEMETRY_DB_PATH` (default `./data/telemetry.db`). Configure sampling rates to control storage growth: ```bash TELEMETRY_SAMPLE_SUCCESS=0.1 # Sample 10% of successful requests TELEMETRY_SAMPLE_ERRORS=1.0 # Record all errors TELEMETRY_RETENTION_DAYS=30 # Auto-purge old data ``` ### Log Levels Set log verbosity via `LOG_LEVEL`: ```bash LOG_LEVEL=debug # trace, debug, info, warn, error, fatal ``` ## Content Moderation Block ArNS names or transaction IDs from being served. ### Setup ```bash MODERATION_ENABLED=true MODERATION_ADMIN_TOKEN= ``` Generate a secure token: ```bash openssl rand -base64 32 ``` The blocklist is stored at `MODERATION_BLOCKLIST_PATH` (default `./data/blocklist.json`) and is hot-reloaded on changes. ### API Endpoints All admin endpoints require `Authorization: Bearer ` header. ```bash # Block an ArNS name curl -X POST http://localhost:3000/wayfinder/moderation/block \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"type":"arns","value":"badcontent","reason":"Policy violation"}' # Block a transaction ID curl -X POST http://localhost:3000/wayfinder/moderation/block \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"type":"txid","value":"abc123...","reason":"DMCA takedown"}' ``` ```bash # List all blocked content curl http://localhost:3000/wayfinder/moderation/blocklist \ -H "Authorization: Bearer YOUR_TOKEN" # Check if content is blocked (no auth required) curl http://localhost:3000/wayfinder/moderation/check/arns/somename # Moderation statistics curl http://localhost:3000/wayfinder/moderation/stats \ -H "Authorization: Bearer YOUR_TOKEN" ``` ```bash # Unblock content curl -X DELETE http://localhost:3000/wayfinder/moderation/block/arns/badcontent \ -H "Authorization: Bearer YOUR_TOKEN" # Reload blocklist from disk curl -X POST http://localhost:3000/wayfinder/moderation/reload \ -H "Authorization: Bearer YOUR_TOKEN" ``` ## Cache Management For production, enable disk-backed caching. See [Configuration](/build/run-wayfinder-router/configuration#cache) for all cache settings. ### Clearing Caches **From source:** ```bash bun run clear:telemetry # Clear telemetry database bun run clear:all # Clear all data (telemetry + cache) ``` **Binary or Docker:** Delete the data directory contents directly: ```bash rm -rf ./data/content-cache/* rm ./data/telemetry.db ``` ## Graceful Shutdown The router handles SIGTERM and SIGINT signals with a two-phase shutdown: 1. **Drain phase** - Stop accepting new connections, wait for in-flight requests 2. **Force exit** - If drain exceeds timeout, force shutdown ```bash # Configuration SHUTDOWN_DRAIN_TIMEOUT_MS=15000 # 15s drain period SHUTDOWN_TIMEOUT_MS=30000 # 30s total timeout ``` ```bash # Graceful stop kill -TERM # Docker (sends SIGTERM, waits 10s, then SIGKILL) docker stop wayfinder-router # Docker with custom timeout docker stop -t 30 wayfinder-router ``` ## Troubleshooting ### Port Conflicts If port 3000 or 3001 is already in use: ```bash PORT=3080 ADMIN_PORT=3081 ./wayfinder-router-linux-x64 ``` `ADMIN_PORT` must differ from `PORT` - the router validates this at startup. ### Gateway Health Issues Check gateway status via the admin UI Gateways page or: ```bash curl http://localhost:3000/wayfinder/stats/gateways ``` If all gateways show as unhealthy: - Verify internet connectivity - Check `ROUTING_GATEWAY_SOURCE` - if `static`, ensure URLs are correct - Check circuit breaker settings - Review logs: `LOG_LEVEL=debug` ### ArNS Resolution Failures ArNS names require consensus across multiple verification gateways. If resolution fails: - Check `ARNS_CONSENSUS_THRESHOLD` (default: 2) - Verify the ArNS name exists on the network - Check verification gateway health - Ensure `VERIFICATION_ENABLED=true` ### Subdomain Routing Not Working ArNS subdomains require `BASE_DOMAIN` to match your actual domain: ```bash # For local development BASE_DOMAIN=localhost # For production BASE_DOMAIN=yourdomain.com ``` Requests to `{name}.yourdomain.com` are only recognized as ArNS subdomains if `BASE_DOMAIN=yourdomain.com`. ### Content Verification Failures If content consistently fails verification: - Check `VERIFICATION_GATEWAY_COUNT` - more gateways increases reliability but adds latency - Verify `VERIFICATION_GATEWAY_SOURCE` is correctly configured - Check if the content transaction is still being seeded - Review logs for specific hash mismatch details ### Memory Usage If memory usage is high: - Enable disk-backed cache: `CONTENT_CACHE_PATH=./data/content-cache` - Reduce cache size: `CONTENT_CACHE_MAX_SIZE_BYTES` - Reduce telemetry retention: `TELEMETRY_RETENTION_DAYS` - Increase sampling: `TELEMETRY_SAMPLE_SUCCESS=0.01` ### Slow Response Times If responses are slow: - Check routing strategy - `temperature` adapts to gateway performance - Verify verification gateway health - Check network connectivity to gateways - Review `STREAM_TIMEOUT_MS` setting - Consider reducing `VERIFICATION_GATEWAY_COUNT` if latency is acceptable ## Production Checklist Before deploying to production: - [ ] Set `BASE_DOMAIN` to your actual domain - [ ] Configure `ROOT_HOST_CONTENT` if serving at root - [ ] Enable disk cache: `CONTENT_CACHE_PATH=./data/content-cache` - [ ] Set appropriate cache size limits - [ ] Enable rate limiting: `RATE_LIMIT_ENABLED=true` - [ ] Secure admin UI: `ADMIN_TOKEN` or keep `ADMIN_HOST=127.0.0.1` - [ ] Configure telemetry sampling rates - [ ] Set up monitoring (health checks, Prometheus) - [ ] Configure reverse proxy (nginx/Caddy) with SSL - [ ] Set up log aggregation - [ ] Plan for graceful shutdown in your orchestration # Quick Start (/build/run-wayfinder-router/quick-start) Get your Wayfinder Router running in **30 seconds**. No configuration needed - the admin UI wizard will guide you through setup. ## Quickstart Download the latest binary for your platform from [GitHub Releases](https://github.com/ar-io/wayfinder-router/releases): | Platform | Binary | |----------|--------| | Linux x64 | `wayfinder-router-linux-x64` | | Linux ARM64 | `wayfinder-router-linux-arm64` | | macOS Intel | `wayfinder-router-darwin-x64` | | macOS Apple Silicon | `wayfinder-router-darwin-arm64` | | Windows x64 | `wayfinder-router-windows-x64.exe` | Each release includes `checksums.txt` with SHA256 hashes for verification. ### Download and make executable ```bash # Download (example for Linux x64) curl -LO https://github.com/ar-io/wayfinder-router/releases/latest/download/wayfinder-router-linux-x64 # Make executable chmod +x wayfinder-router-linux-x64 # Verify checksum (optional but recommended) sha256sum wayfinder-router-linux-x64 ``` ### Run ```bash ./wayfinder-router-linux-x64 ``` The admin UI automatically opens at `http://localhost:3001` with a setup wizard. ### Test ```bash # Fetch a test transaction curl http://localhost:3000/4jBV3ofWh41KhuTs2pFvj-KBZWUkbrbCYlJH0vLA6LM # Expected output: test ``` ### Run with Docker ```bash docker run -p 3000:3000 -p 3001:3001 ghcr.io/ar-io/wayfinder-router:latest ``` Or with Docker Compose (uses ports 3020/3021 by default): ```bash git clone https://github.com/ar-io/wayfinder-router cd wayfinder-router docker compose up -d ``` ### Open Admin UI Navigate to `http://localhost:3001` in your browser. The setup wizard will guide you through configuration. ### Test ```bash curl http://localhost:3000/4jBV3ofWh41KhuTs2pFvj-KBZWUkbrbCYlJH0vLA6LM # Expected output: test ``` **Production Docker:** ```bash docker run -p 3000:3000 -p 3001:3001 \ --env-file .env \ -v ./data:/app/data \ ghcr.io/ar-io/wayfinder-router:latest ``` The `./data` volume persists telemetry, content cache, and blocklist data. Requires [Bun](https://bun.sh) >= 1.0.0. ### Clone and install ```bash git clone https://github.com/ar-io/wayfinder-router cd wayfinder-router bun install ``` ### Configure (optional) ```bash cp .env.example .env # Edit .env as needed, or skip and use the setup wizard ``` ### Run ```bash # Development with hot reload bun run dev # Production bun run start ``` The admin UI opens at `http://localhost:3001`. **Build standalone binaries:** ```bash bun run build:binaries # outputs to ./builds/ ``` ## Setup Wizard On first run (when `BASE_DOMAIN=localhost`), the admin UI shows a guided setup wizard: ### Domain Configuration Configure your base domain, port, and optional root host content. For local development, keep the defaults. ### Routing Settings Choose your operating mode and routing strategy: | Setting | Options | Recommended | |---------|---------|-------------| | **Mode** | `proxy` or `route` | `proxy` for full verification | | **Strategy** | `fastest`, `random`, `round-robin`, `temperature` | `temperature` for production | | **Gateway Source** | `network`, `trusted-peers`, `static` | `network` for full decentralization | ### Verification Settings Enable content verification and configure trust settings: | Setting | Default | Description | |---------|---------|-------------| | **Enabled** | `true` | Verify content hashes before serving | | **Trust Source** | `top-staked` | Use top-staked gateways for verification | | **Gateway Count** | `3` | Number of gateways to query | | **Consensus** | `2` | Minimum gateways that must agree | ### Save Configuration The wizard generates a `.env` file. You can copy it to clipboard or save directly to the server. ## Accessing Content Once running, access Arweave content through your router: ```bash # By transaction ID http://localhost:3000/{txId} http://localhost:3000/{txId}/path/to/file # By ArNS name (subdomain) http://{arns-name}.localhost:3000/ http://{arns-name}.localhost:3000/path/to/file ``` ## Serving Content at Root Domain Configure `ROOT_HOST_CONTENT` to serve an ArNS name or transaction ID at your root domain: ```bash # In .env ROOT_HOST_CONTENT=wayfinder # Or a transaction ID ROOT_HOST_CONTENT=bNbA3TEQVL60xlgCcqdz4ZPHFZ711cZ3hmkpGttDt_U ``` With this configuration: - `https://yourdomain.com/` serves the configured content - `https://yourdomain.com/docs` serves the `/docs` path within that content - `https://yourdomain.com/wayfinder/info` still returns router info ## What's Next? } title="Configuration" description="Full reference for all environment variables" href="/build/run-wayfinder-router/configuration" /> } title="Admin UI" description="Monitor gateways, view telemetry, manage moderation" href="/build/run-wayfinder-router/admin-ui" /> } title="Operations" description="Production deployment, monitoring, and troubleshooting" href="/build/run-wayfinder-router/operations" /> # Accessing Data (/build/testnet/accessing-data) The gateway at [**ar-io.dev**](https://ar-io.dev) indexes your upload optimistically — usually within seconds — then serves and queries it. ## Fetching your data - **Exact bytes:** `GET https://ar-io.dev/raw/` returns the data item's exact bytes, unmodified, with no redirect. - **Rendered / path-aware:** `GET https://ar-io.dev/` serves the content and, for **manifests**, resolves paths (e.g. `//index.html`). For browser security the gateway **302-redirects** each item to its own **sandbox subdomain** — `https://.ar-io.dev/` — so each item is isolated on its own origin. Follow redirects (`curl -L`; browsers do it automatically). ## Querying with GraphQL `POST https://ar-io.dev/graphql` — query your data by id, tags, or owner as soon as it's indexed: ```graphql query { transactions(first: 1, owners: [""]) { edges { node { id tags { name value } } } } } ``` ## Data response headers On a data fetch (`/raw/` or the sandbox `/`): | Header | Meaning | |---|---| | `x-ar-io-data-id` | The data item id served | | `x-ar-io-verified` | **`false`** here — optimistic sandbox data, **not** weave-verified | | `x-ar-io-trusted` | `true` — served from the trusted sandbox bundler | | `x-ar-io-stable` | `false` — the tx is not in a stable (mainnet) block (it never will be) | | `x-ar-io-digest` / `content-digest` / `etag` | sha-256 of the bytes (integrity / caching) | | `x-ar-io-hops`, `x-cache` | Gateway hop count; cache HIT/MISS | | `x-arweave-owner-address`, `x-arweave-tag-*` | The data item's owner and ANS-104 tag metadata | Seeing **`x-ar-io-verified: false`** is expected and correct on this sandbox — the data isn't on mainnet Arweave. Don't treat it as an error. ## ArNS resolution headers On an ArNS resolution (`https://.ar-io.dev`): | Header | Meaning | |---|---| | `x-arns-name` / `x-arns-basename` | The resolved name | | `x-arns-resolved-id` | The target data id the name points at | | `x-arns-record` | The record served (`@` = apex, else the undername) | | `x-arns-ttl-seconds` | Resolution TTL (e.g. `3600`) | | `x-arns-ant-id` / `x-arns-ant-program-id` | The backing ANT and the **devnet** ANT program | | `x-arns-undername-limit`, `x-arns-record-index` | Undername capacity and record index | ## Next steps # ArNS Names (/build/testnet/arns) The sandbox runs a **devnet ArNS registry**. You can buy **real names** paid in **staging-ARIO credits** and resolve them at `https://.ar-io.dev`. New to ArNS? Start with the [ArNS overview](/learn/arns). There are two ways to register: the **bundler route** (simplest — pay with your Turbo credits), or **directly with the ar.io SDK** against the devnet programs. ## Buying through the bundler Buying through the bundler debits your [upload credits](/build/testnet/uploading-and-credits#getting-credits), and (like mainnet) **Turbo takes custody of nothing** — the ANT (Metaplex Core asset) backing the name is minted straight to a Solana key you control. Use the [Turbo SDK](/sdks/turbo-sdk) rather than hand-rolling the raw HTTP calls: buying needs your **owner** key to sign the ANT-creation transaction Turbo prepares, and the SDK drives that signature exchange for you. ```ts const secretKeyBase58 = bs58.encode(secretKey); // your devnet Solana keypair const turbo = TurboFactory.authenticated({ signer: new HexSolanaSigner(secretKeyBase58), // the payer — spends your upload credits token: 'solana', paymentServiceConfig: { url: 'https://payment.services.ar-io.dev' }, }); // The owner can be the same wallet as the payer, or a different one you control. const owner = solanaOwnerSigner(secretKeyBase58); const { antId, messageId } = await turbo.buyArNSName({ name: '', owner, type: 'lease', // or 'permabuy' years: 1, // leases only }); ``` Check the price first with `turbo.getArNSPriceForName({ intent: 'Buy-Name', name, type: 'lease', years: 1 })`, or hit the same route directly: ```http GET https://payment.services.ar-io.dev/v1/arns/price/buy-name/?type=lease&years=1 ``` **Minimum name length: 8 characters — bundler route only.** This is an anti-squat rule on the shared registry; shorter names return `400` (`Name must be at least 8 characters…`). Buying [directly with the SDK](#advanced-buying-directly-with-the-sdk) is not subject to this floor. ## Advanced: buying directly with the SDK You can also buy and manage names **directly** against the devnet programs — paying the name price in **staging ARIO** and **devnet SOL** yourself — instead of going through the bundler proxy. This path is **not** bound by the bundler's 8-character floor (though very short or premium names may still be reserved on-chain). ```ts const ario = ARIO.init({ backend: 'solana', rpc: createSolanaRpc(DEVNET_RPC_URL), rpcSubscriptions: createSolanaRpcSubscriptions(DEVNET_RPC_URL.replace(/^https/, 'wss')), // writes need this coreProgramId: DEVNET_PROGRAM_IDS.core, // pass each program id; there is no `programIds` option garProgramId: DEVNET_PROGRAM_IDS.gar, arnsProgramId: DEVNET_PROGRAM_IDS.arns, antProgramId: DEVNET_PROGRAM_IDS.ant, signer, }); const cost = await ario.getTokenCost({ /* name, type, years */ }); await ario.buyRecord({ /* name, type, years */ }); ``` The payment mint is read from on-chain config automatically (the staging ARIO mint). For the devnet program IDs and the ar.io SDK's devnet configuration, see the [Reference](/build/testnet/reference) and the SDK's [advanced networks](/sdks/ar-io-sdk/advanced-networks) and [networks](/sdks/ar-io-sdk/networks) guides. ## Resolving a name Once bought, `https://.ar-io.dev` resolves the record, returning `x-arns-*` response headers. To point a name at your uploaded data, set its record with `turbo.setArNSRecord({ antId, owner, transactionId, undername, ttlSeconds })` (same sponsored, credits-charged bundler route as buying — see [managing an ANT](/sdks/turbo-sdk/the-twelve-sponsored-actions)), or through your ANT directly with the ar.io SDK. **Names outlive data.** A name keeps resolving because the record lives on the devnet registry — but the **data behind it disappears** after the ~3-day purge. Re-upload and re-point when you need it back. See [Accessing Data](/build/testnet/accessing-data#arns-resolution-headers) for the full set of ArNS resolution headers. ## Next steps # Funds & Faucet (/build/testnet/funds-and-faucet) Everything in the sandbox runs on **testnet value**. Depending on what you're doing you need two or three tokens — all free. | Token | Network | What it's for | Where to get it | |---|---|---|---| | **SOL** | Solana **devnet** | Transaction fees on every Solana action | `solana airdrop 1 --url devnet` or [faucet.solana.com](https://faucet.solana.com) | | **ARIO (staging)** | Solana **devnet** | Buying ArNS names and paying for uploads | The ar.io faucet → [faucet.services.ar-io.dev](https://faucet.services.ar-io.dev) | | **Base Sepolia ETH** | Base **Sepolia** | *(optional)* Funding credits with `base-eth` or the x402 path | Any [Base Sepolia faucet](https://docs.base.org/docs/tools/network-faucets/) | | **USDC (devnet)** | Solana **devnet** | *(optional)* Funding credits with `solana-usdc` | [faucet.circle.com](https://faucet.circle.com) — choose **Solana Devnet**; 20 USDC per address every 2 hours | You always need a little **devnet SOL** — even after claiming ARIO — because SOL pays the transaction fee for *using* your tokens (funding credits, buying names, etc.). ## The ARIO faucet The staging-ARIO faucet lives at [**faucet.services.ar-io.dev**](https://faucet.services.ar-io.dev). It's **GitHub-gated** to prevent sybil abuse: ### Sign in with GitHub Your GitHub account must be at least **30 days old**. ### Enter your Solana address The address you want to receive staging ARIO on. ### Claim The faucet dispenses **2,500 ARIO per claim** and pays the transfer fee for you. **Rate limit:** one claim per GitHub account per **8-hour** window. A wallet that **already holds the maximum (2,500 ARIO)** won't receive more — you'll see `Recipient … already has more than the maximum quantity`. Claim to a fresh address, or spend down first. The faucet page also displays the current **Solana devnet program IDs and the ARIO mint**, copy-ready — handy if you're accessing the devnet contracts directly with the [ar.io SDK](/build/testnet/arns#advanced-buying-directly-with-the-sdk). See the [Reference](/build/testnet/reference) for the current values. Trouble claiming, or your GitHub account didn't pass verification? Ask in the [ar.io Discord](https://discord.com/invite/HGG52EtTc2). ## Agents and CI The faucet's GitHub OAuth consent is a browser flow and **can't be completed headlessly**, so an agent or CI job can't claim on its own. The reliable pattern is: ### A human claims once A person signs in with GitHub and claims **2,500 ARIO** to the wallet address your agent will use. ### The agent uses the ARIO From there, everything is a scriptable API call — uploading, funding credits, and buying/resolving ArNS names all work without any browser interaction. Re-claim (again by a human) when the balance runs low. **Faucet agent skill.** The faucet ships an `ario-testnet-faucet` agent skill documenting the claim API and flow. Pair it with the human-claims-once pattern above so your agent has funds without needing to solve the OAuth gate. ## Next steps # Testnet Sandbox (/build/testnet) The **ar.io Testnet Sandbox** is a free, public environment where you can build against the **entire ar.io stack** — uploading, payments, ArNS names, and gateway serving — running on **Solana devnet** with a staging ARIO token. It's the fastest way to wire up an integration end to end **without spending real money or touching mainnet**. It's also an ideal target for **AI coding agents and CI**: the services are plain HTTP APIs, the tokens are free from a faucet, and nothing you do here can incur real cost or post permanent data. See [building with an agent](#building-with-an-agent) below. An item of **5 MiB or less uploads free** with nothing but a devnet Solana keypair, within 100 MiB per wallet: no faucet, no credits, no USDC. Start with [Uploading & Credits](/build/testnet/uploading-and-credits). **Read this first — the three rules of the sandbox:** 1. **Data is ephemeral.** Uploads are served and indexed for a short window (**~3 days**), then purged. Nothing here is permanent — don't store anything you need to keep. 2. **Never on-chain.** Your data is *never* posted to mainnet Arweave. The gateway serves it optimistically, so it returns `x-ar-io-verified: false` and disappears when purged. 3. **Testnet value only.** Fund with testnet tokens or Stripe **test cards**. Real tokens and cards are rejected — and there is no mainnet settlement, so you'd be throwing value away. ## What you get } title="Turbo bundler (upload + payment)" description="Accepts testnet tokens, issues upload credits, and includes a free tier for small items." /> } title="Gateway (ar-io.dev)" description="Indexes, serves, and GraphQL-queries your uploads, and resolves ArNS names on Solana devnet." /> } title="ArNS on devnet" description="Buy real ArNS names on the devnet registry, paid in staging ARIO, and resolve them at .ar-io.dev." /> } title="Faucets" description="Free devnet SOL, staging ARIO, and Base Sepolia ETH for everything you need to test." /> ## Endpoints | Service | URL | Health check | |---|---|---| | Upload API | `https://upload.services.ar-io.dev` | `GET /v1/info` | | Payment API | `https://payment.services.ar-io.dev` | `GET /v1/info` | | Gateway (serve / resolve / GraphQL) | `https://ar-io.dev` | `GET /ar-io/healthcheck` | | ARIO faucet (staging ARIO) | `https://faucet.services.ar-io.dev` | `GET /healthcheck` | **Network:** primarily Solana **devnet** — staging **ARIO** SPL mint `6vTw5CysRXQ4ybbHkDUiisHWVsBeMtUzYvJqs2iqHyaN` (6 decimals), and Circle's devnet **USDC** mint `4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU` for the `solana-usdc` funding path. The optional `base-eth` funding path and x402 use **Base Sepolia**. Point your app at `*.services.ar-io.dev` and `ar-io.dev` to exercise the full **ar.io** stack — upload, payment, ArNS, and gateway — together. These are shared sandbox endpoints, so treat them as a testbed rather than a production plane. ## Quick start ### Get a Solana devnet keypair and a little devnet SOL SOL covers transaction fees on every Solana action. Airdrop some from [faucet.solana.com](https://faucet.solana.com) or `solana airdrop 1 --url devnet`. ### Claim staging ARIO from the faucet Sign in with GitHub at [faucet.services.ar-io.dev](https://faucet.services.ar-io.dev) and claim **2,500 ARIO** to your Solana address. See [Funds & Faucet](/build/testnet/funds-and-faucet). ### Upload a small file for free Items up to **105 KiB** are eligible for the free tier. Use the Turbo SDK pointed at the sandbox, or `POST /v1/tx`. See [Uploading & Credits](/build/testnet/uploading-and-credits). ### Need more? Top up credits with your staging ARIO Fund upload credits with `ario`, `solana`, `solana-usdc`, or `base-eth`, or Stripe **test cards**. See [Uploading & Credits](/build/testnet/uploading-and-credits#getting-credits). ### Buy an ArNS name (≥ 8 characters) Paid in ARIO credits, minted straight to a Solana key you control so it resolves out of the box — Turbo takes custody of nothing. See [ArNS Names](/build/testnet/arns). ### Resolve it Your name is live at `https://.ar-io.dev`. See [Accessing Data](/build/testnet/accessing-data). ## What you must NOT do - **Don't rely on persistence.** Treat every upload as disposable (≤ 3 days). - **Don't send mainnet tokens or real cards** to the payment service — they're rejected. - **Don't point production apps at these endpoints.** This is a shared, wipeable sandbox with reserved short names and rate limits, not a production plane. - **Don't try to defeat the faucet's GitHub gate or the rate limits** — they keep the shared testbed usable for everyone. ## Building with an agent The sandbox is a natural fit for AI coding assistants and automated tests — free tokens, plain HTTP endpoints, and no way to spend real value or write permanent data. **One human step: the faucet.** The ARIO faucet is **GitHub-gated** to prevent abuse, and the OAuth consent can't be completed headlessly. The standard pattern is: a **human claims once** to the wallet address your agent will use, then the agent simply **uses** the ARIO. After that, uploading, funding, and buying ArNS names are all scriptable API calls. See [Funds & Faucet → Agents and CI](/build/testnet/funds-and-faucet#agents-and-ci). This documentation is also available as machine-readable text (`/llms-full.txt`) and every page has an **Open in AI** action, so you can hand the full sandbox reference to your agent in one step. ## Next steps # Reference (/build/testnet/reference) ## Solana devnet identifiers For direct access with the [ar.io SDK](/build/testnet/arns#advanced-buying-directly-with-the-sdk) (`ARIO.init({ backend: 'solana', coreProgramId, garProgramId, arnsProgramId, antProgramId })`, or the `DEVNET_PROGRAM_IDS` constant exported from `@ar.io/sdk/solana`, one field per program): | Program / mint | Address | |---|---| | `ario_core` | `8Njx9wPkXiNzDCgjwVsJFRjpAEV34gGW3n8DzX3V23m1` | | `ario_gar` | `7WsDTrtZBsfKtnP33XkjuqXCY69JE7n4QVYpynqJCFxz` | | `ario_arns` | `6EZNezcg4rc5hnh8HG34vGquT3WpW5xXypzPb24uyEpp` | | `ario_ant` | `DbHbRwUD1oAn1mrDSqtWtvwGcNrmhWdD2g8L4xmeQ7NX` | | **ARIO mint** | `6vTw5CysRXQ4ybbHkDUiisHWVsBeMtUzYvJqs2iqHyaN` (6 decimals) | **Program IDs rotate.** The staging programs are periodically redeployed, so the program addresses above can change (the ARIO mint is stable). Always verify against the current values shown copy-ready on the [faucet](https://faucet.services.ar-io.dev), or read them from the ar.io SDK's devnet constants — see [advanced networks](/sdks/ar-io-sdk/advanced-networks). If ArNS resolutions return an unexpected `x-arns-ant-program-id`, you're likely on a stale program ID. The ar.io SDK also hardcodes the staging ARIO mint as its `DEVNET_ARIO_MINT`, and reads the payment mint from on-chain config automatically — so for the direct SDK path you generally only need the program IDs. ## Responses you'll see | Code | Where | Meaning | |---|---|---| | `200` `winc:"0"` | Upload | Uploaded, free | | `402 FREE_TIER_EXHAUSTED` | Upload | Out of free allowance → [top up credits](/build/testnet/uploading-and-credits#getting-credits) | | `413` | Upload | Data item over 10 MiB | | `400 Token not supported` | Payment | Non-testnet funding token | | `400 Name must be at least 8 characters…` | ArNS | Name too short (bundler route only) | | `302` → `*.ar-io.dev/` | Gateway | [Sandbox-subdomain redirect](/build/testnet/accessing-data#fetching-your-data) (follow it) | | `status: recorded` | ArNS poll | Purchase settled | ## Caveats recap - **Ephemeral:** data purges after **~3 days**. A name keeps resolving (the record lives on the devnet registry), but the data behind it disappears once purged. - **Testnet only:** don't send real tokens or real cards; they're rejected, and there is no mainnet settlement. - **Shared and rate-limited:** short ArNS names are reserved, the faucet is gated and rate-limited, and free-tier allowances are per-wallet and per-IP. ## Related # Uploading & Credits (/build/testnet/uploading-and-credits) Uploads go through the sandbox **Turbo bundler**. Data items are **ANS-104** — use the [Turbo SDK](/sdks/turbo-sdk) pointed at the sandbox endpoints, or `POST` a signed data item directly. **5 MiB or less? It is free, and the first example below is all you need.** No credits, no faucet, no USDC. The sandbox's x402 endpoints answer `402` asking for Base Sepolia USDC: that is the x402 path for paying in USDC, not a sign that the free tier is used up. Pay with credits or x402 only for items over the free size. ## Uploading with the Turbo SDK Point the SDK at the sandbox upload and payment services and sign with your **devnet Solana** keypair: ```ts const signer = new HexSolanaSigner(bs58.encode(secretKey)); // your devnet Solana keypair const turbo = TurboFactory.authenticated({ signer, token: 'solana', gatewayUrl: 'https://api.devnet.solana.com', // Solana RPC — used to VERIFY funding txs, NOT the ar.io gateway uploadServiceConfig: { url: 'https://upload.services.ar-io.dev' }, paymentServiceConfig: { url: 'https://payment.services.ar-io.dev' }, }); const { id, winc } = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream(path), fileSizeFactory: () => size, }); // Read the bytes back from the ar.io gateway, not from Turbo: // https://ar-io.dev/raw/ ``` **Watch the `gatewayUrl`.** Here it's the **Solana RPC** — how Turbo verifies your funding transactions — *not* `ar-io.dev`. You fetch uploaded data from the [gateway](/build/testnet/accessing-data) separately. Funding with `base-eth` instead? Use `token: 'base-eth'` and an EVM signer. Funding with USDC on Solana? Use `token: 'solana-usdc'` with the *same* Solana signer — you just need devnet USDC alongside a little devnet SOL for the transaction fee. ### Raw POST If you've already signed an ANS-104 data item, post the raw bytes: ```http POST https://upload.services.ar-io.dev/v1/tx Content-Type: application/octet-stream ``` A `200` returns `{ id, winc, dataCaches, ... }`. **`winc: "0"` means the upload was free.** ## Limits & responses - **Hard max size: 10 MiB** per data item. Larger uploads return **`413`** (`Data item is too large…`). - **Free tier** (no payment required), far larger than mainnet's 105 KiB: - Up to **5 MiB per item** is eligible to be free. - **100 MiB total free per wallet** (lifetime) **and** **100 MiB per IP /24 subnet** (lifetime): an upload must fit under **both** or it isn't free. - `GET https://upload.services.ar-io.dev/info` reports the current limits as `freeTier`. - Over the free allowance, or an item too big to be free, returns **`402`** `{ code: "FREE_TIER_EXHAUSTED", topUpUrl, byteCount }` → [top up credits](#getting-credits). ### Check your remaining free allowance Before uploading, check how many free-tier bytes a wallet has left with `GET /v1/account/free` — **no signature required**, any wallet by address: ```http GET https://payment.services.ar-io.dev/v1/account/free?address= ``` ```jsonc // bytesRemaining: null = unlimited (exempt wallet); 0 = free tier disabled { "bytesRemaining": 7340032 } ``` With the [Turbo SDK](/sdks/turbo-sdk): ```ts const { bytesRemaining } = await turbo.getFreeStatus(); // your own wallet (authenticated) const { bytesRemaining } = await turbo.getFreeStatus(''); // any wallet (unauthenticated) ``` Or the CLI: `turbo free-status --address `. `bytesRemaining` is a wallet-side figure and advisory — the per-subnet cap and the authoritative free/charge decision are applied at upload time. Deployment-wide free-tier limits are reported by `GET /v1/info`. ## Getting credits Credits (Turbo **"winc"**) are what the bundler debits for **paid uploads** and **ArNS purchases**. Fund them with **testnet** value only. **Accepted funding tokens:** | Token | Network | Notes | |---|---|---| | `ario` | ARIO staging (Solana devnet) | Mint `6vTw5CysRXQ4ybbHkDUiisHWVsBeMtUzYvJqs2iqHyaN`, 6 decimals — **fee-free** | | `solana` | Solana devnet | | | `solana-usdc` | Solana devnet | Circle devnet USDC, mint `4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU`, 6 decimals — priced 1:1 in USD | | `base-eth` | Base Sepolia | | | `base-usdc` | Base Sepolia | Testnet USDC contract | | `ethereum` | Ethereum Sepolia | Verification is network-agnostic and this deployment points at **Sepolia**, so only Sepolia ETH can be credited | | `usdc` | Ethereum Sepolia | Testnet USDC contract | `arweave`, `matic`/`pol` and `kyve` are **rejected** with `Token not supported` — they settle on mainnet, and there is no mainnet settlement here. Note that `ethereum` and `usdc` above are *accepted*: the sandbox pins them to Sepolia RPCs and testnet contracts, so no mainnet value is ever involved. `GET /v1/info`'s `addresses` map advertises exactly the tokens this deployment accepts, so it is the authoritative list if the table above ever drifts. ### Crypto top-up Submit your funding transaction to `POST /v1/account/balance/:token`. Get the per-token receiving addresses from `GET /v1/info` (the `addresses` map). The transaction is verified on the **testnet** RPC for that token, then your balance is credited. ### Stripe (test cards) The fiat top-up flow runs in **test mode**. Use Stripe test cards — e.g. `4242 4242 4242 4242` with any future expiry and CVC. No real money moves. ### x402 (USDC) The unsigned x402 upload path runs on **Base Sepolia** testnet USDC: `POST /x402/upload/unsigned`, then pay the returned `402` quote via an `X-PAYMENT` header. See [x402 uploading](/build/upload/x402-uploading-to-turbo) for the full flow. ## Next steps # Advanced Uploading with Turbo (/build/upload/advanced-uploading-with-turbo) Learn how to upload data to Arweave using the **Turbo SDK** for a streamlined upload experience with multiple payment options and authentication methods. ## What You'll Learn - How to install and authenticate with the Turbo SDK - Different authentication methods (Arweave, Ethereum, Solana, etc.) - How to purchase Turbo Credits - How to upload files, strings, binary data, and entire folders to Arweave - Browser and Node.js implementation examples - Using the versatile `upload` method for all data types ## Prerequisites - Node.js environment or modern web browser - Wallet for authentication (Arweave, Ethereum, Solana, etc.) - Basic understanding of JavaScript/TypeScript ## Quick Start ### Install the Turbo SDK ```bash # For Node.js npm install @ardrive/turbo-sdk # For Yarn users yarn add @ardrive/turbo-sdk ``` ### Authenticate with Your Wallet Choose your preferred authentication method: ```typescript // Load your Arweave JWK file const jwk = JSON.parse(fs.readFileSync('wallet.json', 'utf-8')) const turbo = await TurboFactory.authenticated({ privateKey: jwk, // ArweaveJWK type token: 'arweave', // Default token type }) ``` ```typescript // Your Ethereum private key (with 0x prefix) const privateKey = '0x1234...' // EthPrivateKey type // Create an Ethereum signer instance const signer = new EthereumSigner(privateKey) const turbo = await TurboFactory.authenticated({ signer, token: 'ethereum', }) ``` ```typescript // Your Solana secret key (as Uint8Array) const secretKey = new Uint8Array([...]) // SolSecretKey type const turbo = await TurboFactory.authenticated({ privateKey: bs58.encode(secretKey), token: 'solana' }) ``` ```typescript // Your Polygon private key (with 0x prefix) const privateKey = '0x1234...' // EthPrivateKey type // Create an Ethereum signer instance for Polygon const signer = new EthereumSigner(privateKey) const turbo = await TurboFactory.authenticated({ signer, token: 'matic', // or 'pol' }) ``` ```typescript async function initializeTurbo() { await window.arweaveWallet.connect([ 'ACCESS_ADDRESS', 'ACCESS_PUBLIC_KEY', 'SIGN_TRANSACTIONS', 'SIGN_MESSAGE', 'SIGNATURE', ]) const turbo = await TurboFactory.authenticated({ signer: new ArConnectSigner(window.arweaveWallet), }) } ``` ```typescript // Global variables for Wagmi config and connector let config = null let connector = null let turboInstance = null // Function to set up Wagmi configuration config = wagmiConfig connector = wagmiConnector } // Function to initialize Turbo with Wagmi try { if (!config || !connector) { throw new Error( 'Wagmi config and connector not set. Call setWagmiConfig first.', ) } console.log('Initializing Turbo client...') // Create a provider that uses wagmi's signMessage const provider = { getSigner: () => ({ signMessage: async (message) => { const arg = message instanceof String ? message : { raw: message } const ethAccount = getAccount(config) return await signMessage(config, { message: arg, account: ethAccount.address, connector: connector, }) }, }), } // Create the Turbo signer const signer = new InjectedEthereumSigner(provider) // Set up the public key signer.setPublicKey = async () => { const message = 'Sign this message to connect to Turbo' const ethAccount = getAccount(config) const signature = await signMessage(config, { message: message, account: ethAccount.address, connector: connector, }) const hash = await hashMessage(message) const recoveredKey = await recoverPublicKey({ hash, signature, }) signer.publicKey = Buffer.from(toBytes(recoveredKey)) } // Initialize the signer await signer.setPublicKey() turboInstance = await TurboFactory.authenticated({ signer: signer, token: 'base-eth', // Can be changed to 'ethereum' or 'matic', etc. }) console.log('Turbo client initialized successfully') return turboInstance } catch (error) { console.error('Error initializing Turbo client:', error) turboInstance = null throw error } } ``` ```typescript if (!window.ethereum) { throw new Error('Please install MetaMask to use this application') } try { const accounts = await window.ethereum.request({ method: 'eth_requestAccounts', }) const metaMaskProvider = window.ethereum.providers?.find( (p) => p.isMetaMask, ) const provider = new BrowserProvider(metaMaskProvider ?? window.ethereum) const signer = await provider.getSigner() const turbo = TurboFactory.authenticated({ signer: new InjectedEthereumSigner({ getSigner: () => signer }), token: 'ethereum', }) return { turbo, address: accounts[0] } } catch (error) { console.error('Connection failed:', error) throw error } } ``` ```typescript try { // Check if Phantom is installed if (window.solana) { const provider = window.solana const publicKey = new PublicKey((await provider.connect()).publicKey) const wallet: SolanaWalletAdapter = { publicKey, signMessage: async (message: Uint8Array) => { // Call Phantom's signMessage method const { signature } = await provider.signMessage(message) return signature }, } solanaTurboInstance = TurboFactory.authenticated({ token: 'solana', walletAdapter: wallet, }) } } catch (err) { console.error(err) } } ``` ### Purchase Turbo Credits [Turbo Credits](/build/upload/turbo-credits) are the payment medium used by the Turbo Upload Service. Each Credit represents a 1:1 conversion from the upload power of the Arweave native token (AR). - **Fiat Currency**: Credit/debit cards via the [Turbo Top Up App](https://console.ar.io/topup) - **Cryptocurrencies**: AR, ETH, SOL, POL, ARIO, USDC, ETH (BASE) - **Multiple Wallets**: Ethereum, Solana, and Arweave wallets supported ```typescript // Initialize authenticated client const turbo = await TurboFactory.authenticated({ privateKey: jwk }) // Top up with AR tokens const topUpResult = await turbo.topUpWithTokens({ tokenAmount: WinstonToTokenAmount(100_000_000), // 0.0001 AR }) ``` ```typescript // Initialize authenticated client const turbo = await TurboFactory.authenticated({ signer: new EthereumSigner(privateKey), token: 'ethereum', }) // Top up with ETH tokens const topUpResult = await turbo.topUpWithTokens({ tokenAmount: 0.001, // 0.001 ETH }) ``` ```typescript // Initialize authenticated client const turbo = await TurboFactory.authenticated({ privateKey: bs58.encode(secretKey), token: 'solana' }) // Top up with SOL tokens const topUpResult = await turbo.topUpWithTokens({ tokenAmount: 0.1, // 0.1 SOL }) ``` ```typescript // Initialize authenticated client const turbo = await TurboFactory.authenticated({ signer: new EthereumSigner(privateKey), token: 'matic', }) // Top up with POL tokens const topUpResult = await turbo.topUpWithTokens({ tokenAmount: 1.0, // 1.0 POL }) ``` ### Upload Your First File ```typescript // Upload a single file using the versatile upload method const result = await turbo.upload({ data: file, // Can be File, Blob, Buffer, Uint8Array, ArrayBuffer, or string dataItemOpts: { tags: [ { name: "Content-Type", value: file.type || "application/octet-stream" }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, }); console.log("File uploaded!", { id: result.id, url: `https://turbo-gateway.com/${result.id}`, owner: result.owner, dataCaches: result.dataCaches, }); ``` ## Uploading Files ### Basic File Upload ```typescript // Upload a single file using the versatile upload method const result = await turbo.upload({ data: file, // Can be File, Blob, Buffer, Uint8Array, ArrayBuffer, or string dataItemOpts: { tags: [ { name: "Content-Type", value: file.type || "application/octet-stream" }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, }); console.log("File uploaded!", { id: result.id, url: `https://turbo-gateway.com/${result.id}`, owner: result.owner, dataCaches: result.dataCaches, }); ``` ### Upload with Custom Tags ```typescript const result = await turbo.upload({ data: file, dataItemOpts: { tags: [ { name: "Content-Type", value: "application/json" }, { name: "App-Name", value: "MyApp-v1.0" }, { name: "App-Version", value: "1.0.0" }, { name: "Description", value: "My application data" }, ], }, }); ``` ### Upload Strings ```typescript // Upload a string const stringResult = await turbo.upload({ data: "Hello, Arweave!", dataItemOpts: { tags: [ { name: "Content-Type", value: "text/plain" }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, }); ``` ### Upload JSON Data ```typescript // Upload a JSON object const jsonData = { message: "Hello", timestamp: Date.now() }; const jsonResult = await turbo.upload({ data: JSON.stringify(jsonData), dataItemOpts: { tags: [ { name: "Content-Type", value: "application/json" }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, }); ``` ### Upload Binary Data ```typescript // Upload binary data const binaryData = new Uint8Array([1, 2, 3, 4, 5]); const binaryResult = await turbo.upload({ data: binaryData, dataItemOpts: { tags: [ { name: "Content-Type", value: "application/octet-stream" }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, }); ``` ### Upload Multiple Files ```typescript const files = [file1, file2, file3]; const uploadPromises = files.map((file) => turbo.upload({ data: file, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type || "application/octet-stream", }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, }) ); const results = await Promise.all(uploadPromises); console.log("All files uploaded!", results); ``` ### Upload an Entire Folder (Node.js) ```typescript const folderResult = await turbo.uploadFolder({ folderPath: "./my-website", dataItemOpts: { tags: [ { name: "App-Name", value: "MyWebsite-v1.0" }, { name: "Content-Type", value: "application/x.arweave-manifest+json" }, ], }, manifestOptions: { indexFile: "index.html", fallbackFile: "404.html", }, }); console.log("Folder uploaded!", { manifestId: folderResult.manifestResponse?.id, fileCount: folderResult.fileResponses.length, manifest: folderResult.manifest, }); ``` ### Upload Multiple Files as a Folder (Browser) ```typescript const files = [file1, file2, file3]; const webFolderResult = await turbo.uploadFolder({ files: files, dataItemOpts: { tags: [{ name: "App-Name", value: "MyWebsite-v1.0" }], }, manifestOptions: { indexFile: "index.html", }, }); ``` ## Browser Implementation Examples ### File Input with Drag & Drop ```html Turbo Upload Example .drop-zone { border: 2px dashed #ccc; border-radius: 10px; padding: 20px; text-align: center; margin: 20px 0; } .drag-over { border-color: #007bff; background-color: #f8f9fa; } Drag and drop files here or click to select // Your Turbo initialization code here // ... (authentication code from above) const fileInput = document.getElementById("file-input"); const dropZone = document.getElementById("drop-zone"); // File input handler fileInput.addEventListener("change", async (event) => { const files = Array.from(event.target.files); for (const file of files) { await uploadFile(file); } }); // Drag and drop handlers dropZone.addEventListener("dragover", (e) => { e.preventDefault(); e.stopPropagation(); dropZone.classList.add("drag-over"); }); dropZone.addEventListener("dragleave", (e) => { e.preventDefault(); e.stopPropagation(); dropZone.classList.remove("drag-over"); }); dropZone.addEventListener("drop", async (e) => { e.preventDefault(); e.stopPropagation(); dropZone.classList.remove("drag-over"); const files = Array.from(e.dataTransfer.files); for (const file of files) { await uploadFile(file); } }); async function uploadFile(file) { try { const result = await turbo.upload({ data: file, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type || "application/octet-stream", }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, }); console.log("File uploaded!", { id: result.id, url: `https://turbo-gateway.com/${result.id}`, name: file.name, size: file.size, }); } catch (error) { console.error("Upload failed:", error); } } ``` ## Advanced Features ### Check Upload Costs ```typescript // Get upload cost for specific file size const costs = await turbo.getUploadCosts({ bytes: [file.size], }); console.log(`Upload cost: ${costs[0].winc} winc`); console.log(`USD cost: $${costs[0].usd}`); ``` ### Check Balance ```typescript // Get current balance const balance = await turbo.getBalance(); console.log(`Available credits: ${balance.controlledWinc} winc`); ``` ### Upload with Progress Tracking ```typescript const result = await turbo.upload({ data: file, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type || "application/octet-stream" }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, events: { onUploadProgress: (progress) => { console.log( `Upload progress: ${Math.round((progress.processedBytes / progress.totalBytes) * 100)}%` ); }, onSigningProgress: (progress) => { console.log( `Signing progress: ${Math.round((progress.processedBytes / progress.totalBytes) * 100)}%` ); }, }, }); ``` ### Upload with Error Handling ```typescript try { const result = await turbo.upload({ data: file, dataItemOpts: { tags: [ { name: "Content-Type", value: file.type || "application/octet-stream" }, { name: "App-Name", value: "MyApp-v1.0" }, ], }, events: { onUploadError: (error) => { console.error("Upload failed:", error); }, onSigningError: (error) => { console.error("Signing failed:", error); }, }, }); console.log("Upload successful:", result); } catch (error) { console.error("Upload error:", error); // Handle error appropriately } ``` ## Benefits of Using Turbo - **Versatile upload method** - Upload files, strings, binary data, or entire folders with a single method - **Multiple payment options** - Pay with fiat, crypto, or AR tokens - **Easy integration** - Simple SDK for both Node.js and browsers - **Automatic retry** - Built-in retry logic for failed uploads - **Cost transparency** - See upload costs before confirming - **Fast uploads** - Optimized for speed and reliability - **Folder support** - Upload entire directories with automatic manifest generation ## Next Steps } > Buy Turbo Credits with fiat or crypto }> Discover best practices for organizing your data } > Learn how to organize files with manifests # Getting Started with Turbo (/build/upload/bundling-services) Upload data using Turbo, an ar.io gateway run by the ArDrive team. Turbo handles the infrastructure so you can focus on shipping. ## What is Turbo? Turbo is an ar.io gateway run by the ArDrive team. It serves permanent data like any ar.io gateway, and it adds paid uploads: it bundles your files into ANS-104 data items, issues the credits you pay with, and posts the data permanently to Arweave. **How the pieces fit.** [ar.io](/learn) is the network: gateways serve and index permanent data, ArNS names it, and anyone can run a gateway. Turbo is one of those gateways, at `turbo-gateway.com`, run by the ArDrive team with uploads and payments attached. [ArDrive](https://app.ardrive.io) is the app built on it. Arweave is the storage layer underneath, where the data ends up. So the part that makes Turbo different from a gateway you run yourself is not reading, which it does the same way: it is that Turbo accepts paid uploads, and carries the funding, performance tuning and operational complexity that come with them. Turbo supports flexible payment options, including fiat and major cryptocurrencies, and is built to handle high-throughput workloads with consistent performance. Whether you’re prototyping or running production systems, Turbo provides a reliable upload layer backed by enterprise-grade infrastructure. If you’re looking for a managed upload service tailored to your use case — including custom limits, performance, or integration support — book a call to learn how we can get one set up for you: [Contact Us!](https://ar.io/contact) ## Get Started ### Install the SDK ```bash npm install @ardrive/turbo-sdk ``` ### Set Up Your Wallet Create a new wallet or use an existing one: ```bash # Create a new wallet (easy way) npx permaweb/wallet > key.json ``` Then load it in your code: ```js // Load your wallet const jwk = JSON.parse(fs.readFileSync("./key.json", "utf-8")); const signer = new ArweaveSigner(jwk); // Initialize Turbo const turbo = TurboFactory.authenticated({ signer }); ``` ### Get Turbo Credits Purchase Turbo Credits to pay for uploads. When you upload, credits are automatically used and Turbo handles the payment to Arweave. **Option 1: Via the Web Interface** - Go to [console.ar.io/topup](https://console.ar.io/topup) - Pay with fiat currencies (credit cards) or crypto tokens (ARIO, USDC, SOL, ETH, POL, AR) **Option 2: Via the SDK** ```js // Purchase credits programmatically const fundResult = await turbo.topUpWithTokens({ tokenAmount: TOKEN_AMOUNT, tokenType: "solana", // or 'ethereum', 'matic', 'arweave' }); ``` **Check Your Balance** ```js const balance = await turbo.getBalance(); console.log(`Balance: ${balance.winc} winc`); ``` ### Upload Your Data ```js tab="Upload File" const fileData = fs.readFileSync("./myfile.jpg"); const result = await turbo.upload({ data: fileData, dataItemOpts: { tags: [ { name: "Content-Type", value: "image/jpeg" }, { name: "Title", value: "My Image" }, ], }, }); console.log("Upload ID:", result.id); console.log("Owner:", result.owner); ```` ```js tab="Upload Folder" const folderResult = await turbo.uploadFolder({ folderPath: "./my-folder", dataItemOpts: { tags: [ { name: "Bundle-Format", value: "binary" }, { name: "Bundle-Version", value: "2.0.0" }, ], }, }); console.log("Folder Upload ID:", folderResult.id); console.log("Manifest ID:", folderResult.manifestId); ```` ```js tab="Upload Raw Data" const data = JSON.stringify({ message: "Hello Arweave!", timestamp: Date.now(), }); const result = await turbo.upload({ data: data, dataItemOpts: { tags: [ { name: "Content-Type", value: "application/json" }, { name: "App-Name", value: "MyApp" }, ], }, }); console.log("Upload ID:", result.id); console.log("Owner:", result.owner); ``` ## Advanced Features ### Turbo Credits System Learn about our flexible payment system that supports multiple currencies and payment methods. → [Understanding Turbo Credits](/build/upload/turbo-credits) ### Data Organization - [**Tagging**](/build/upload/tagging) - Organize your data with metadata - [**Manifests**](/build/upload/manifests) - Create folder structures and bundles - [**Encryption**](/build/upload/encryption) - Secure your sensitive data - [**ArFS**](/build/advanced/arfs) - File system protocol for structured storage ## Production Ready Turbo implements the **[ANS-104 bundling specification](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md)**, providing enterprise-grade infrastructure for permanent data storage. **Perfect for:** Developers, production applications, high-volume uploads, and any project needing reliable permanent storage with flexible payment options. | Feature | Turbo Bundling | Alternative Options | | -------------------- | -------------------------------- | ---------------------------- | | **Payment Options** | Fiat, ARIO, USDC, SOL, POL, AR | AR tokens only | | **Implementation** | Simple SDK integration | Manual transaction handling | | **Performance** | Optimized bundling & retry logic | Depends on implementation | | **Reliability** | Built-in redundancy | Manual error handling | | **Cost** | Optimized for large uploads | Higher per-transaction costs | | **Setup Complexity** | Easy with SDK | Complex protocol knowledge | **Need help deciding?** Most developers should use Turbo for its simplicity and payment flexibility. Only consider alternatives for specialized use cases requiring maximum control. ## Ready to Get Started? } > Start building with Turbo's powerful bundling service. }> Explore the full SDK documentation and examples. } > Organize your data with metadata and tags. # Encryption (/build/upload/encryption) **Arweave has no built-in encryption.** All encryption and decryption must be handled client-side before uploading data to the network. Arweave is completely data-agnostic - it stores whatever data you provide without any knowledge of whether it's encrypted or not. ## How Encryption Works on Arweave **Critical Points:** - **No native encryption**: Arweave provides no encryption services whatsoever - **Client-side only**: You must encrypt data before uploading - **Data-agnostic storage**: Arweave stores any data type, including encrypted data - **Your responsibility**: You handle all encryption, key management, and decryption - **Permanent security**: Once encrypted and stored, data remains secure forever ## Encryption Options ### 1. Manual Client-Side Encryption Encrypt your data before uploading with Turbo: ```js // Encrypt sensitive data const data = "Sensitive information"; const secretKey = "your-secret-key"; const encryptedData = CryptoJS.AES.encrypt(data, secretKey).toString(); // Upload encrypted data const result = await turbo.upload({ data: encryptedData, dataItemOpts: { tags: [ { name: "Content-Type", value: "application/octet-stream" }, { name: "Encrypted", value: "true" }, { name: "Cipher", value: "AES-256-GCM" }, { name: "Cipher-IV", value: "YWJjZGVmZ2hpams=" }, // 12 byte initialization vector as Base64 ], }, }); ``` ## Encryption Standards ### Encryption Methods - **AES-256-GCM**: Authenticated encryption (recommended) - **AES-256-CTR**: Stream cipher for large files - **Any encryption method**: Arweave supports any encryption you choose (must be indicated in `Cipher` tag for ArFS compliance) ### Required Tags When uploading encrypted data, include these tags: ```js { name: "Content-Type", value: "application/octet-stream" // Required for encrypted data }, { name: "Cipher", value: "AES-256-GCM" // Specify encryption method }, { name: "Cipher-IV", value: "base64-encoded-iv" // Initialization vector } ``` ## ArFS Protocol (Optional Standardization) The [Arweave File System (ArFS)](/build/advanced/arfs) protocol provides optional standardization for encrypted storage: - **Private Drives**: Encrypt entire file systems - **File-level encryption**: Each file has its own encryption key - **Selective sharing**: Share individual files without exposing the entire drive - **Key derivation**: Uses HKDF-SHA256 with wallet signatures - **Completely optional**: You can use any encryption method you prefer **ArDrive Web App:** Data uploaded through the ArDrive web app to Private Drives is encrypted for you using the standards set in the ArFS protocol. ArDrive is simply a web application that implements ArFS - there is no separate "ArDrive Encryption Service." **ArFS Privacy:** To learn more about ArFS encryption schema, key derivation, and private drive management, see our detailed [ArFS Privacy & Encryption documentation](/build/advanced/arfs/privacy). ## Getting Started For most users, the ArDrive web app provides the easiest way to encrypt and store data using ArFS standards: **Create a private drive** in the ArDrive web app **Set a strong password** for your drive **Upload files** - they're automatically encrypted using ArFS **Access files** using your password and wallet For developers who need custom encryption: **Choose an encryption library** (Crypto-JS, Web Crypto API) **Encrypt your data** before uploading **Add proper tags** to indicate encryption **Store keys securely** for decryption ## Security Considerations **Important:** Never store encryption keys in your code or public repositories. Use secure key management practices and consider hardware security modules for production applications. **Best Practices:** - Use strong, randomly generated keys - Implement proper key rotation - Store keys securely (not in code) - Use authenticated encryption (AES-GCM) - Validate data integrity after decryption ## Next Steps } > Use the ArDrive web app for easy encrypted file storage using ArFS. }> Explore the Arweave File System protocol for structured storage. } > Purchase credits for programmatic uploads. # Give an Agent a Budget (/build/upload/give-an-agent-a-budget) An agent that uploads for you needs a way to pay that you can cap, watch and cut off, and it must never hold your key. Turbo gives you two ways to do that. Pick one before you hand anything over. | | Credit share | x402 | |---|---|---| | Who holds the funds | You, as Turbo credits. The agent spends against an approval | The agent, in USDC on Base | | The cap | An amount of credits per approval | An amount of USDC per upload, plus whatever the wallet holds | | Expiry | A number of seconds you set | None. The wallet balance is the limit | | Cutting it off | Revoke the approval at any time; unused credits return to you | Move the USDC out of the wallet | | What the agent holds | Its own keypair, with no balance | Its own keypair and the USDC | | Where it is documented | this page, and [Turbo Credit Sharing](/sdks/turbo-sdk/turbo-credit-sharing) | [x402 Uploading To Turbo](/build/upload/x402-uploading-to-turbo) | A credit share is the tighter control: a cap, a clock and a revoke, with the credits staying in your Turbo account. x402 caps each upload but not the total, and has no revoke: it is for an agent that already holds USDC or that pays many services the same way. The rest of this page is the credit-share workflow, with a short x402 section at the end. Items within the [free tier](/build/upload/upload-and-verify#before-you-start) upload free whichever path the agent uses, and do not draw on the budget. ## Before you start - **Your wallet holds credits.** Buy them with any supported wallet; see [Paying for Uploads](/build/upload/turbo-credits). - **The agent has its own keypair.** Generate one for it. It never needs a balance of its own, and it never sees your key. The agent needs two things from you: its own key, and your wallet address. - **Node.js 18+** and `npm install @ardrive/turbo-sdk bs58` on both sides. The examples use Solana keypairs. On the testnet sandbox, use devnet keypairs and the sandbox service URLs from [Upload and Verify](/build/upload/upload-and-verify#before-you-start). ## Part 1: Fund the agent ### Step 1: Decide the cap Price the work first. `getUploadCosts` returns the credit price (`winc`) for a size, so price the largest upload the agent will make, multiply by how many, and add a margin: the signed item is a little larger than the file, so the charge runs a little over the quote. ```ts const [{ winc: perUpload }] = await turbo.getUploadCosts({ bytes: [6 * 1024 * 1024] }); const cap = BigInt(perUpload) * 10n * 12n / 10n; // ten uploads of 6 MiB, plus 20% ``` The cap is the most the agent can ever spend from this approval. Setting it to what the task needs, not to what you hold, is the control. ### Step 2: Create the approval ```ts const approval = await turbo.shareCredits({ approvedAddress: agentAddress, approvedWincAmount: cap.toString(), expiresBySeconds: 60 * 60 * 24, // one day }); ``` The response is the approval: ```json { "approvalDataItemId": "", "payingAddress": "", "approvedAddress": "", "approvedWincAmount": "", "usedWincAmount": "0", "creationDate": "", "expirationDate": "" } ``` While the approval lives, the shared amount leaves the balance you can spend (`winc` on `getBalance`) and stays in the balance you control (`controlledWinc`). It comes back to `winc` when you revoke the approval or it expires. Set an expiry even when you expect to revoke by hand: an agent that stalls with an open approval is otherwise a cap that never closes. ### Step 3: Watch it `getCreditShareApprovals` returns every approval an address has given and every one it has received. `usedWincAmount` on a given approval is what the agent has spent so far. ```ts const { givenApprovals } = await turbo.getCreditShareApprovals({ userAddress: myAddress }); for (const a of givenApprovals) { console.log(a.approvedAddress, a.usedWincAmount, "of", a.approvedWincAmount, "expires", a.expirationDate); } ``` ### Step 4: Cut it off Revoking removes every approval you have given to that address, at once, and returns the unused credits to you. ```ts await turbo.revokeCredits({ revokedAddress: agentAddress }); ``` The agent's next paid upload against your address fails, unless Turbo already holds the same bytes (see [Upload](#upload)). There is no partial revoke: to lower a cap, revoke and share again with the new amount. ## Part 2: Upload as the agent The agent signs with its own key and names the funder in `paidBy`. Turbo uses the approval for the upload before it looks at the agent's own balance, which is zero. ### Check the budget first ```ts const size = fs.statSync(path).size; const { receivedApprovals } = await turbo.getBalance(); const [{ winc: price }] = await turbo.getUploadCosts({ bytes: [size] }); const remaining = receivedApprovals .filter((a) => a.payingAddress === funderAddress) .reduce((sum, a) => sum + BigInt(a.approvedWincAmount) - BigInt(a.usedWincAmount), 0n); if (BigInt(price) > remaining) throw new Error("stop: this upload is over the remaining budget"); ``` `remaining` is what is left on the approvals this funder has given the agent. `effectiveBalance` on the same response is the agent's own credits plus what is left on every approval it holds, from any funder. The check is deliberately strict. `getUploadCosts` returns the list price even for an item small enough to be free, so the check can stop an upload that would have cost nothing. That is the safe direction: an agent that stops reports, and an agent that guesses spends. ### Upload ```ts const receipt = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream(path), fileSizeFactory: () => size, dataItemOpts: { tags: [{ name: "Content-Type", value: "application/octet-stream" }], paidBy: [funderAddress], }, }); ``` The receipt is the same as for any upload, and `winc` on it is what the approval was charged. Verify the bytes the way [Upload and Verify](/build/upload/upload-and-verify#step-2-fetch-the-bytes-back) does; nothing about verification changes because someone else paid. Uploading the same bytes twice returns the first item's id with `winc: "0"`: Turbo keeps one copy and charges nothing for the second request. `paidBy` takes a list. With more than one funder, Turbo draws on the approvals in the order you give them. ## When to stop | You see | It means | Do | |---|---|---| | `price` over `remaining` before the upload | The next upload does not fit what is left on the approval | Stop and report what is left. Do not retry with a smaller file unless that is the task | | `402` on an upload with `paidBy` set | The approval cannot pay for this item: it is spent, expired, revoked or was never created for this key, and the item is over the free size | Stop and report the funder's address and the receipt ids so far. Do not pay the x402 offer in the body and do not top up the agent's own balance. The funder decides whether to share more | | `402` on an upload without `paidBy` | The agent's own balance is zero, which is by design | Set `paidBy`. Do not pay the x402 offer in the body | | `receivedApprovals` is empty | No approval exists for this key, or it has expired or been revoked | Stop. Check the agent is using the address the funder approved | | `winc: "0"` on the receipt for a small item | The item fit the free tier | Not a problem. The budget was not touched | | The id of an earlier upload, with `winc: "0"` | The same bytes were uploaded before | Not a problem. Use the id | The `402` arrives the same way whichever of those is the cause, and the same way as for an exhausted free tier, so the response alone does not say whether the budget is spent or never existed. Recorded on the sandbox for a 6 MiB item posted to `/v1/tx` with `x-paid-by` set and no approval left (status 402, headers abridged): ``` Content-Type: application/json; charset=utf-8 X-Free-Tier-Exhausted: true X-Payment-Required: x402-1 {"x402Version":1,"accepts":[{"scheme":"exact","network":"base-sepolia","maxAmountRequired":"440164","resource":"https://upload.services.ar-io.dev/v1/tx", ...}]} ``` The SDK throws on it with `status` set to `402` and the body in the message; the headers are not on the error. To tell the cases apart, read `receivedApprovals`: a spent approval is still listed, with less left (`approvedWincAmount` minus `usedWincAmount`) than the item's price; an expired or revoked one is not listed. The x402 offer in that body is payable by anyone holding USDC on Base, and an agent with x402 tooling can pay it without asking. A credit-share agent must treat it as a stop, never as an invitation. ## Part 3: x402 per upload Give the agent a wallet with USDC on Base and let it pay for each upload as it goes. x402 also needs `npm install x402-fetch`; without it the upload fails before anything is signed. The cap is per upload: ```ts const turbo = TurboFactory.authenticated({ privateKey, token: "base-usdc" }); await turbo.upload({ data, fundingMode: new X402Funding({ maxMUSDCAmount: 1_000_000 }), // at most 1 USDC for this upload }); ``` `maxMUSDCAmount` is in millionths of a USDC, so `1_000_000` is 1 USDC. When the quoted price is over it, the x402 client the SDK uses throws `Payment amount exceeds maximum allowed` and pays nothing, which is the stop condition. Without `maxMUSDCAmount` that client caps each payment at 0.1 USDC. There is no expiry and no revoke: the wallet's balance is the budget, so fund it with what the task needs and no more. The endpoints, the unsigned variant and the ecosystem tooling are on [x402 Uploading To Turbo](/build/upload/x402-uploading-to-turbo). ## Full scripts Two files, one per side. Both run against mainnet by default and against the sandbox with `SANDBOX=1`. `fund-agent.mjs`, run by you: ```js // Share a capped, expiring credit budget with an agent's address, list it, or revoke it. // Usage: KEYFILE=./funder.json node fund-agent.mjs share // KEYFILE=./funder.json node fund-agent.mjs list // KEYFILE=./funder.json node fund-agent.mjs revoke const sandbox = process.env.SANDBOX === "1"; const secretKey = Uint8Array.from(JSON.parse(fs.readFileSync(process.env.KEYFILE, "utf8"))); const turbo = TurboFactory.authenticated({ signer: new HexSolanaSigner(bs58.encode(secretKey)), token: "solana", ...(sandbox && { gatewayUrl: "https://api.devnet.solana.com", uploadServiceConfig: { url: "https://upload.services.ar-io.dev" }, paymentServiceConfig: { url: "https://payment.services.ar-io.dev" }, }), }); const myAddress = bs58.encode(secretKey.subarray(32)); const [action, agentAddress, bytes, uploads] = process.argv.slice(2); if (action === "share") { const [{ winc: perUpload }] = await turbo.getUploadCosts({ bytes: [Number(bytes)] }); const cap = (BigInt(perUpload) * BigInt(uploads) * 12n) / 10n; // plus 20% for fees const { winc } = await turbo.getBalance(); if (cap > BigInt(winc)) throw new Error(`stop: cap ${cap} is over the ${winc} winc you can spend`); const approval = await turbo.shareCredits({ approvedAddress: agentAddress, approvedWincAmount: cap.toString(), expiresBySeconds: 60 * 60 * 24, }); console.log(JSON.stringify(approval, null, 2)); } else if (action === "list") { const { givenApprovals } = await turbo.getCreditShareApprovals({ userAddress: myAddress }); console.log(JSON.stringify(givenApprovals, null, 2)); } else if (action === "revoke") { const revoked = await turbo.revokeCredits({ revokedAddress: agentAddress }); console.log(JSON.stringify(revoked, null, 2)); } else { throw new Error("usage: share | list | revoke "); } ``` `agent-upload.mjs`, run by the agent: ```js // Upload one file on a funder's credit share. Stops before the upload when its list price is over what is left. // Usage: KEYFILE=./agent.json FUNDER= node agent-upload.mjs ./file.bin const sandbox = process.env.SANDBOX === "1"; const path = process.argv[2]; const funder = process.env.FUNDER; if (!path || !funder) throw new Error("give the file as the first argument and the funder's address in FUNDER"); const secretKey = Uint8Array.from(JSON.parse(fs.readFileSync(process.env.KEYFILE, "utf8"))); const turbo = TurboFactory.authenticated({ signer: new HexSolanaSigner(bs58.encode(secretKey)), token: "solana", ...(sandbox && { gatewayUrl: "https://api.devnet.solana.com", uploadServiceConfig: { url: "https://upload.services.ar-io.dev" }, paymentServiceConfig: { url: "https://payment.services.ar-io.dev" }, }), }); // 1. Budget check. The agent's own balance is zero by design; the approval is what pays. const size = fs.statSync(path).size; const [{ winc: price }] = await turbo.getUploadCosts({ bytes: [size] }); const { receivedApprovals } = await turbo.getBalance(); const remaining = receivedApprovals .filter((a) => a.payingAddress === funder) .reduce((sum, a) => sum + BigInt(a.approvedWincAmount) - BigInt(a.usedWincAmount), 0n); console.log(JSON.stringify({ size, price, remaining: remaining.toString() })); if (BigInt(price) > remaining) { throw new Error("stop: over the remaining budget; report what is left and wait for the funder"); } // 2. Upload against the approval. Keep the receipt. let receipt; try { receipt = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream(path), fileSizeFactory: () => size, dataItemOpts: { tags: [{ name: "Content-Type", value: "application/octet-stream" }], paidBy: [funder], }, }); } catch (e) { if (e.status === 402) { throw new Error("stop: the approval cannot pay for this item; do not pay the x402 offer; report to the funder"); } throw e; } fs.writeFileSync(`${path}.receipt.json`, JSON.stringify(receipt, null, 2)); console.log(JSON.stringify({ id: receipt.id, winc: receipt.winc, paidBy: funder })); ``` Recorded on the sandbox on 2026-10-09 with these two scripts and `SANDBOX=1`, sharing enough for one upload of 5,347,738 bytes (a little over the sandbox's 5 MiB free size) and uploading three files of that size: ``` $ node fund-agent.mjs share 5347738 1 { "approvedWincAmount": "83100467793", "usedWincAmount": "0", "expirationDate": "2026-10-10T04:16:13.438Z", ... } $ node agent-upload.mjs ./a.bin {"size":5347738,"price":"69250389828","remaining":"83100467793"} {"id":"W5_xhak5tI9HKRqr6u376ZBZaAq0HYef0UPbRmSGCww","winc":"69252409803","paidBy":""} $ node agent-upload.mjs ./b.bin {"size":5347738,"price":"69250389828","remaining":"13848057990"} Error: stop: over the remaining budget; report what is left and wait for the funder $ node fund-agent.mjs list [ { "approvedWincAmount": "83100467793", "usedWincAmount": "69252409803", ... } ] $ node fund-agent.mjs revoke [ { "usedWincAmount": "69252409803", "inactiveReason": "revoked", ... } ] $ node agent-upload.mjs ./c.bin {"size":5347738,"price":"69250389828","remaining":"0"} Error: stop: over the remaining budget; report what is left and wait for the funder ``` The second and third uploads never reached the service: the budget check stopped them. The charge on the first ran 2,019,975 winc over the quote, which is what the 20% margin is for. Separate checks on the same day, with the same calls: - **An expired approval drops off and its credits come back.** An approval set to expire in 60 seconds was gone from the agent's `receivedApprovals` 75 seconds later, and the funder's `winc` was back to its value before the share. - **A free-size item does not touch the approval.** A 1 KiB item uploaded with `paidBy` returned `winc: "0"` and left `usedWincAmount` at `0`. - **A spent approval answers `402`.** With the budget check removed, a second 6 MiB upload against a used approval returned the `402` shown under [When to stop](#when-to-stop), and so did an upload after the revoke. ## Next steps - [Upload and Verify](/build/upload/upload-and-verify): prove the agent's upload is what it says it is. - [Turbo Credit Sharing](/sdks/turbo-sdk/turbo-credit-sharing): the CLI commands, `--paid-by`, and the rules on re-sharing. - [Paying for Uploads](/build/upload/turbo-credits): buying credits and just-in-time payment for the funder's side. # Upload Data (/build/upload) import { CreditCard, Upload, Code, Tag, Shield, FolderOpen, Zap, Check, Image, } from "lucide-react"; Arweave enables **permanent data storage** with a single payment. Unlike traditional cloud storage that requires ongoing fees, your data is preserved forever. ## Upload Methods There are multiple ways to upload data to Arweave. Each has its own attributes and characteristics to help you decide which is best for your use case. Turbo{" "} Recommended } description="An ar.io gateway run by the ArDrive team: it serves data like any gateway, and adds paid uploads • Pay with credit cards, AR, ETH, SOL, POL • Free uploads under 105 KiB • Automatic retry & confirmation" href="/build/upload/bundling-services" icon={} /> } /> } /> ## Why Developers Choose Turbo **Cost Effective** - Pay per byte, not empty chunks - only pay for actual data uploaded - Free uploads under 105 KiB - subsidized small file uploads - No failed upload charges - automatic retry without extra costs **Developer Experience** - TypeScript & CLI support - choose your preferred tools - Simple 3-line integration - get started in minutes - Comprehensive documentation - extensive guides and examples **Enterprise Ready** - High Availability - reliable service for production apps - Handles millions of uploads daily - battle-tested infrastructure used by ArDrive - Open source infrastructure - fully auditable and transparent ## Get Started in Minutes With Turbo, uploading to Arweave is as simple as using any cloud storage API: ```typescript const turbo = TurboFactory.authenticated({ privateKey }); const uploadResult = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream("./my-file.pdf"), }); // Your file is now permanently stored! ``` Adding uploads to a server that already exists, where a new dependency has to get past a review? [`turbo-upload`](/sdks/turbo-upload) does the signing and uploading with no dependencies and nothing else. The Turbo SDK is the right choice everywhere else. ## Organize Your Data Before uploading, learn best practices for structuring and tagging your data for optimal retrieval and organization. ## Additional Resources # Manifests (/build/upload/manifests) Manifests enable friendly-path-name routing for data on Arweave, greatly improving the programmability of data relationships. Instead of accessing data with complex transaction IDs, manifests allow you to organize files with readable paths and relative links. ## What are Manifests? Manifests, also known as "Path Manifests" or "Arweave Manifests," are JSON objects that connect various Arweave data items and define relational paths for easy navigation. A common use case is permanently hosting websites on Arweave by linking all necessary files together. ### The Problem Manifests Solve Without manifests, accessing data on Arweave looks like this: ``` http:///cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI (txID of a website's index.html) http:///3zFsd7bkCAUtXUKBQ4XiPiQvpLVKfZ6kiLNt2XVSfoV (txID of its js/style.css) http:///or0_fRYFcQYWh-QsozygI5Zoamw_fUsYu2w8_X1RkYZ (txID of its assets/img/logo.png) ``` With manifests, the same data becomes: ``` http:/// (resolves to the txID of index.html) http:////js/style.css http:////assets/img/logo.png ``` ## Manifest Structure Manifests are JSON objects that define how data items are connected and accessed through friendly paths. ### Sample Manifest ```json { "manifest": "arweave/paths", "version": "0.2.0", "index": { "path": "index.html" }, "fallback": { "id": "iXo3LSfVKVtXUKBzfZ4d7bkCAp6kiLNt2XVUFsPiQvQ" }, "paths": { "index.html": { "id": "cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI" }, "404.html": { "id": "iXo3LSfVKVtXUKBzfZ4d7bkCAp6kiLNt2XVUFsPiQvQ" }, "js/style.css": { "id": "3zFsd7bkCAUtXUKBQ4XiPiQvpLVKfZ6kiLNt2XVSfoV" }, "css/style.css": { "id": "sPiQvpAUXLVK3zF6iXSfo7bkCVQkiLNt24dVtXUKBfZ" }, "css/mobile.css": { "id": "fZ4d7bkCAUiXSfo3zFsPiQvpLVKVtXUKB6kiLNt2XVQ" }, "assets/img/logo.png": { "id": "or0_fRYFcQYWh-QsozygI5Zoamw_fUsYu2w8_X1RkYZ" }, "assets/img/icon.png": { "id": "0543SMRGYuGKTaqLzmpOyK4AxAB96Fra2guHzYxjRGo" } } } ``` ### How it Works A resolver, typically an ar.io gateway, resolves URLs requesting content based on a manifest transaction ID to the corresponding path key in the `paths` object. The URL schema for this type of request is `https:////`. ### Example Usage Assume the manifest above is uploaded to Arweave with the transaction ID `UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk`. The below table shows https requests to the ar.io gateway `turbo-gateway.com`: | Request Path | Manifest Path | Data served from txID | | ---------------------------------------------------------------------------- | ------------- | ------------------------------------------- | | https://turbo-gateway.com/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk | index | cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI | | https://turbo-gateway.com/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/index.html | index.html | cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI | | https://turbo-gateway.com/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/js/style.css | js/style.css | 3zFsd7bkCAUtXUKBQ4XiPiQvpLVKfZ6kiLNt2XVSfoV | | https://turbo-gateway.com/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/foobar | fallback | iXo3LSfVKVtXUKBzfZ4d7bkCAp6kiLNt2XVUFsPiQvQ | ## Creating Manifests with Turbo Turbo makes it easy to create manifests automatically when uploading folders, or you can create custom manifests manually. ### Folder Upload with Manifest ```js const folderResult = await turbo.uploadFolder({ folderPath: "./my-website", dataItemOpts: { tags: [ { name: "Bundle-Format", value: "binary" }, { name: "Bundle-Version", value: "2.0.0" }, { name: "App-Name", value: "Website" }, ], }, }); console.log("Folder Upload ID:", folderResult.id); console.log("Manifest ID:", folderResult.manifestId); ``` ### Custom Manifest Creation ```js const manifest = { manifest: "arweave/paths", version: "0.2.0", index: { path: "index.html", }, fallback: { id: "fallback-tx-id", }, paths: { "index.html": { id: "abc123...def789", }, "css/style.css": { id: "def456...ghi012", }, }, }; const manifestResult = await turbo.upload({ data: JSON.stringify(manifest), dataItemOpts: { tags: [ { name: "Content-Type", value: "application/x.arweave-manifest+json" }, { name: "App-Name", value: "CustomManifest" }, ], }, }); ``` ## Manifest Specifications ### Required Transaction Tags Manifests must be uploaded with specific tags so that ar.io gateways can recognize and properly resolve them: ```json { "name": "Content-Type", "value": "application/x.arweave-manifest+json" } ``` **Important:** This tag must be attached to the upload transaction, NOT placed inside the JSON object. Failure to provide this tag will result in resolvers not recognizing the manifest. ### Required JSON Attributes #### manifest ```json "manifest": "arweave/paths" ``` Must have the value `arweave/paths` for gateways to resolve the manifest. #### version ```json "version": "0.2.0" ``` Defines the version of manifest schema being used. #### index ```json "index": { "path": "index.html" } ``` or ```json "index": { "id": "cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI" } ``` Defines the base or 'starting' data item. Accepts either `path` (key in paths object) or `id` (specific transaction ID). If both are defined, `id` overrides `path`. #### fallback ```json "fallback": { "id": "iXo3LSfVKVtXUKBzfZ4d7bkCAp6kiLNt2XVUFsPiQvQ" } ``` Defines a fallback data item for when requested paths don't exist (like a 404 page). #### paths ```json "paths": { "index.html": { "id": "cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI" }, "css/style.css": { "id": "3zFsd7bkCAUtXUKBQ4XiPiQvpLVKfZ6kiLNt2XVSfoV" } } ``` Defines the URL paths that a manifest can resolve to. Each path maps to a specific Arweave transaction ID. ## Relative Path Routing Ar.io gateways support relative path routing, making it easy to develop and maintain websites hosted on Arweave. Instead of using fully qualified URLs: ```html ``` You can use relative paths: ```html ``` This makes HTML more readable and ensures links remain valid even if the hosting domain changes. ## Best Practices ### File Organization - Use descriptive file paths - Organize files in logical folders - Keep manifest files small - Use consistent naming conventions ### Performance Considerations - Minimize manifest size - Use relative paths - Avoid deep nesting - Consider file size limits ## Next Steps } > Capture timestamped upload attestations for chronology and audits. } > Secure your sensitive data with encryption. } > Advanced file organization with ArFS. } > Complete upload guide with Turbo. # Receipts (/build/upload/receipts) Turbo upload receipts provide a durable record of what was uploaded, who uploaded it, and what upload cost was assessed in Turbo Credits (`winc`). They are useful for provenance, compliance, incident response, and operational debugging. ## What Are Turbo Receipts? A Turbo receipt is a signed upload attestation returned by Turbo upload flows when a data item is successfully accepted and processed. In practice, receipts help you: - Link app-level records to immutable data item IDs - Track upload ownership and storage cost (`winc`) - Keep machine-readable proof metadata for investigations and audits Receipts are proof of Turbo upload acceptance and signed receipt metadata for a specific upload event. They are not a full substitute for your own retrieval checks, gateway checks, or finality policies. ## Why Time and Ordering Matter Receipt timestamps are especially valuable because they let you prove when upload events occurred and how related uploads were ordered. - A receipt `timestamp` supports evidence of event time in incident and audit workflows. - Ordered receipts create chronology across related data items (for example, original item, revision, and derived artifacts). - Chronology improves chain-of-custody reconstruction and post-incident analysis. For production systems, preserving upload order can be as important as preserving data IDs. ## When Receipts Are Created Turbo receipts are created in three common contexts: 1. Standard SDK uploads: `turbo.upload()` and `turbo.uploadFile()` return upload result payloads that include receipt metadata such as `id`, `owner`, `winc`, `dataCaches`, and `fastFinalityIndexes`. 2. Multipart uploads: for larger uploads that use chunking, Turbo finalizes the upload and returns a finalized receipt payload when multipart status reaches `FINALIZED`. 3. x402 uploads: Turbo still returns an upload receipt, and x402 tooling can also provide a separate payment settlement receipt. Regardless of flow, store receipt timestamps and your own sequence metadata so ordered events can be reconstructed later. Turbo does not store and retrieve your receipts for you. If you need receipts later, persist the exact payload returned by Turbo in your own storage. You can also optionally store receipt records on Arweave for long-term archival. See [Capturing Receipts in Turbo SDK](#capturing-receipts-in-turbo-sdk) for an implementation example. ## Receipt Anatomy | Field | Meaning | Notes | | ----- | ------- | ----- | | `id` | Data item transaction ID | Primary key to join with your app records | | `owner` | Normalized address that signed/owns the data item | Useful for audit and policy checks | | `winc` | Upload cost measured in Turbo Credits | Useful for billing and reporting. See [Paying for Uploads](/build/upload/turbo-credits) for unit and payment context | | `dataCaches` | Caches that accepted the data item | Operational visibility for upload path | | `fastFinalityIndexes` | Fast finality indexes that accepted the data item | Useful for observability | | `timestamp` | Receipt creation time in milliseconds | Core field for chronology, ordering, and evidence of event time | | `version` | Receipt schema/version identifier | Determines how verification inputs should be interpreted | | `deadlineHeight` | Deadline block height recorded in receipt payload | Useful as additional upload context | | `public` | Public key that signed the receipt | Needed to verify signature validity | | `signature` | Base64URL receipt signature | Required to verify receipt authenticity | Some fields vary by upload path and service response shape. Do not assume every SDK return object includes every signed receipt field in all flows. ## Capturing Receipts in Turbo SDK Capture the upload response as your receipt record and persist it alongside your own object IDs. Below is a simple example, in practice you might chose to store this alongside your existing logs in S3, Sentry or similar: ```typescript const turbo = TurboFactory.authenticated({ privateKey }); const receipt = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream("./report.json"), fileSizeFactory: () => fs.statSync("./report.json").size, dataItemOpts: { tags: [ { name: "Content-Type", value: "application/json" }, { name: "App-Name", value: "AnalyticsPipeline-v1.0" }, ], }, }); await fetch("https://api.yourdomain.com/upload-receipts", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ appObjectId: "report-2026-02-16", capturedAt: new Date().toISOString(), receipt, }), }); console.log("Stored Turbo receipt:", { id: receipt.id, timestamp: receipt.timestamp, owner: receipt.owner, winc: receipt.winc, }); ``` Turbo CLI upload commands print JSON output that can be captured as a receipt record in scripts and CI pipelines. ## Why Receipts Matter Across ar.io Use Cases | ar.io Use Case | Receipt Value | | -------------- | ------------- | | [File Storage](https://ar.io/use-cases/file-storage/) | Map internal file objects to immutable upload IDs and timestamped upload events | | [Websites & Apps](https://ar.io/use-cases/sites-and-apps/) | Track deployment artifacts and ordering of publish history over time | | [Permanent Assets](https://ar.io/use-cases/digital-assets/) | Prove that specific asset versions were accepted and in which sequence they were released | | [Media Provenance](https://ar.io/use-cases/durable-media/) | Establish chain-of-custody metadata for original and derivative media with event ordering | | [Verifiable AI Data](https://ar.io/use-cases/verifiable-ai/) | Attach signed upload evidence and chronology to datasets, prompts, and model outputs | | [Oracles & Onchain Data](https://ar.io/use-cases/oracles-and-onchain-data/) | Bind compute inputs/outputs to durable upload receipts for reproducibility and timeline checks | | [Institutional Archiving](https://ar.io/use-cases/institutional-archiving/) | Maintain publish-time evidence for disclosures, reports, and compliance records | ## Deep Dive: Media Provenance Receipts strengthen provenance workflows by giving each media upload a signed event record with a timestamp and ordering context. - Preserve the receipt for original media ingestion - Store receipts for edited or transformed derivatives - Link parent and derivative records in your metadata model - Use receipt IDs, timestamps, and signed fields as part of publishing audit trails This gives teams a clearer chain-of-custody model for authenticity claims, moderation workflows, and external verification requests. ## Deep Dive: Verifiable AI Data AI pipelines benefit from receipts because reproducibility depends on stable, attributable inputs and outputs over time. - Store receipts for datasets, prompts, and generated artifacts - Link receipts to model versions, run IDs, and evaluation jobs - Preserve ordered receipt timelines to reconstruct dataset-to-output lineage - Use receipts as evidence in regulated or policy-bound AI workflows This makes lineage more transparent and reduces ambiguity when debugging model behavior or validating published results. ## Verifying Receipts Later Delayed verification is valuable for audits, disputes, compliance reviews, and reproducibility checks that happen long after initial upload. ### Verification Checklist 1. Load the stored receipt payload. 2. Validate required fields (`id`, `version`, `public`, `signature`, and contextual fields you depend on). 3. Verify the signature against the public key using receipt-version-specific hashing/signing rules. 4. Confirm data item status and availability via Turbo or gateway status endpoints. 5. Compare verified receipt data with your internal upload metadata and flag mismatches. Example verification workflow: ```typescript type StoredReceipt = { id: string; version?: string; public?: string; signature?: string; timestamp?: number; owner?: string; }; async function verifyStoredReceipt(receipt: StoredReceipt) { if (!receipt.id || !receipt.version || !receipt.public || !receipt.signature) { throw new Error("Receipt is missing required verification fields."); } // Implement this in your backend with version-aware receipt rules. const signatureValid = await verifyReceiptSignatureForVersion(receipt); if (!signatureValid) { throw new Error(`Invalid receipt signature for ${receipt.id}`); } const statusResponse = await fetch( `https://upload.ardrive.io/v1/tx/${receipt.id}/status`, ); if (!statusResponse.ok) { throw new Error(`Unable to retrieve status for ${receipt.id}`); } const status = await statusResponse.json(); return { id: receipt.id, signatureValid, status, }; } ``` Verification inputs are receipt-version dependent. Do not assume every response field is signature-bound in every version, and do not assume a high-level SDK helper exists unless it is explicitly documented for your target version. ## Best Practices - Persist raw receipt payloads server-side and keep them immutable - Store receipts with your own object IDs, pipeline IDs, and environment metadata - Index by data item `id` for fast traceability across systems - Preserve upload chronology (for example sequence numbers, parent/child links, and timestamps) - Capture both upload receipts and payment receipts when using x402 - Re-check status/finality in stricter workflows using service or gateway checks Treat a receipt as proof of Turbo upload acceptance and signing for that event, not as a blanket guarantee for every downstream retrieval state. ## Next Steps }> Connect receipt evidence to structured path-based content organization. }> Protect sensitive content before uploading to permanent storage. } > Add just-in-time payment flows to your upload pipeline. } > Understand Turbo Credits, top ups, and funding workflows. # Tagging (/build/upload/tagging) Tags are key-value pairs that provide metadata about your uploaded data on Arweave. They enable discoverability, proper content serving, and integration with various protocols. ## Essential Tags Every upload should include these tags: - **Content-Type**: Required - tells gateways how to serve your data - **App-Name**: Best practice - identifies your application for discoverability ```js const result = await turbo.upload({ data: fileData, dataItemOpts: { tags: [ { name: "Content-Type", value: "image/jpeg" }, { name: "App-Name", value: "MyApp-v1.0" }, { name: "Title", value: "My Image" }, ], }, }); ``` ## Common Tag Types ### Content Types - `image/jpeg`, `image/png` - Images - `application/json` - JSON data - `text/html` - HTML pages - `video/mp4` - Videos - `application/pdf` - Documents ### App-Specific Tags - `App-Name` - Your application identifier (e.g., "MyApp-v1.0", "PhotoGallery-2024") - `Title` - Human-readable title - `Description` - Content description - `Author` - Content creator - `Version` - Application version ### Protocol Tags - `License` - Universal Data License (UDL) transaction ID - `License-Fee` - Fee for UDL licensing **UDL Integration:** Learn about the [Universal Data License](https://mirror.xyz/0x64eA438bd2784F2C52a9095Ec0F6158f847182d9/AjNBmiD4A4Sw-ouV9YtCO6RCq0uXXcGwVJMB5cdfbhE) for monetizing your data. ## Advanced Tagging ### Folder Uploads ```js const folderResult = await turbo.uploadFolder({ folderPath: "./my-website", dataItemOpts: { tags: [ { name: "Bundle-Format", value: "binary" }, { name: "Bundle-Version", value: "2.0.0" }, { name: "App-Name", value: "MyWebsite-v2.1" }, { name: "Version", value: "2.1.0" }, ], }, }); ``` ### Licensed Content ```js const licensedTags = [ { name: "Content-Type", value: "image/jpeg" }, { name: "App-Name", value: "ArtGallery-v3.2" }, { name: "Version", value: "3.2.1" }, { name: "License", value: "udl-tx-id-here" }, { name: "License-Fee", value: "1000000" }, // Fee in Winston ]; ``` ## App-Name Best Practices ### Naming Convention Use descriptive, versioned App-Name values for better organization: - **Include version**: `MyApp-v1.0`, `PhotoGallery-2024` - **Be specific**: `EcommerceStore-v2.1` instead of just `Store` - **Use consistent format**: `ProjectName-vMajor.Minor` - **Include year for time-based apps**: `YearlyReport-2024` ## Tag Limitations - **4KB total** for bundled data items (Turbo) - **2KB total** for direct L1 uploads - **No maximum number of tags** (limited by total size) - Tag names are case-sensitive - No duplicate tag names allowed **Important:** Total tag size is limited to 4KB (bundled) or 2KB (L1). For larger metadata, store it in the data payload instead. ## Querying Data by Tags Once you've tagged your data, you can use GraphQL to search and filter based on those tags. This enables powerful discovery and retrieval of your stored content. ## Next Steps } > Organize files with manifests for better structure. }> Secure your sensitive data with encryption. } > Advanced file organization with ArFS. # Paying for Uploads (/build/upload/turbo-credits) import { Bot, Code, CreditCard, FileText, FolderOpen, Globe, Tag, Upload, Zap, } from "lucide-react"; Data storage on Arweave is paid for in Arweave's native [AR](https://www.coingecko.com/en/coins/arweave) token, divisible to the "Winston" (10¹² per AR). Turbo makes paying for data storage on Arweave faster and even simpler via Turbo Credits. **No account, no subscription.** Credits are bought up front, by any wallet, and spent per upload; nothing renews. An agent or a script can fund itself and upload without a human opening an account, which is what the [just-in-time](#just-in-time-payments) and [x402](/build/upload/x402-uploading-to-turbo) paths are for. ## What are Turbo Credits? Turbo Credits are the payment medium used by Turbo's upload service, providing a 1:1 representation of upload power on the Arweave network, and enabling precise pricing for uploads of any size. Unlike traditional crypto tokens, Turbo Credits cannot be traded or exchanged on secondary markets - they exist solely for uploading data to Arweave. However, credit holders can [share their credits](#credit-sharing) with other wallets, enabling organizational and collaborative use cases. **Important:** Turbo Credits are non-refundable and cannot be withdrawn or exchanged for other cryptocurrencies. ## How to Purchase Credits There are two important choices to make when purchasing credits: 1. Pay in either fiat via Stripe or via crypto tokens on a supported network 2. The address of a [supported crypto wallet](#supported-payment-wallets) type to make the primary holder of the credits When paying in fiat or crypto, the destination wallet address can be one from any [supported crypto wallet](#supported-payment-wallets). It is optional, however, when paying in crypto; the payer wallet is simply used as the default choice in its absence. ### Payment Methods Add credits to your wallet and they will be deducted automatically on each upload. For faster payment networks, it's possible to pay [just-in-time](#just-in-time-payments) with each upload. #### Fiat Payments Top up any address of a [supported wallet type](#supported-wallet-types) by paying with a credit or debit card via Stripe. #### Crypto Tokens Top up any address of a [supported wallet type](#supported-wallet-types) by paying in any of these supported crypto tokens: | Network | Tokens | | ----------- | -------------- | | Arweave | [AR](https://www.coingecko.com/en/coins/arweave) | | Solana | [ARIO](https://www.coingecko.com/en/coins/ar-io-network), [SOL](https://www.coingecko.com/en/coins/solana), [USDC](https://solscan.io/token/EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v) | | Base | [ETH](https://www.coingecko.com/en/coins/ethereum), [USDC](https://basescan.org/token/0x833589fcd6edb6e08f4c7c32d4f71b54bda02913) | | Ethereum | [ETH](https://www.coingecko.com/en/coins/ethereum), [USDC](https://etherscan.io/token/0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48) | | Polygon | [POL](https://www.coingecko.com/en/coins/polygon) | ARIO has a lower cap per top up in the ar.io Console than other tokens. When you sign in with a Solana wallet, Buy Credits offers ARIO (from Console 4.10.0), up to $100 per top up at the day's ARIO rate (from 4.11.0). In the Console's name checkout, ARIO also pays for an ArNS name directly, which never touches credits. ### Supported Payment Wallets | Network Type | Signature Types | Supported Wallets | | ------------ | -------------- | ------------------------------------------------------ | | Arweave | RSA | JWK Keyfile, [Wander](https://www.wander.app/), [Beacon](https://getbeaconapp.com/) | | EVM | ECDSA (secp256k1) | JSON Keystore File, [MetaMask](https://metamask.io/), [Rainbow](https://rainbow.me/), [Brave](https://brave.com/wallet/), [WalletConnect](https://walletconnect.com/), [Privy](https://www.privy.io/) | | Solana | ED25519 | JSON Keypair File, [Phantom](https://phantom.com/), [Solflare](https://www.solflare.com/), [MetaMask](https://metamask.io/) (its Solana account) | ### Purchasing Interfaces #### Console App Visit [console.ar.io/topup](https://console.ar.io/topup) to purchase credits with fiat or crypto using a connected wallet. #### ArDrive App Visit [app.ardrive.io](https://app.ardrive.io) to purchase credits with fiat for Arweave wallets. #### Turbo Typescript SDK ```js const turbo = TurboFactory.authenticated({ signer, token: "base-usdc" }); // Top up with crypto const result = await turbo.topUpWithTokens({ tokenAmount: USDCToTokenAmount(1), destinationAddress: "optional-different-wallet-address", }); // Check balance const { winc } = await turbo.getBalance(); // Check remaining free-tier allowance (bytes). null = unlimited, 0 = free tier off. const { bytesRemaining } = await turbo.getFreeStatus(); // Review your own top-up history (crypto + fiat), newest first, paginated const { payments, hasMore, cursor } = await turbo.getPaymentHistory({ limit: 25 }); ``` See the [Turbo SDK documentation](/sdks/turbo-sdk) and [GitHub repository](https://github.com/ardriveapp/turbo-sdk) for complete examples. #### Turbo CLI ```bash # Top up with crypto turbo top-up --token base-usdc --amount 1 # Check balance turbo balance # Check remaining free-tier upload allowance turbo free-status # Review your own top-up (payment) history turbo payment-history --limit 25 ``` See the [Turbo Node.js CLI documentation](https://github.com/ardriveapp/turbo-sdk/?tab=readme-ov-file#cli) for installation and usage. #### HTTP API Direct API integration is available at [payment.ardrive.io/api-docs](https://payment.ardrive.io/api-docs) for custom implementations. #### Payment Interfaces By Token | Payment Method | Turbo SDK | Turbo CLI | Turbo API | Console App | ArDrive App | x402 | | ---------------------------- | --------- | --------- | --------- | --------- | ----------- | ---- | | **Fiat (credit/debit card)** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | | **AR** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **ETH (L1)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **ETH (Base)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **SOL** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **POL** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **ARIO (Solana)** | ✅ | ✅ | ✅ | ✅ (Solana sign-in, up to $100 per top up) | ❌ | ❌ | | **USDC (L1)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **USDC (Base)** | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | | **USDC (Solana)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ## Just-in-Time Payments Just-in-time (JIT) payments allow you to pay for uploads at the moment of upload rather than pre-funding a credit balance. This is ideal for agentic workflows, automated pipelines, and systems that prefer pay-per-upload over balance management. JIT payments are available via the Turbo SDK, CLI, or direct HTTP API integration for tokens with sufficiently fast confirmation times: | Token / Currency | Network | JIT Supported | | ---------------------- | -------- | ------------- | | ARIO | Solana | ✅ | | SOL | Solana | ✅ | | USDC | Solana | ✅ | | ETH | Base | ✅ | | USDC | Base | ✅ | | POL | Polygon | ✅ | | AR | Arweave | ❌ | | ETH | L1 | ❌ | | Fiat | Stripe | ❌ | ### x402 Protocol [x402](https://www.x402.org/) is a protocolized approach to JIT payments that leverages the HTTP 402 "Payment Required" status code. Turbo currently supports x402 payments made in **USDC on Base**. The following Turbo features can utilize x402: **Note:** x402 facilitator fees may be applicable for each x402 payment. - **Uploads:** pay for storage at the time of upload - **Top up:** useful when x402 is the preferred payment medium, but facilitator fees on individual uploads are undesirable Example SDK usage: ```js const turbo = TurboFactory.authenticated({ signer, token: "base-usdc" }); const result = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream("./myfile.jpg"), fileSizeFactory: () => fs.statSync("./myfile.jpg").size, fundingMode: new X402Funding({ maxMUSDCAmount: 1_000_000 }), // Max 1 USDC }); ``` See the [Turbo SDK documentation](/sdks/turbo-sdk) and [HTTP API docs](https://payment.ardrive.io/api-docs) for complete integration details. ## Credit Sharing Turbo Credits can be shared with other users to fund their uploads while maintaining total control over who may spend them, how many they may spend, and for how long they can do so. This is perfect for organizations, collaborations, and teams. Credits can be shared via the Console App at [console.ar.io/share](https://console.ar.io/share) or programmatically via the [Turbo CLI](https://github.com/ardriveapp/turbo-sdk/?tab=readme-ov-file#cli), [SDK](/sdks/turbo-sdk), or [API](https://payment.ardrive.io/api-docs). ### How Credit Sharing Works - **Authorize Users**: Grant specific wallets access to your credits - **Set Limits**: Control how much each user can spend - **Time Limits**: Set expiration dates for access - **Revoke Anytime**: Regain control of shared credits instantly ### Use Cases - **Organizational Funds**: Central wallet shares credits with employees - **Onboarding**: Give new users free upload power for trials - **Collaboration**: Share credits with project contributors - **Educational Programs**: Provide students with controlled access ## Using Credits to Buy ArNS Names Turbo Credits are not only for uploads: they are also the payment medium for buying, extending and upgrading [ArNS](/learn/arns) names. This draws on the same credit balance as uploads, so no separate top-up is needed if you already hold credits. Buying a name mints the ANT (a Solana asset) that the name resolves to and registers the name against it, in one purchase. **You own that ANT from the moment it exists.** Turbo pays the Solana fees and rent but never takes custody, and your Solana balance can stay at zero throughout. Buying is the one action that needs the owner's signature, once, ever: ```js const turbo = TurboFactory.authenticated({ privateKey: payerKey }); // The Solana wallet that will own the ANT. It signs once and never needs SOL. const owner = solanaOwnerSigner(bs58SolanaSecretKey); // Price it first: quote wincTotal, not winc const price = await turbo.getArNSPriceForName({ intent: "Buy-Name", name: "my-name", type: "lease", years: 1, }); const { antId, messageId } = await turbo.buyArNSName({ name: "my-name", owner, type: "lease", years: 1, }); ``` Requires `@ardrive/turbo-sdk` 2.1.0 or newer. The samples above were type-checked against 2.1.0 from a clean install. Extending, upgrading and adding undernames need no signature at all. See the dedicated [ArNS Names](/sdks/turbo-sdk/the-twelve-sponsored-actions) guide for the full action lifecycle, pricing, record management and transfers. ## Pricing & Fees Three charges, and nothing else: the **storage rate** for the bytes, a **per-item fee** for each item uploaded, and the **infrastructure fee** taken out of each top up. The [ar.io Console](https://console.ar.io/settings) shows all three live, with the same names. ### Infrastructure fee **It is taken out of what you pay, not added on top.** At a 35% fee, for example, a $100 top up buys credits worth $65, rather than costing $135. It covers the costs of managing infrastructure and, for crypto top ups, token liquidity. Spending credits carries no percentage fee: this one applies on the way in. **Turbo sets the fee for each currency, and it can change.** Read it live rather than from these docs: - The [ar.io Console Settings page](https://console.ar.io/settings) shows the fee on card top ups. - Every top up quote from the payment service, `GET /v1/price/:currency/:amount`, lists the fee it applied in its `fees` array. ### Storage rate and per-item fee - The **storage rate** is priced per byte and quoted per GiB. Credits maintain a 1:1 peg of storage purchasing power as Arweave's data storage rates and token price fluctuate. - The **per-item fee** is a flat charge for each data item, one per file, on top of the byte cost, covering per-item processing overhead. It is pegged in dollars at **$0.00002 per item**, i.e. **$0.02 per 1,000**, valued at AR's market price. The credits figure moves as AR does, so read it live from `perDataItemFeeWinc` in [`/v1/rates`](https://payment.ardrive.io/v1/rates), which is what the Console displays, and what every cost estimate there adds per file. - **Small uploads can be free.** Items up to **105 KiB** are eligible for the free tier, subject to a **10 MiB** lifetime allowance tracked per-wallet **and** per-subnet. An upload must fit under both, and the allowance never resets. Check a wallet's remaining free allowance with `turbo.getFreeStatus()` (below). ## Getting Started Ready to start using Turbo Credits? Choose your path: } > Buy credits instantly with credit cards or crypto } > Learn how to upload data with your new credits }> Integrate credit sharing and advanced features } > Use credits to buy, extend, and upgrade ArNS names } > Pay-per-upload for automated workflows ## Next Steps } > Complete upload guide with Turbo. }> Organize with metadata and tags. } > Create folder structures with manifests. # Upload and Verify (/build/upload/upload-and-verify) This is the smallest complete thing you can do on ar.io: put one file on the network, get it back from a gateway, and prove the bytes are the same. It is written so a script or an agent can follow it without a person: every step says what it costs, what it prints, and what to do when it does not. ## Before you start - **Node.js 18+** and `npm install @ardrive/turbo-sdk bs58`. - **A wallet.** The example uses a Solana keypair in the Solana CLI format, a JSON array of 64 numbers. Ethereum keys and Arweave wallets work too; see [Signer setup](/sdks/turbo-sdk). - **Nothing else for a small file.** Items within the free tier upload with no credits and no payment call. Larger items need [credits](/build/upload/turbo-credits). Decide where you are running. Mainnet is permanent and costs real value above the free tier; the sandbox is free and purges data after about three days. | | Mainnet | [Testnet sandbox](/build/testnet) | |---|---|---| | Upload service | `https://upload.ardrive.io` (the SDK default) | `https://upload.services.ar-io.dev` | | Gateway to read from | `https://turbo-gateway.com`, or any ar.io gateway | `https://ar-io.dev` | | Free per item | 105 KiB | 5 MiB | | Free per wallet, lifetime | 10 MiB | 100 MiB | | Wallet | Solana mainnet | Solana devnet | `GET /info` on either upload service reports the current limits as `freeTier`. ## What it costs Ask before you upload. `getUploadCosts` returns the price in Turbo Credits (`winc`) for a size, and `getFreeStatus` returns how many free bytes the wallet has left. ```ts const [{ winc: price }] = await turbo.getUploadCosts({ bytes: [size] }); const { bytesRemaining } = await turbo.getFreeStatus(); ``` - `price` is the list price for that size. It does not apply the free tier, so a small file still shows a price here. - `bytesRemaining` at or above `size`, and `size` within the per-item limit: the upload is free, and the receipt says `winc: "0"`. - Otherwise the upload costs `price`, deducted from the wallet's credits. You pay once; nothing renews. - `bytesRemaining` is `null` for a wallet with an unlimited allowance and `0` where the free tier is off. ## Step 1: Upload ```ts const receipt = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream(path), fileSizeFactory: () => size, dataItemOpts: { tags: [{ name: "Content-Type", value: "text/markdown" }] }, }); ``` The response is the receipt. Expect this shape: ```json { "id": "tCdA4ePIjMvKSapeQfe3zBUhODPKjXYERqjvEoOM3f8", "owner": "HbmimaUBGYtimZiYbeXLXQS8pKD9wBTMfMTaYmfsyBMj", "winc": "0", "timestamp": 1791483824036, "dataCaches": ["ar-io.dev"], "fastFinalityIndexes": ["ar-io.dev"], "version": "0.2.0", "deadlineHeight": 1234567, "public": "", "signature": "" } ``` `id` is the data item's permanent identifier. `winc` is what you were charged. Save the whole object: Turbo does not keep receipts for you, and [Receipts](/build/upload/receipts) explains what each field proves and how to verify the signature later. Set `Content-Type` yourself. Without it, a gateway serves the bytes as `application/octet-stream`. ## Step 2: Fetch the bytes back ```http GET https://turbo-gateway.com/raw/ ``` Use the `/raw/` path. A plain `/` resolves manifests, so for a manifest it serves the index file rather than the manifest's own bytes. `/raw/` always returns the data item's bytes as stored. Expect `200` with these headers: | Header | Meaning | |---|---| | `x-ar-io-data-id` | The id the gateway served, which must equal the one you asked for | | `content-digest` | `sha-256=::`, the hash of the bytes the gateway served | | `x-ar-io-digest` | The same hash, base64url | | `x-ar-io-verified` | `true` once the gateway has verified the item against the chain; `false` is normal for a fresh upload | | `x-ar-io-stable` | `true` once the item is in a block old enough to be considered final | A gateway that has not seen the item yet returns `404`. Turbo's own gateway normally serves a new upload at once; other gateways serve it once they have indexed it. Retry a `404` with a pause between attempts, up to a limit you choose, and treat anything other than `200` or `404` as a stop. ## Step 3: Compare Hash the local file and the fetched bytes with SHA-256 and compare. That comparison is the proof: the bytes this gateway serves under the id are your file. `x-ar-io-digest` is the gateway's own hash of the item it holds. Comparing it as well catches a gateway that sent something other than what it indexed, but it arrives in the same response as the bytes, so it is a consistency check, not independent evidence. ```ts const want = crypto.createHash("sha256").update(fs.readFileSync(path)).digest(); const got = crypto.createHash("sha256").update(Buffer.from(await response.arrayBuffer())).digest(); const match = got.equals(want) && response.headers.get("x-ar-io-digest") === want.toString("base64url"); ``` Both must be true. A gateway then serves your file, byte for byte, under this id. Whether the gateway has also verified the item against the chain is what `x-ar-io-verified` reports, and it lags the upload. ## When to stop | You see | It means | Do | |---|---|---| | `402` with the header `X-Free-Tier-Exhausted: true` (the body is an x402 payment offer, or `code: "FREE_TIER_EXHAUSTED"`) | The free bytes for this wallet or this IP range are used up, or the item is over the free size | Stop. [Top up credits](/build/upload/turbo-credits), or pay just in time, then retry once. If you uploaded with `paidBy` on someone else's credits, stop and report instead, and do not pay the x402 offer in the body: see [Give an Agent a Budget](/build/upload/give-an-agent-a-budget#when-to-stop) | | `402` naming a network and USDC, without the `X-Free-Tier-Exhausted` header (raw HTTP only: the SDK's error carries no headers, so an SDK error on an upload with `paidBy` set belongs to the row above) | You posted to an x402 endpoint, which only takes USDC | Wrong path. Use `uploadFile` or `/v1/tx` as above | | `413` | The item is over the service's maximum item size (10 MiB on the sandbox) | Stop. Split the data into smaller items and tie them together with a [manifest](/build/upload/manifests) | | `404` from the gateway after your retry limit | The gateway has not indexed the item | Stop and report the `id`. Try `turbo-gateway.com`, which serves Turbo uploads first | | Hashes differ | The bytes served are not the file | Stop. Do not retry blindly; report both hashes and the `id` | | `x-ar-io-verified: false` | The gateway has not verified the item yet | Not a failure. Verification follows indexing | ## Full script Runs against mainnet by default and against the sandbox with `SANDBOX=1`. It stops, with a reason, at every condition in the table above. ```js // Upload one file through Turbo, fetch it back from an ar.io gateway, prove the bytes match. // Usage: KEYFILE=./solana-keypair.json node upload-verify.mjs ./report.md // Set SANDBOX=1 to run against the testnet sandbox instead of mainnet. const sandbox = process.env.SANDBOX === "1"; const gateway = sandbox ? "https://ar-io.dev" : "https://turbo-gateway.com"; const path = process.argv[2]; if (!path) throw new Error("give the file to upload as the first argument"); // 1. Signer. A Solana CLI keypair file is a JSON array of 64 bytes. const secretKey = Uint8Array.from(JSON.parse(fs.readFileSync(process.env.KEYFILE, "utf8"))); const signer = new HexSolanaSigner(bs58.encode(secretKey)); const turbo = TurboFactory.authenticated({ signer, token: "solana", ...(sandbox && { gatewayUrl: "https://api.devnet.solana.com", uploadServiceConfig: { url: "https://upload.services.ar-io.dev" }, paymentServiceConfig: { url: "https://payment.services.ar-io.dev" }, }), }); // 2. Cost before upload. winc "0" means the item fits the free tier. const size = fs.statSync(path).size; const [{ winc: price }] = await turbo.getUploadCosts({ bytes: [size] }); const { bytesRemaining } = await turbo.getFreeStatus(); const free = bytesRemaining === null || bytesRemaining >= size; console.log(JSON.stringify({ size, price, bytesRemaining, free })); if (!free && BigInt(price) > BigInt((await turbo.getBalance()).winc)) { throw new Error("stop: not free and the wallet's credits cannot cover it; top up first"); } // 3. Upload. The response is the receipt; keep it. const receipt = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream(path), fileSizeFactory: () => size, dataItemOpts: { tags: [{ name: "Content-Type", value: "text/markdown" }] }, }); fs.writeFileSync(`${path}.receipt.json`, JSON.stringify(receipt, null, 2)); console.log(JSON.stringify({ id: receipt.id, winc: receipt.winc, timestamp: receipt.timestamp })); // 4. Fetch the raw bytes back and compare hashes. A gateway that has not seen the item yet returns 404. const want = crypto.createHash("sha256").update(fs.readFileSync(path)).digest(); let response; for (let attempt = 1; attempt setTimeout(r, 10_000)); } if (!response.ok) throw new Error("stop: gateway still returns 404 after two minutes; report the id"); const got = crypto.createHash("sha256").update(Buffer.from(await response.arrayBuffer())).digest(); const servedDigest = response.headers.get("x-ar-io-digest"); // base64url of the sha-256 the gateway served const match = got.equals(want) && servedDigest === want.toString("base64url"); console.log(JSON.stringify({ fetched: `${gateway}/raw/${receipt.id}`, sha256Match: match, verified: response.headers.get("x-ar-io-verified"), stable: response.headers.get("x-ar-io-stable"), })); if (!match) throw new Error("stop: the bytes the gateway served differ from the file; do not retry blindly"); ``` Output for a 27 KB file on the sandbox, as run on 2026-10-08, three lines of JSON. `price` is the list price; `winc` on the receipt is what was charged: ```json {"size":27018,"price":"353098377","bytesRemaining":104857600,"free":true} {"id":"tCdA4ePIjMvKSapeQfe3zBUhODPKjXYERqjvEoOM3f8","winc":"0","timestamp":1791483824036} {"fetched":"https://ar-io.dev/raw/tCdA4ePIjMvKSapeQfe3zBUhODPKjXYERqjvEoOM3f8","sha256Match":true,"verified":"false","stable":"false"} ``` ## Next steps - [Receipts](/build/upload/receipts): what the receipt proves and how to verify its signature later. - [Paying for Uploads](/build/upload/turbo-credits): credits, just-in-time payment and giving a second wallet a budget. - [Manifests](/build/upload/manifests): upload a folder so that paths resolve, the step before publishing a site. # x402 Uploading To Turbo (/build/upload/x402-uploading-to-turbo) ## x402 Uploads To Turbo Turbo provides multiple ways to upload data using the x402 Protocol. Users can upload data using signed data items or by leveraging the unsigned raw data API. Tooling is provided within the Turbo SDK and CLI to facilitate uploads using both methods. Integrators can also leverage existing x402 ecosystem tooling to upload data to Turbo with minimal code. **x402 is for paying with USDC on Base.** It is one of three ways to pay for an upload, not a requirement. To upload an item of **105 KiB or less** on mainnet, you need none of them: it is free with any signer through [`upload`](/sdks/turbo-sdk) or a signed data item posted to `/v1/tx`. To pay from a Solana, Ethereum or ARIO wallet, use [credits](/build/upload/turbo-credits). A `402` from an x402 endpoint means that endpoint wants USDC; it does not mean the free tier is used up. ### Signed Data vs Unsigned Raw Data Users can choose to send data via signed data items or by using the unsigned raw data API and allow Turbo to handle signing the data items on their behalf. - `upload.ardrive.io/v1/x402/data-item/signed` - This endpoint accepts signed data items. Users are responsible for signing the ANS-104 data items before sending them to Turbo. This method provides users with full ownership and control over the data being uploaded. - `upload.ardrive.io/v1/x402/data-item/unsigned` - This endpoint accepts raw data payloads. Users send the raw data to Turbo, which then creates and signs the ANS-104 data items on behalf of the user. This method simplifies the upload process by offloading the signing responsibility to Turbo. ### Turbo CLI Signed Data Item Upload: ```sh turbo upload-file -f ../path/to/file --wallet-file ../path/to/evm/wallet.json --x402 --token base-usdc ``` Unsigned Raw Data Upload: ```sh turbo x402-unsigned-upload -f ../path/to/file --wallet-file ../path/to/evm/wallet.json --token base-usdc --tags 'App-Name' 'My-Cool-App' 'My-Custom-Tag' 'My-Custom-Value' ``` ### Turbo SDK Signed Data Item Upload: ```typescript const turbo = TurboFactory.authenticated({ privateKey, token: "base-usdc", }); await turbo.upload({ data: "The contents of my file!", fundingMode: new X402Funding({ maxMUSDCAmount: 1_000_000 }), // Max 1 USDC }); ``` Unsigned Raw Data Upload: ```typescript const turbo = TurboFactory.authenticated({ privateKey, token: "base-usdc", }); await turbo.uploadRawX402Data({ data: myRawData, maxMUSDCAmount: 1_000_000, // Max 1 USDC. Opt out if too expensive }); ``` NOTE: For free uploads under 105 KiB, this method of upload currently does not require a signature and can be used with an unauthenticated client. ```ts // Unsigned free upload of raw data under 105 KiB const turbo = TurboFactory.unauthenticated({ token: "base-usdc" }); await turbo.uploadRawX402Data({ data: myRawData, }); ``` **Receipt context:** x402 integrations can surface both an upload receipt and a payment settlement receipt. For how to capture and store Turbo upload receipts, see [Receipts](/build/upload/receipts#verifying-receipts-later). ### x402 Ecosystem Tooling Using the raw data API, developers can upload data to Turbo with minimal code using existing x402 ecosystem tooling. - x402 Fetch: ```ts // Create x402 signer const signer = privateKeyToAccount( process.env.EVM_PRIVATE_KEY as `0x${string}`, ); // Create x402 client and register schemes const client = new x402Client(); registerExactEvmScheme(client, { signer }); // Wrap fetch with payment handling const fetchWithPayment = wrapFetchWithPayment(fetch, client); const customTags = [ { name: "App-Name", value: "My-Cool-App" }, { name: "My-Custom-Tag", value: "My-Custom-Value" }, ]; // Make request - payment is handled automatically const response = await fetchWithPayment( "https://upload.ardrive.io/v1/x402/data-item/unsigned", { method: "POST", body: readFileSync("../path/to/file"), headers: { "Content-Type": "application/octet-stream", // Set appropriate content type "x-data-item-tags": JSON.stringify(customTags), // Include custom tags as a header }, ); ``` - x402 Axios: ```ts import { x402Client, withPaymentInterceptor, x402HTTPClient, } from "@x402/axios"; // Create signer const signer = privateKeyToAccount( process.env.EVM_PRIVATE_KEY as `0x${string}`, ); // Create x402 client and register schemes const client = new x402Client(); registerExactEvmScheme(client, { signer }); // Create an Axios instance with payment handling const api = withPaymentInterceptor( axios.create({ baseURL: "https://upload.ardrive.io" }), client, ); const customTags = [ { name: "App-Name", value: "My-Cool-App" }, { name: "My-Custom-Tag", value: "My-Custom-Value" }, ]; // Make request - payment is handled automatically const response = await api.post( "/x402/data-item/unsigned", readFileSync("../path/to/file"), { headers: { "Content-Type": "application/octet-stream", // Set appropriate content type "x-data-item-tags": JSON.stringify(customTags), // Include custom tags as a header }, }, ); console.log("Response:", response.data); // Get payment receipt const httpClient = new x402HTTPClient(client); const paymentResponse = httpClient.getPaymentSettleResponse( (name) => response.headers[name.toLowerCase()], ); console.log("Payment settled:", paymentResponse); ``` # Verifiable AI with ar.io (/build/verifiable-ai) import { Database, FileCheck, ShieldCheck, } from "lucide-react"; ## Verifiable MLflow lifecycle The verifiable AI stack is built around two projects: - [`ar-io-mlflow`](https://github.com/ar-io/ar-io-mlflow): a Python MLflow plugin that signs compact lifecycle proofs and anchors them to Arweave through ar.io Turbo. - [`ar-io-verify`](https://github.com/ar-io/ar-io-verify): an ar.io gateway sidecar that independently verifies Arweave data and can issue operator-signed attestations. This workflow keeps MLflow as the system of record and uses Arweave as a public, permanent witness. The plugin is currently alpha. Its proof format, verification flow, and API surface are stable enough for evaluation, but production deployments should use dedicated wallets, explicit signing keys, monitoring, and ar.io Verify attestations. Verifiable AI builds on ar.io's broader verification model for gateway claims, client-side verification, and network accountability. For the network-level concepts, see [Verification and Accountability](/learn/verification). ## Architecture The lifecycle is: 1. Train a model in MLflow and anchor the run. 2. Register the model and anchor the registry event. 3. Load the model through an integrity-checking wrapper before inference. 4. Anchor prediction proofs without publishing raw inputs or outputs. 5. Verify and attest the Arweave transactions through an ar.io gateway. Each lifecycle event produces a small signed envelope on Arweave. The envelope contains event metadata, a SHA-256 hash of canonical payload bytes, the previous proof hash for that event chain, the public key, and an Ed25519 signature. The canonical payload stays in MLflow as an `ario/payload.json` artifact, or as `ario/predictions//payload.json` for inference. A verifier fetches the envelope from an ar.io gateway, re-hashes the MLflow payload, re-derives the live MLflow record, and verifies the signature. Nothing sensitive needs to be written to Arweave. Source data, model artifacts, inputs, outputs, metrics, and traces remain in your MLflow store. Arweave stores the commitment that lets an auditor prove whether those records changed after anchoring. ## What you will build } /> } /> } /> ## Verification checks All verification paths reduce to the same core checks: 1. **Proof found**: the signed envelope is retrievable from an ar.io gateway for the recorded transaction ID. 2. **Record matches**: MLflow's stored payload re-hashes to the envelope's `payload_hash`, and a separate live MLflow surface re-derives the same canonical bytes. 3. **Signature confirmed**: the envelope signature verifies against the embedded Ed25519 public key. 4. **ar.io attestation**: optional gateway-operator verification that can be signed independently by the operator. This gives teams a tamper-evident record for training, registration, promotion, and inference without making Arweave the place where private AI data lives. # Model and Decision Proofs (/build/verifiable-ai/model-decision-proofs) The model registry is where training provenance becomes runtime protection. `ar-io-mlflow` provides a drop-in MLflow client that anchors registration and promotion events, plus a `VerifiedModel` wrapper that checks model artifact integrity before loading the underlying pyfunc model. ## Prerequisites - Completed [Training Provenance](/build/verifiable-ai/training-provenance) - A model logged to MLflow with `ario_mlflow.anchor()` - MLflow model registry access - Optional `ARIO_MLFLOW_ARIO_VERIFY_URL` for ar.io gateway attestations ## Anchor model registration ### Register with ArioMlflowClient Use `ArioMlflowClient` where you would normally use `mlflow.tracking.MlflowClient`. Registration returns immediately; anchoring runs in a background thread. ```python from ario_mlflow import ArioMlflowClient client = ArioMlflowClient() model_version = client.create_model_version( name="credit-risk-scorer", source=f"runs:/{run_id}/model", ) client.wait_for_anchor( "registration", "credit-risk-scorer", str(model_version.version), timeout=30, ) status = client.anchor_status( "registration", "credit-risk-scorer", str(model_version.version), ) print(status) ``` The status is one of `anchoring`, `anchored`, `signed`, `failed`, or `unknown`. ### Verify the model version ```bash MLFLOW_TRACKING_URI=file:///absolute/path/to/mlruns \ ar-io-mlflow verify model credit-risk-scorer/1 ``` Model verification re-hashes the registered model artifact, checks the registration proof, and confirms that the proof chains back to the training event where possible. ### Audit lineage Use the audit command when you want a registry-oriented view of the proof chain. ```bash MLFLOW_TRACKING_URI=file:///absolute/path/to/mlruns \ ar-io-mlflow audit credit-risk-scorer/1 ``` ## Guard inference with VerifiedModel `VerifiedModel` verifies artifact integrity before loading the model. If the model bytes in MLflow were swapped after registration, the constructor raises `IntegrityError` before user model code can execute. ```python from ario_mlflow import IntegrityError, VerifiedModel try: model = VerifiedModel("models:/credit-risk-scorer/1") except IntegrityError as exc: # Treat this as a security incident in production. raise RuntimeError("Registered model artifact failed verification") from exc result = model.predict([78000, 0.18, 0.22, 72, 745]) print("decision_id:", result.decision_id) print("proof_status:", result.proof_status) print("tx_id:", result.tx_id) ``` Prediction anchoring is asynchronous. The prediction returns as soon as the model has produced an output, and the proof upload completes in the background. When you need the transaction ID before responding to a caller, wait explicitly: ```python result.wait_for_anchor(timeout=10) print(result.tx_id, result.anchor_error) ``` ## Verify an inference proof Each `VerifiedModel.predict()` call writes prediction proof data to MLflow trace tags and `ario/predictions//payload.json`. ```bash MLFLOW_TRACKING_URI=file:///absolute/path/to/mlruns \ ar-io-mlflow verify trace ``` The verification flow checks: 1. The prediction envelope exists on ar.io for the recorded transaction ID. 2. The prediction payload artifact hashes to the envelope's `payload_hash`. 3. The trace tag mirror, `ario.payload_json`, re-derives the same canonical payload. 4. The envelope signature is valid. 5. Optional ar.io Verify attestation passes if configured. ## What gets written On model versions, the client writes tags such as: - `ario.artifact_verified` - `ario.registration_tx` - `ario.promotion_tx` - `ario.arweave_url` On prediction traces, `VerifiedModel` writes tags such as: - `ario.decision_id` - `ario.model_name` - `ario.model_version` - `ario.input_hash` - `ario.output_hash` - `ario.payload_hash` - `ario.proof_status` - `ario.prediction_tx` - `ario.payload_json` Raw inference inputs and outputs are not written to Arweave by the plugin. The proof stores hashes and lineage metadata. Verification proves that the model artifact and recorded decision metadata match what was anchored. It does not prove that the model is accurate, unbiased, or semantically correct for the task. ## Failure behavior - Registration and promotion still succeed if the background anchor fails. Inspect `anchor_status()` and alert on `signed` or `failed`. - `VerifiedModel(...)` fails closed on artifact hash mismatch by raising `IntegrityError`. - `predict()` returns even if proof anchoring later fails. Inspect `result.proof_status` and `result.anchor_error`. - If registration anchoring has not completed when a model is loaded, early predictions may chain to `GENESIS` instead of the model version registration transaction. ## Next steps Continue to [Operator Attestations](/build/verifiable-ai/operator-attestations) to add independent gateway verification, PDFs, attestations, and batch audits. # Operator Attestations (/build/verifiable-ai/operator-attestations) [`ar-io-verify`](https://github.com/ar-io/ar-io-verify) is a verification sidecar for ar.io gateways. It fetches data through a gateway, reconstructs the cryptographic proof, reports the strongest verification level reached, and can sign the result as an operator attestation. For verifiable AI, this gives an independent party a way to verify the envelopes written by `ar-io-mlflow` without trusting the training or inference service. For the broader network model behind gateway claims, client-side checks, and economic accountability, see [Verification and Accountability](/learn/verification). ## Verification levels | Level | Name | What it proves | | --- | --- | --- | | 1 | Existence confirmed | The transaction exists on Arweave and is associated with a confirmed block. | | 2 | Partially verified | The sidecar downloaded the bytes and computed a matching SHA-256 fingerprint. | | 3 | Verified | The sidecar verified the original Arweave or ANS-104 signature against the signed data. | When the operator configures a wallet, the sidecar signs the verification result with RSA-PSS SHA-256. That attestation can be checked later using the operator public key. ## Deploy the sidecar ### Start from the reference repo ```bash git clone https://github.com/ar-io/ar-io-verify.git cd ar-io-verify pnpm install ``` ### Run locally for development In one terminal, run the server: ```bash pnpm run dev ``` In another terminal, run the standalone web UI: ```bash pnpm --filter verify-web run dev ``` ### Run beside an ar.io gateway The Docker deployment expects an existing gateway on the `ar-io-network` Docker network. ```bash cd deploy cp .env.example .env ``` Edit `.env`: ```bash VERIFY_IMAGE=ar-io-verify:local GATEWAY_URL=http://ar-io-node-envoy-1:3000 GATEWAY_HOST=example.com PUBLIC_GATEWAY_URL=https://example.com VERIFY_PORT=4001 NODE_ENV=production WALLET_FILE=/absolute/path/to/operator-wallet.json ``` Start the sidecar and nginx proxy: ```bash bash start.sh ``` ## Connect ar-io-mlflow to ar.io Verify Set the plugin's Verify endpoint to the sidecar base URL exposed by your operator. ```bash ``` After this is configured, `ar-io-mlflow verify run`, `verify model`, and `verify trace` include the ar.io attestation row and write attestation metadata back to MLflow tags when available: - `ario.attestation_level` - `ario.report_url` - `ario.attested_by` - `ario.attested_at` The configured base URL must expose `GET /health` and `POST /api/v1/verify`. If you mount the sidecar under a path prefix, make sure those routes are still available under that same prefix. ## Single-transaction API Submit a transaction ID for verification: ```bash curl -X POST http://localhost:4001/api/v1/verify \ -H "Content-Type: application/json" \ -d '{"txId":""}' ``` Fetch the cached result: ```bash curl http://localhost:4001/api/v1/verify/ ``` Download the PDF certificate: ```bash curl -L http://localhost:4001/api/v1/verify//pdf \ -o verification.pdf ``` Fetch the programmatic attestation: ```bash curl http://localhost:4001/api/v1/verify//attestation ``` The same API is used by the standalone `/verify/` UI and by ar.io Console integrations. ## Batch verification jobs Use jobs for periodic audits, upload reconciliation, or verifier workflows that need to check many transaction IDs. ```bash curl -X POST http://localhost:4001/api/v1/jobs \ -H "Content-Type: application/json" \ -H "X-Tenant-Id: audit-team" \ -H "Idempotency-Key: run-2026-05-14" \ -d '{"txIds":["",""]}' ``` Check status: ```bash curl -H "X-Tenant-Id: audit-team" \ http://localhost:4001/api/v1/jobs/ ``` Fetch the signed verification bundle: ```bash curl -H "X-Tenant-Id: audit-team" \ http://localhost:4001/api/v1/jobs//report ``` The bundle is canonical JSON. To verify it offline, re-serialize the bundle without `signature` and `payloadHash`, compute SHA-256, compare that to `payloadHash`, then verify the RSA-PSS SHA-256 signature against `operatorPublicKey`. ## Operations The sidecar exposes: - `GET /health` for liveness - `GET /ready` for DB and gateway readiness - `GET /metrics` for Prometheus metrics - `GET /api-docs/` for Swagger UI Production deployments should restrict `/metrics` and administrative routes at the proxy layer, inject `X-Tenant-Id` from authenticated infrastructure for batch jobs, and treat the attestation wallet as a production signing secret. ## Production checklist - Configure `GATEWAY_URL` to a gateway reachable from the sidecar container. - Set `GATEWAY_HOST` to the public operator hostname included in attestations. - Set `PUBLIC_GATEWAY_URL` when browser previews need a public gateway URL. - Mount `WALLET_FILE` read-only if operator-signed attestations are required. - Monitor readiness failures, gateway request failures, job duration, and verification outcomes. - Configure `ARIO_MLFLOW_ARIO_VERIFY_URL` in MLflow verification environments. With these pieces in place, training, model, and inference proofs anchored by `ar-io-mlflow` can be independently verified and attested by an ar.io gateway operator. # Training Provenance (/build/verifiable-ai/training-provenance) Training provenance starts with a simple question: can someone prove that a model's current MLflow record matches what was recorded when training finished? `ar-io-mlflow` answers that by writing a canonical payload to MLflow, signing a compact commitment envelope, and anchoring only that envelope to Arweave. The plugin can also anchor MLflow dataset records so downstream trainers can refer to an immutable dataset proof without publishing rows or raw files. ## Prerequisites - Python 3.10 or newer - MLflow 2.14 or newer - A local or remote MLflow tracking store - Optional Arweave JWK wallet for production identity If no wallet is configured, the plugin generates one at `~/.ario-mlflow/wallet.json` and reuses it. That is convenient for evaluation. In production, set `ARIO_MLFLOW_ARWEAVE_WALLET` to a dedicated wallet file from your secrets manager. ## Install Install [`ar-io-mlflow`](https://github.com/ar-io/ar-io-mlflow) from source. ```bash git clone https://github.com/ar-io/ar-io-mlflow.git cd ar-io-mlflow pip install -e . ``` For the quickstart example, install scikit-learn as well: ```bash pip install scikit-learn ``` ## Anchor a training run ### Configure MLflow Point MLflow at the tracking store you want to use. This can be a local folder for development or your normal remote tracking URI. ```python from pathlib import Path import mlflow tracking_dir = Path("./mlruns").resolve() mlflow.set_tracking_uri(f"file://{tracking_dir}") mlflow.set_experiment("verifiable-ai") ``` ### Log a dataset and model Log the dataset through MLflow so the plugin can include dataset provenance in the training proof. The dataset proof commits to the dataset name, source, digest, and schema hash, not to the row contents. ```python import mlflow.data from sklearn.datasets import load_iris from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split X, y = load_iris(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, ) with mlflow.start_run() as run: dataset = mlflow.data.from_numpy( X_train, targets=y_train, source="https://archive.ics.uci.edu/dataset/53/iris", name="iris-train", ) mlflow.log_input(dataset, context="training") model = LogisticRegression(max_iter=200).fit(X_train, y_train) mlflow.log_params({"max_iter": 200, "random_state": 42}) mlflow.log_metric("accuracy", model.score(X_test, y_test)) mlflow.sklearn.log_model(model, name="model") ``` ### Anchor the proof Call `ario_mlflow.anchor()` before the run exits. The call hashes model artifacts, writes `ario/payload.json`, signs the envelope, uploads it through Turbo, and writes `ario.*` tags back to MLflow. ```python import ario_mlflow with mlflow.start_run() as run: # Fit model, log params, log metrics, log dataset, log model... result = ario_mlflow.anchor( metadata={"service_name": "credit-risk-training"}, ) print("run_id:", run.info.run_id) print("payload_hash:", result["payload_hash"]) print("training_tx:", result["tags"].get("ario.training_tx")) print("verify_status:", result["tags"]["ario.verify_status"]) ``` If the Arweave upload fails, the run still succeeds and the envelope remains signed locally. In that case `ario.verify_status` is `signed` and `ario.training_tx` is absent. ### Verify the run later Run the CLI against the same MLflow tracking store. ```bash MLFLOW_TRACKING_URI=file:///absolute/path/to/mlruns \ ar-io-mlflow verify run ``` Verification checks that the envelope exists on ar.io, the MLflow payload still hashes to the anchored commitment, the live MLflow run still re-derives the same canonical bytes, and the Ed25519 signature is valid. ## Standalone dataset proofs Dataset publishers can anchor a dataset proof without an active training run and hand the transaction ID to downstream teams. ```python import mlflow import ario_mlflow dataset = mlflow.data.from_pandas( df, source="s3://example-bucket/training/q1.parquet", name="credit-risk-q1", ) result = ario_mlflow.anchor(dataset=dataset) print(result["tx_id"]) ``` This pattern is useful when a data platform team publishes approved datasets and model teams consume them later. The proof records an immutable commitment to the dataset descriptor, while the source data remains in S3, a lakehouse, or another controlled system. ## What gets written On the MLflow run, the plugin writes tags such as: - `ario.enabled` - `ario.version` - `ario.public_key` - `ario.verify_status` - `ario.artifact_hash` - `ario.payload_hash` - `ario.training_tx` - `ario.arweave_url` - `ario.wallet_mode` It also writes `ario/payload.json` as the canonical payload artifact. Arweave receives only the compact signed envelope, usually hundreds of bytes rather than the source data or model artifact. ## Production notes - Use a dedicated wallet per environment through `ARIO_MLFLOW_ARWEAVE_WALLET`. - Set `ARIO_MLFLOW_SIGNING_KEY` explicitly if you need controlled key rotation. - Configure `ARIO_MLFLOW_GATEWAYS` with at least two gateways for fetch fallback. - Monitor runs where `ario.verify_status = signed`, because those were signed but not anchored. - Keep MLflow artifacts backed up independently. Arweave preserves the envelope, but MLflow still holds the canonical payload used for full record matching. ## Next steps After training proofs are anchored, continue to [Model and Decision Proofs](/build/verifiable-ai/model-decision-proofs) to anchor registry events and enforce artifact integrity before inference. # Glossary (/glossary) ## ANS-104 Data Item A file packaged for bundled upload. Turbo signs your file as a data item and packs many of them into a single Arweave transaction. Each data item keeps its own id, so you can fetch it from any gateway exactly like a transaction. ## ar.io A permanent cloud network: independent gateways serve, index and verify permanent data, and ArNS gives it names. Anyone can run a gateway and join. ## Ar.io Name System (ArNS) A decentralized naming system in the ar.io network that maps human-readable names to Arweave transaction IDs and application/content routes. ## Ar.io Name Token (ANT) An NFT on Solana associated with a registered ArNS name. ANT ownership controls name-level configuration such as records, undernames, and transfer operations. ## ArDrive The app for storing and sharing files permanently, built on Turbo. ArDrive is the product for people who want a drive; these docs are for building your own upload and access paths. ## ARIO The native token of the ar.io network, an SPL Token on Solana with 6 decimals (1 ARIO = 1,000,000 mARIO). It is used to stake and reward gateways, and to register and extend ArNS names. See [the ARIO token](/learn/token). ## Arweave The storage layer underneath everything here. You pay once and the data persists, replicated across many nodes. Arweave stores the data; ar.io makes it fast to find, read and name. ## Bundle An Arweave transaction, or a data item, whose data holds a set of other data items packed in the ANS-104 format. A bundle contains data items; it is not one of them. See [Bundling](#bundling). ## Bundling Packing many data items into one Arweave transaction. Bundling is what makes small uploads cheap and quick to confirm, and it is what Turbo does with every upload it accepts. ## CPI (Cross-Program Invocation) The mechanism by which Solana programs call into each other. The ar.io architecture uses CPI for token operations — ario-gar and ario-arns call into ario-core for SPL Token transfers. ## Cranker A permissionless actor (bot or user) that submits transactions to drive lazy protocol operations, including the epoch pipeline and other state advancement tasks. Any participant can crank if protocol conditions are met. ## Data Item ID The id of a [data item](#ans-104-data-item), assigned when it is signed. A gateway can look up a data item by its data item id alone and resolve it to the root transaction that contains it. ## Epoch Pipeline The 6-step process for observation, reward calculation, and distribution each epoch: create_epoch → tally_weights → prescribe_epoch → save_observations → distribute_epoch → close_epoch. ## Gateway An ar.io node that serves Arweave data over HTTP: it retrieves, caches and indexes data, resolves ArNS names, and exposes standard interfaces for the permaweb. Anyone can run one. Turbo is an ar.io gateway run by the ArDrive team, and it also accepts paid uploads. See [Gateways](/learn/gateways). ## Gateway Address Registry (GAR) The protocol-level registry of joined gateways and their metadata. GAR powers gateway discovery and is a core input to observer selection, rewards, and network operations. ## Index Band One piece of a shared index, covering a range of block heights. Older bands rarely change; the band at the chain tip is rebuilt as new data arrives. A gateway installs the bands it needs and replaces one without pausing lookups. See [Index Sharing](/learn/gateways/index-sharing). ## Index Publication The signed document an Index Sharing publisher serves at `/ar-io/indexes`. It lists every index band and every file's size and SHA-256, and is signed with the gateway's registered observer key, so a subscriber can check it against the Gateway Address Registry. See [Reading Shared Indexes](/build/advanced/index-publications). ## Index Sharing How ar.io gateways share the indexes that locate data items inside their bundles. A gateway publishes an [index publication](#index-publication) signed with its registered observer key, listing its index bands and every file's SHA-256; other gateways and clients verify it and use the bands without trusting the server or peer they fetched them from, over HTTP or BitTorrent. See [Index Sharing](/learn/gateways/index-sharing). ## Infohash A torrent's identifier: the hash of its list of files and the hashes of their pieces. Index publications sign each band's infohashes, so a subscriber can reject any torrent that doesn't match. Publishers of the same bytes produce the same infohash and share one swarm. See [Index Sharing](/learn/gateways/index-sharing). ## Metaplex Core The NFT standard used for Ar.io Name Tokens (ANTs) on Solana. Metaplex Core assets are tradeable on standard NFT marketplaces like Tensor and Magic Eden. ## Native Address An address format that uses the raw public key bytes directly, without additional encoding or transformation. ## Normalized Address A standardized address format that ensures consistent representation across different systems and contexts. Normalized addresses help prevent issues with address matching and lookup operations. ## Observer A gateway selected to evaluate peer gateways during an epoch. Observers test assigned ArNS resolutions and submit reports used by the protocol for performance evaluation and rewards. A gateway's observer key also signs its [Index Sharing](#index-sharing) publications. ## Operations Address A second Solana address that a gateway operator authorizes to update the gateway's metadata and claim the ArNS gateway operator discount. It can't change stake, delegation settings, the observer address, or the operations address itself. See [Delegate gateway management](/build/run-a-gateway/manage/operations-address). ## Optimistic Indexing A data indexing strategy where new data is immediately made available for queries while verification processes continue in the background. This approach improves performance while maintaining data integrity through eventual consistency. ## PDA (Program Derived Address) Deterministic Solana account addresses derived from a program ID and a set of seeds. The ar.io protocol uses PDAs extensively — for gateways, delegations, withdrawals, vaults, ArNS records, ANT configs, and more. ## Permaweb The permanent web of files, applications and identities built on Arweave, served through ar.io gateways and addressed by transaction id or ArNS name. ## Program ID The unique onchain address of a deployed Solana program. Ar.io protocol interactions target specific program IDs for instructions and state access. ## Protocol Balance A protocol-controlled ARIO token pool that accumulates relevant inflows (for example, protocol fees) and serves as a source for incentive distributions according to protocol rules. ## Public Key A cryptographic key that can be shared publicly and is used to verify digital signatures or encrypt data. In the ar.io context, Solana public keys (Ed25519, base58-encoded) are used to identify wallet addresses and verify transactions. ## Root Transaction The Arweave transaction that contains a data item, found by walking up through every bundle around it. A gateway resolves a data item's root transaction to serve and locate it. ## SPL Token Solana Program Library token standard. ARIO is implemented as an SPL Token on Solana, making it compatible with the full Solana wallet and DeFi ecosystem. ## Transaction (Solana) A signed unit of interaction with Solana containing one or more instructions. Instructions in a transaction are processed atomically: all succeed, or the transaction fails without applying state changes. ## Turbo An ar.io gateway run by the ArDrive team. It serves permanent data like any [ar.io gateway](/learn/gateways) and adds paid uploads: it bundles your files into ANS-104 data items, issues the credits you pay with, and posts the data permanently to Arweave. See [Getting Started with Turbo](/build/upload/bundling-services). ## Turbo Credits Prepaid upload balance, bought with a card or crypto and spent on uploads and ArNS names. Priced in [winc](#winc) and non-transferable. See [Paying for Uploads](/build/upload/turbo-credits). ## Vault A protocol-level token lock with a start time, optional end time, and token balance. Vaults are used to hold ARIO over time and may be revocable depending on how they were created. ## Wayfinder A client library that chooses an ar.io gateway for each request and verifies that what comes back matches what was stored, so an app does not have to trust a single gateway. See [Wayfinder](/learn/wayfinder). ## WebSeed An HTTP address (BEP 19) that serves the same bytes as a torrent swarm, so a download completes even with no peers. On a gateway it is `/ar-io/indexes/webseed/`, rate limited and x402-priced while the swarm itself is free. Index torrents do not list it: subscribers add a publisher's WebSeed themselves, and only when peers stall. See [Index Sharing](/learn/gateways/index-sharing). ## winc The unit Turbo Credits are counted in, as it appears in the Turbo API. Receipt and balance fields return amounts in `winc`. Not to be confused with Winston, the smallest unit of AR. ## Winston Winston is the smallest unit of Arweave's native AR token. One AR equals 10^12 Winston. # What are Bundles? (/learn/(introduction)/ans-104-bundles) ANS-104 bundles are **data packaging standards** that efficiently bundle multiple data items and submit them to Arweave as single transactions, reducing transaction overhead and improving network efficiency. ## The Problem ANS-104 Solves **Individual Arweave transactions have inherent limitations:** - **Transaction overhead** - Each transaction requires separate processing and storage - **Network inefficiency** - Multiple small transactions consume more network resources - **Indexing complexity** - Individual transactions are harder to organize and query - **Storage fragmentation** - Related data items are stored separately **ANS-104 provides:** - **Reduced transaction overhead** by batching multiple data items - **Improved network efficiency** through consolidated transactions - **Better indexing capabilities** with structured data item format - **Standardized data format** for interoperability across applications ## How ANS-104 Bundling Works ### The ANS-104 Standard ANS-104 is the [official specification](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md) for bundling data on Arweave: - **Data Items** - Individual pieces of data with standardized binary format - **Bundle** - Single Arweave transaction containing multiple data items - **Binary Serialization** - Consistent format for data item structure - **Standardized Format** - Ensures interoperability across applications ### How ANS-104 Works 1. **Data Item Creation** - Create individual data items with ANS-104 format 2. **Bundle Assembly** - Combine multiple data items into a single bundle 3. **Transaction Creation** - Submit bundle as one Arweave transaction 4. **Network Processing** - Miners process the single bundle transaction 5. **Data Retrieval** - Individual data items can be extracted and indexed ## Key Benefits of ANS-104 **Reduced Overhead** - Bundle multiple data items into a single transaction to reduce processing overhead **Network Efficiency** - Consolidate multiple uploads into fewer network transactions **Standardized Format** - Consistent binary serialization ensures interoperability across applications **Better Indexing** - Structured data item format enables more efficient data retrieval and querying ## Why ANS-104 Matters for the Permaweb ANS-104 bundles are essential for building scalable applications on the permaweb because they: - **Enable efficient data storage** by reducing transaction overhead for multiple data items - **Improve network performance** through consolidated transactions - **Support better data organization** with standardized data item formats - **Enable scalable applications** that need to store many related data items efficiently ## Explore Bundling } title="Upload Data" description="Learn how to upload data to Arweave using bundling" href="/build/upload" /> } title="Run a Bundler" description="Deploy your own bundling infrastructure" href="/build/extensions/bundler" /> } title="Gateway Extensions" description="Integrate bundling with your ar.io gateway" href="/build/extensions" /> } title="Turbo SDK" description="Use Turbo SDK for easy data uploads and bundling" href="/sdks/turbo-sdk" /> # Introduction (/learn/(introduction)) import { BookOpen, Wrench, Package, Code, Server, Globe, ArrowRight, Zap, Shield, Infinity, } from "lucide-react"; Ar.io is a data infrastructure solution for long-term access. We help enterprises, institutions, and platforms ensure that critical information remains accessible, verifiable, and intact over time — even as systems, providers, technologies, or attacks disrupt the environments around them. **For AI and LLM users**: Start at llms.txt for a machine-readable index, SKILL.md for code-generation recipes, or llms-full.txt for the complete documentation as plain text. See [AI Agents & LLMs](/build/agents) for the full toolkit. ## Explore the Documentation } title="What is ar.io?" description="Learn about the decentralized gateway protocol and how it powers the permanent web" href="/learn/what-is-ario" /> } title="Build" description="Get started building applications, running gateways, and uploading data" href="/build" /> } title="SDKs" description="Integrate ar.io services into your applications with our developer SDKs" href="/sdks" /> } title="API Reference" description="Complete API documentation for ar.io Node and Turbo services" href="/apis" /> ## Quick Start Guides } title="Upload Data to Arweave" description="Learn how to permanently store files and data using Turbo SDK" href="/build/upload" /> } title="Run a Gateway" description="Deploy your own ar.io gateway and participate in the network" href="/build/run-a-gateway" /> } title="Register ArNS Names" description="Get human-readable names for your permanent applications" href="/learn/arns" /> ## ar.io Ecosystem } title="ArNS Registry" description="Register and manage permanent names for your applications" href="https://arns.ar.io" /> } title="Network Portal" description="Monitor gateway performance and network statistics" href="https://gateways.ar.io" /> } title="ArDrive" description="User-friendly permanent storage for files and folders" href="https://ardrive.io" /> ## Join the Community Connect with developers, gateway operators, and the ar.io team. Get help, share ideas, and stay updated on the latest developments. - [Join Discord](https://discord.com/invite/HGG52EtTc2) - [View Guides](/build/guides) # Protocol Architecture (/learn/(introduction)/protocol-architecture) ## Ar.io Protocol Architecture The ar.io protocol operates through five Solana programs that work together via cross-program invocation (CPI). This architecture organizes protocol responsibilities into a coordinated set of programs with clear boundaries. For the deployed Solana mainnet program IDs and ARIO token mint, see [Mainnet Addresses](/learn/token#mainnet-addresses). ```mermaid graph TD subgraph CORE["ario-core"] direction TB BAL[SPL Token / Balances] VAULTS[Vaults] PRIMARY[Primary Names] end subgraph GAR["ario-gar"] direction TB GW_REG[Gateway Registry] STAKING[Staking & Delegation] EPOCHS[Epochs & Rewards] end subgraph ARNS["ario-arns"] direction TB NAME_REG[Name Registry] DEMAND[Demand Factor] PRICING[Pricing & Auctions] end subgraph ANT["ario-ant"] direction TB NFT[Metaplex Core NFTs] RECORDS[DNS-like Records] CTRL[Controllers] end subgraph ESCROW["ario-ant-escrow"] direction TB ANT_ESC[ANT Escrow] TOKEN_ESC[Token / Vault Escrow] CLAIMS[Arweave + Ethereum Claims] end CORE --> GAR CORE --> ARNS ARNS --> ANT ARNS -.->|gateway data| GAR ESCROW -->|cpi: transfer/update ANT| MPL[Metaplex Core] classDef coreProgram fill:#e3f2fd classDef garProgram fill:#e8f5e8 classDef arnsProgram fill:#fff3e0 classDef antProgram fill:#f3e5f5 classDef escrowProgram fill:#fce4ec classDef externalProgram fill:#eeeeee,stroke-dasharray: 5 5 class BAL,VAULTS,PRIMARY coreProgram class GW_REG,STAKING,EPOCHS garProgram class NAME_REG,DEMAND,PRICING arnsProgram class NFT,RECORDS,CTRL antProgram class ANT_ESC,TOKEN_ESC,CLAIMS escrowProgram class MPL externalProgram ``` `ario-ant-escrow` has no arrow to `ario-core`, `ario-gar`, or `ario-arns` above, and that is deliberate, not an omission: current `main` shows no CPI between them. Its only onchain call out is `invoke`/`invoke_signed` into Metaplex Core (`mpl_core_cpi.rs`, called from `deposit.rs`, `claim_arweave_attested.rs`, `claim_ethereum.rs`, `cancel.rs`, and `admin_purge_unclaimed.rs`) to move the escrowed ANT. It also calls `CpiContext`/`token::transfer` into the standard SPL Token program for liquid ARIO and vault transfers (`deposit_tokens.rs`, `deposit_vault.rs`, and their cancel/claim counterparts); that edge is not drawn, matching how this diagram never draws `ario-core`'s own SPL Token calls as a separate node either. Arweave claims are checked by instruction introspection of a sibling `Ed25519Program` instruction, and Ethereum claims by the `secp256k1_recover` syscall: neither is a CPI, so neither gets an edge. ## Programs ### ario-core The core program manages the ARIO SPL Token, vaults, and primary names: - **SPL Token Operations**: ARIO is a standard SPL Token (6 decimals, 1 ARIO = 1,000,000 mARIO). Transfers, balances, and token accounts follow the SPL Token standard. - **Vaults**: Time-locked token deposits for various purposes, including ecosystem programs and other protocol-managed incentives. Other programs (ario-gar, ario-arns) call into ario-core via CPI for all token operations. ### ario-gar (Gateway Address Registry) Manages the network's gateway infrastructure, staking, delegation, and the epoch reward pipeline: - **Gateway Registry**: An onchain registry of network gateways. Each gateway is a PDA storing operator address, stake, observer address, settings, and performance stats. - **Staking & Delegation**: Operator stakes, delegated stakes (separate PDA per gateway-delegator pair), withdrawals, redelegation, and allowlists. - **Observer Address Uniqueness**: An ObserverLookup PDA enforces that no two gateways share the same observer address. - **Epoch Pipeline**: A 6-step permissionless pipeline driven by [cranker bots](/learn/oip/epoch-pipeline): 1. `create_epoch` — Initialize epoch, compute reward rate 2. `tally_weights` — Batched weight computation 3. `prescribe_epoch` — Select observers and prescribed names via weighted roulette 4. `save_observations` — Observers submit pass/fail reports 5. `distribute_epoch` — Batched reward distribution 6. `close_epoch` — Close completed observer epoch accounts and return remaining SOL rent to observers - **Gateway Pruning**: Gateways that repeatedly fail observation are removed from the network and subject to stake slashing. ### ario-arns (ArNS Registry) Manages the Ar.io Name System — name registration, leasing, pricing, and returned names: - **Name Registry**: An onchain registry of ArNS domains. Each name is an ArnsRecord PDA storing the owner, ANT mint address, lease type, and expiration. - **Pricing**: Dynamic pricing via a DemandFactor PDA. - **Returned Name Auctions**: When a name expires or is released, it enters a Dutch auction. - **Validation**: Names must meet certain formatting criteria. - **Cost Simulation**: `get_token_cost` view instruction available via `simulateTransaction` for fee estimation. ### ario-ant (Ar.io Name Tokens) ANTs are [Metaplex Core](https://developers.metaplex.com/core) NFTs that represent ownership of ArNS names: - **NFT Standard**: Each ANT is a Metaplex Core asset — tradeable on Tensor, Magic Eden, and other NFT marketplaces. - **DNS-like Records**: Each ANT stores routing records (AntRecord PDAs) mapping undernames to Arweave transaction IDs or IPFS CIDs, with configurable TTL values. - **Controllers**: Controllers can manage records without holding the NFT itself. - **Lazy Reconciliation**: When an ANT is transferred via a marketplace (outside the ar.io protocol), controllers are cleared on the next write operation, ensuring the new owner has full control. ### ario-ant-escrow A trustless escrow that hands an ANT, liquid ARIO, or a time-locked vault to a recipient identified by an Arweave or Ethereum address, claimable once the recipient holds a Solana wallet. No off-chain authority or oracle decides a claim: - **ANT escrow**: Holds a Metaplex Core ANT NFT in a per-mint PDA until the named recipient claims it. - **Token and vault escrow**: Escrows liquid ARIO or a time-locked vault the same way, in a PDA per depositor and asset id. - **Claims**: An Arweave claim is attested off-chain, re-signed with Ed25519, then verified onchain; an Ethereum claim is verified onchain with `secp256k1_recover` and EIP-191; a vault claim uses instruction introspection that preserves the remaining lock. See the [Escrow SDK reference](/sdks/ar-io-sdk/escrow). ## State Model All protocol state is stored in Solana accounts using Program Derived Addresses (PDAs): | Account Type | Program | Derivation Seeds | Purpose | |-------------|---------|-----------------|---------| | ArioConfig | ario-core | `["config"]` | Global token configuration | | Vault | ario-core | `["vault", owner, vault_id]` | Time-locked token deposit | | PrimaryName | ario-core | `["primary_name", name_hash]` | Name → address mapping | | PrimaryNameReverse | ario-core | `["primary_name_reverse", owner]` | Address → name reverse lookup | | GatewayRegistry | ario-gar | `["gateway_registry"]` | Zero-copy gateway slot array | | Gateway | ario-gar | `["gateway", operator]` | Individual gateway state | | Delegation | ario-gar | `["delegation", gateway, delegator]` | Per-pair delegation | | Withdrawal | ario-gar | `["withdrawal", owner, id]` | Pending withdrawal | | ObserverLookup | ario-gar | `["observer_lookup", observer]` | Observer uniqueness check | | Epoch | ario-gar | `["epoch", epoch_index]` | Epoch state and rewards | | NameRegistry | ario-arns | `["name_registry"]` | Zero-copy name slot array | | ArnsRecord | ario-arns | `["arns_record", name_hash]` | Individual name record | | DemandFactor | ario-arns | `["demand_factor"]` | Current pricing multiplier | | AntConfig | ario-ant | `["ant_config", mint]` | ANT metadata and settings | | AntControllers | ario-ant | `["ant_controllers", mint]` | Controller list (max 10) | | AntRecord | ario-ant | `["ant_record", mint, undername_hash]` | DNS-like routing record | | EscrowAnt | ario-ant-escrow | `["escrow_ant", ant_mint]` | ANT held in escrow for a claimant | | EscrowToken | ario-ant-escrow | `["escrow_token" or "escrow_vault", depositor, asset_id]` | ARIO or vault held in escrow for a claimant | ## Security Model The protocol relies on a combination of Solana runtime guarantees, deterministic account ownership, and economic incentives: - **Signer and account checks**: Solana enforces transaction signatures, account ownership, and program execution rules. - **Program Derived Addresses (PDAs)**: Protocol state lives in deterministic accounts controlled by the relevant ar.io programs. - **Economic accountability**: Gateway operators stake ARIO, and repeated failure can lead to removal and slashing. - **Permissionless verification**: Gateway observations and cranking keep network operations open and independently verifiable. # Ar.io for Solana Developers (/learn/(introduction)/solana-developers) If you build on Solana, you already know most of what you need to use ar.io. This page maps familiar Solana concepts to ar.io equivalents and gets you to your first permanent deployment fast. ## What ar.io Does ar.io is a **permanent cloud network**: independent gateways serve, index and verify permanent data, and ArNS gives it names. You get: - **Permanent hosting** — Upload your frontend once, it lives forever. No hosting bills. - **Human-readable URLs** — `yourapp.ar.io` instead of a 43-character transaction ID - **Decentralized CDN** — Gateways worldwide serve your content, no single point of failure - **On-chain naming** — ArNS names are Metaplex Core NFTs tradeable on Tensor/Magic Eden The ar.io protocol runs on Solana. You interact with it using your existing Solana wallet, SOL for fees, and ARIO tokens for protocol operations. ## Concept Mapping If you know Solana, you already understand ar.io: | Solana Concept | ar.io Equivalent | Details | |---|---|---| | **SPL Token** | ARIO token | The protocol token. 6 decimals, 1B supply. Used for staking, ArNS names, delegation. | | **Metaplex Core NFT** | ANT (Ar.io Name Token) | Each ArNS name is backed by an NFT. Trade on Tensor/Magic Eden. Holds DNS-like records. | | **Anchor Program** | ario-core, ario-gar, ario-arns, ario-ant | Four programs that manage the protocol. Interact via SDK or CPI. | | **PDA** | All protocol state | Gateways, delegations, vaults, name records, ANT configs — all PDAs. | | **CPI** | Cross-program token ops | ario-gar and ario-arns call into ario-core for SPL Token transfers. | | **Transaction fee (SOL)** | Same | Every protocol operation costs a small SOL fee ( You DO still need an Arweave wallet (or SOL via Turbo) to **upload data** to Arweave. Turbo accepts SOL directly — see the upload section below. **Want to try before spending real ARIO?** Run this whole flow — upload, buy an ArNS name, and resolve it — for free on the **[Testnet Sandbox](/build/testnet)**, which runs the full ar.io stack on Solana **devnet** with a faucet-funded staging token. It's also the ideal target for AI coding agents and CI. When you're ready for mainnet, the code below is the same minus the sandbox endpoints. ## Quick Start: 5 Minutes to Permanent ### 1. Install ```bash npm install @ar.io/sdk @solana/kit @ardrive/turbo-sdk ``` ### 2. Upload Your App to Arweave Use Turbo to upload your build folder. Turbo accepts SOL directly — no Arweave wallet needed: ```typescript // Authenticate with your Solana private key const turbo = TurboFactory.authenticated({ privateKey: bs58.encode(solanaSecretKey), token: 'solana', }); // Upload your build folder const result = await turbo.uploadFolder({ folderPath: './dist', dataItemOpts: { tags: [{ name: 'App-Name', value: 'my-cool-app' }], }, }); console.log('Manifest TX:', result.manifestResponse.id); // This is the Arweave Transaction ID — your app is now permanent ``` ### 3. Register an ArNS Name ```typescript // Create signer from Solana keypair const keypairBytes = new Uint8Array( JSON.parse(fs.readFileSync('./solana-keypair.json', 'utf-8')), ); const signer = await createKeyPairSignerFromBytes(keypairBytes); const ario = ARIO.mainnet({ signer }); // Buy the name (mints an ANT as a Metaplex Core NFT) const result = await ario.buyRecord({ name: 'my-cool-app', type: 'lease', years: 1, }); ``` ### 4. Point Your Name to Your App ```typescript // Get your ANT's mint address from the ArNS record const record = await ario.getArNSRecord({ name: 'my-cool-app' }); // Initialize the ANT and set the record const ant = ANT.init({ signer, processId: record.processId }); await ant.setRecord({ undername: '@', transactionId: result.manifestResponse.id, // from the Turbo upload ttlSeconds: 3600, }); // Your app is now live at: // https://my-cool-app.ar.io // https://my-cool-app. ``` ## How ArNS URLs Work Every ar.io gateway resolves ArNS names as subdomains: ``` https://my-cool-app.ar.io → served by the ar.io gateway https://my-cool-app. → served by that ar.io gateway ``` All network gateways serve the same content. If one goes down, users access through another. Your app is truly decentralized. Undernames use underscores: ``` https://docs_my-cool-app.ar.io → the "docs" undername https://api_my-cool-app.ar.io → the "api" undername ``` ## Costs at a Glance | Operation | Cost | |---|---| | Upload (up to 105 KiB) | **Free** via Turbo, within a 10 MiB lifetime allowance per wallet and another per IP range | | Upload (larger files) | Pay with SOL, ARIO, or fiat. [See pricing](https://console.ar.io/pricing) | | ArNS name (1 year lease, 5+ chars) | ~200-2,500 ARIO depending on length × demand factor | | ArNS name (permabuy, 5+ chars) | ~200-2,500 ARIO × demand factor | | Set/update a record | SOL fee only ( Use `ario.getTokenCost()` to check exact pricing before any purchase. ArNS prices adjust dynamically based on demand. ## What to Build Ar.io is ideal for: - **Permanent frontends** — Deploy your React/Next.js/Vue app with zero hosting costs forever - **Immutable assets** — Store game assets, NFT metadata, or config files that can never be altered - **Decentralized publishing** — Blog platforms, documentation sites, content archives - **Censorship-resistant apps** — Apps that no single entity can take down - **Version-controlled deployments** — Use undernames for staging, production, and rollbacks ## Next Steps } /> } /> } /> } /> } /> # What is ar.io? (/learn/(introduction)/what-is-ario) Ar.io is the first permanent cloud network. A decentralized infrastructure layer built on Arweave for permanent data storage, with protocol execution on Solana. Think of it as the gateway to Arweave's permaweb, turning its tamper-proof storage into a fully functional, user-friendly ecosystem for apps, websites, and data. The permaweb is the permanent web of files, applications, and identities built on Arweave. Ar.io makes that permanent data usable by providing fast gateway access, indexing, human-readable names, and routing across a decentralized network. ## Features of ar.io ### Gateways Ar.io operates a network of [gateways](/learn/gateways) — nodes that serve as entry points to Arweave’s data. These gateways fetch and deliver data quickly, supporting everything from static files to dynamic web apps. ### Ar.io Name System (ArNS) The [Ar.io Name System (ArNS)](/learn/arns) is a decentralized naming system for Arweave. It allows users to register and resolve human-readable names to Arweave transaction IDs. ### Data Access Ar.io offers a range of tools for accessing and querying data on Arweave, including: - [HTTP Requests](/build/access/fetch-data) via gateways - [GraphQL Queries](/build/access/find-data) for finding data by tags and metadata - [ArNS](/learn/arns) for human-readable URLs - [Wayfinder](/learn/wayfinder) for decentralized content discovery ## The Problem Arweave stores data forever, but accessing and organizing that data isn't always straightforward. Without efficient tools, retrieving files, serving websites, or finding specific content on Arweave's blockweave can be slow or complex, limiting its potential for developers and users. ## The Solution Ar.io builds on Arweave's permanent storage to create a decentralized, scalable access layer. It provides gateways, domain names, and indexing services, making it easy to interact with permaweb content as seamlessly as the traditional web. ### How It Works - **Decentralized Gateways**: ar.io operates a network of gateways—nodes that serve as entry points to Arweave’s data. These gateways fetch and deliver data quickly, supporting everything from static files to dynamic web apps. - **ArNS (Ar.io Name System)**: ar.io introduces decentralized domain names (e.g., yourname.arweave), mapping human-readable names to Arweave’s data IDs. This makes content easy to find and share, like URLs on the traditional web. - **Indexing and Querying**: ar.io enables efficient data indexing, allowing developers to search and retrieve specific content from Arweave’s vast storage without scanning the entire blockweave. - **Routing and verification**: ar.io 's ar://wayfinder Protocol intelligently routes requests to available gateways in the network and verifies the data's authenticity. - **Observation and incentives**: ARIO's [Observation and Incentive Protocol (OIP)](/learn/oip), ensures gateway operators are serving the right data and rewards them in the protocol native token, $ARIO, to create a secure and self-sustaining network. **In Simple Terms**: Imagine Arweave as a massive, unerasable library. Ar.io is the librarian who organizes the shelves, provides a catalog, and hands you the books you need—fast. ## Why It Matters - **Accessible**: Gateways make permaweb content load as quickly as traditional websites. - **Discoverable**: ArNS provides user-friendly domain names, simplifying navigation. - **Scalable**: Supports growing permaweb usage, from small apps to global platforms. - **Decentralized**: No single entity controls access, ensuring censorship resistance. ## Building on ar.io Ar.io empowers developers to create permaweb apps with tools for hosting, querying, and monetizing content, all while leveraging Arweave's permanent storage. **In Simple Terms**: Arweave locks data forever; ar.io makes it ready for the world to use. ## Ready to Dive Deeper? Ar.io transforms Arweave into a vibrant permaweb ecosystem. Ready to start building? Explore our comprehensive guides and start creating on the permanent web. ## Explore ar.io } /> } /> } /> } /> # What is Arweave? (/learn/(introduction)/what-is-arweave) Arweave is a decentralized storage network that ensures data is **permanent**, **affordable**, and **scalable**. Think of it as a global, tamper-proof hard drive where your files—photos, documents, or apps—stay accessible forever. It's the foundation for [ar.io](https://ar.io), powering a "permaweb" where data never disappears. Below, we break down Arweave's core features in a simple, beginner-friendly way. ## A Datachain for Permanent Storage Arweave is like Bitcoin, but for data. It solves one problem really well: **storing data permanently**. Once uploaded, your data—whether a tweet, NFT, or website—is immutable and preserved indefinitely. ### How Does It Work? - **Blockweave Architecture**: Unlike a blockchain's single chain, Arweave's blockweave links each new data block to the previous one and a random older block. Data is split into 256 KiB chunks in a secure Merkle tree, ensuring miners keep all data to add new blocks. - **Succinct Proofs of Random Access (SPoRA)**: Miners prove they store multiple data copies by accessing random chunks, verified efficiently with Verifiable Delay Functions (VDFs). This combines proof-of-work and proof-of-storage, making data loss nearly impossible. **In Simple Terms**: Picture a library where new books reference older ones, and librarians must keep every book to add more. SPoRA ensures they prove they’ve got the books, keeping your data safe forever. ## Pay Once, Store Forever: No Recurring Fees Pay a one-time fee to upload data, and it's stored "forever"—no subscriptions or renewals. ### How Does It Work? - **Endowment Fund**: Your fee, based on 200 years of storage for 20 replicas, goes mostly into a fund that slowly pays miners in AR tokens to maintain data. It assumes storage costs drop over time, making the fund sustainable. **In Simple Terms**: It's a “forever stamp” for data. Your payment funds a pot that keeps paying storage keepers, lasting longer as tech gets cheaper. ## Practically Unlimited Storage Arweave can practically store unlimited data, from small files to entire digital archives, without hitting a ceiling. The theoretical limit is 2^256 bytes which for scale is more atoms than there are in the universe. ### How Does It Work? - **Layer 1 Transactions**: Data is stored as 256 KiB chunks on the blockweave, replicated across many nodes. As more nodes join with standard hardware, storage capacity grows limitlessly. - **Bundling with [ar.io](https://ar.io) and Turbo**: Bundling packs multiple files into one transaction, reducing costs and congestion. Ar.io and Turbo optimize this, enabling fast, cheap uploads of large datasets like websites. Turbo is an ar.io gateway run by the ArDrive team, and it bundles your files before posting them to Arweave. ## What Arweave doesn't solve well? Access Arweave solve's one problem and solve's it well. Storing your data for a very long-time. It doesn't, however, incentivise the indexing and access for data. ## Ready to Dive Deeper? Arweave powers a permaweb where apps, websites and data live forever. For ar.io developers, it's the bedrock for unstoppable decentralized applications. Learn more in the next section: [What is ar.io](/learn/what-is-ario). ## Explore Arweave } /> } /> } /> # Ar.io Name Tokens (ANTs) (/learn/arns/ants) To establish ownership of a record in the ArNS Registry, each record contains both a friendly name and a reference to an Ar.io Name Token (ANT). ANTs are [Metaplex Core](https://developers.metaplex.com/core) NFTs on Solana that give their owners the ability to update the Arweave transaction IDs their associated friendly names point to. ## What is an ANT? An ANT is a Metaplex Core NFT managed by the `ario-ant` Solana program. It implements the Ar.io Name Token specification required by ar.io gateways to resolve ArNS names to Arweave transaction IDs. The program handles record updates, controller management, metadata changes, and ownership reconciliation after transfers. Name Tokens have an owner, who can transfer the token and control its modifiable settings. These settings include record targets, address-resolution TTL values, the ANT name and ticker, and controller permissions. ## Ownership and Control Controllers can manage records, name, and ticker settings, but cannot transfer the ANT or assign additional controllers. ANTs are minted in accordance with network standards by an end user who can transfer ownership or assign controllers as needed. Because ANT logic lives in the `ario-ant` program rather than inside each individual token, protocol-level updates can apply consistently across ANTs. When an ANT is transferred through a marketplace outside the ar.io app, the `ario-ant` program reconciles ownership on the next interaction and clears previously assigned controllers so the new owner has clean control. Loss of a private key for a permanently purchased name can result in the name being inaccessible. ### Undername Ownership Undernames can have an `owner` set on them. This owner is empowered to set that undername as their primary name, remove that undername as their primary name, and control that undername's metadata, such as: - Transaction ID - the data the record resolves to. - TTL seconds - the time to live in seconds the data is cached for by clients. - Owner - the owner of the record. - Description - the description of the record. - Display Name - the display name for the owner of the record. - Keywords - the keywords for the record. - Logo - the logo of the record. They do *NOT* have control over the `priority` of the undername, which is restricted to the ANT Controllers and Owner. ## ANT Interactions The table below indicates some of the possible interactions with the ArNS registry, corresponding ANTs, and who can perform them: | Type | ANT Owner | ANT Controller | Undername Owner | Any ARIO Token Holder | | ----------------------------------------- | ----------- | ---------------- | ----------------- | ----------------------- | | Transfer ownership | ✔ | | | | | Add / remove controllers | ✔ | | | | | Approve/Remove Primary name | ✔ | | ✔ | | | Reassign name to new ANT | ✔ | | | | | Return a permanent name | ✔ | | | | | Set records (pointers, record metadata) | ✔ | ✔ | ✔ | | | Update records, name, ticker | ✔ | ✔ | | | | Update descriptions and keywords | ✔ | ✔ | | | | Create and assign undernames | ✔ | ✔ | | | | Extend / renew lease | ✔ | ✔ | ✔ | ✔ | | Increase undernames | ✔ | ✔ | ✔ | ✔ | | Convert lease to permanent | ✔ | ✔ | ✔ | ✔ | ## Undernames ANT owners and controllers can configure subdomains for their registered ArNS name, known as undernames. These undernames are assigned at registration or added later. Undernames use an underscore (`_`) in place of a dot (`.`) to separate the subdomain from the main ArNS domain. ## Secondary Markets Since ANTs are standard Metaplex Core NFTs, they are tradeable on any compatible NFT marketplace, including **Tensor** and **Magic Eden**. When an ANT is sold on a marketplace, lazy reconciliation clears the existing controllers on the next write operation, ensuring the new owner gets clean control. Additionally, tertiary markets could be created that support the leasing of these friendly names to other users. ## Next Steps } /> } /> } /> # Ar.io Name System (ArNS) (/learn/arns) ## What is ArNS? Arweave URLs and transaction IDs are long, difficult to remember, and occasionally categorized as spam. The Ar.io Name System (ArNS) aims to resolve these problems in a decentralized manner. ArNS is a **censorship-resistant naming system** powered by [ARIO tokens](/learn/token), enabled through [ar.io gateway](/learn/gateways) domains, and used to connect friendly domain names to permaweb apps, web pages, data, and identities. It's an open, permissionless, domain name registrar that doesn't rely on a single TLD. ## How ArNS Works This system works similarly to traditional DNS services, where users can purchase a name in a registry and DNS Name servers resolve these names to IP addresses. The system is flexible and allows users to purchase names permanently or lease them for a defined duration based on their use case. With ArNS, the registry is managed on Solana by the `ario-arns` program, making ownership and registration state globally verifiable. Names are controlled by Ar.io Name Tokens (ANTs), which point to Arweave transaction IDs so gateways can route friendly names to permanent data. ```mermaid graph TD User[User requests friendly name] --> Gateway[ar.io Gateway] Gateway --> Registry[ArNS Registryario-arns] Registry --> ANT[Ar.io Name TokenMetaplex Core NFT] ANT --> Target[Arweave Transaction ID] Gateway --> Content[Permaweb app, page, or data] Target --> Content ``` ## Name Resolution Users can register a name, like `ardrive`, within the ArNS Registry. Ownership is represented by an Ar.io Name Token (ANT), a Metaplex Core NFT on Solana used by ArNS to track control over the name. ANTs allow the owner to set a mutable pointer to any type of permaweb data, like a page, app or file, via its Arweave transaction ID. Each ar.io gateway acts as an ArNS name resolver. Gateways fetch the latest state of both the ArNS Registry and associated ANTs from Solana and use that state to route users to the right Arweave transaction. Ar.io gateways resolve names as gateway subdomains, e.g., `https://ardrive.ar.io`, and proxy requests to the associated Arweave transaction ID. This means an ArNS name can work across ar.io gateways that support ArNS. Users can easily reference these friendly names in their browsers, and other applications and infrastructure can build rich solutions on top of these ArNS primitives. ```mermaid sequenceDiagram participant User participant Gateway as ar.io Gateway participant Registry as ArNS Registry participant ANT as Ar.io Name Token participant Arweave User->>Gateway: Request ardrive.ar.io Gateway->>Registry: Query "ardrive" record Registry-->>Gateway: Returns ANT address Gateway->>ANT: Get target TxID ANT-->>Gateway: Returns TxID (abc123...) Gateway->>Gateway: Check cache for TxID alt TxID not in cache Gateway->>Arweave: Fetch data from TxID Arweave-->>Gateway: Returns permaweb content end Gateway-->>User: Serves content ``` ## Key Benefits - **Human-readable URLs** instead of complex transaction IDs - **Censorship-resistant** and decentralized - **Permanent data routing** to Arweave transaction IDs - **Cross-gateway compatibility** - works on all ar.io gateways - **Flexible ownership** - lease names for a defined period or buy them permanently ## Explore ArNS } /> } /> } /> } /> # Name Registration (/learn/arns/name-registration) ArNS names can be registered as leases or permanent purchases, depending on how long the user needs the name and how much they want to commit up front. ## Registration Types ### Lease Registration A name may be **leased on a yearly basis**. Leases lower the barrier to entry, support temporary projects, and allow inactive names to eventually return to public availability. ### Permanent Registration (Permabuy) A name may be **purchased for an indefinite duration** with no expiration date. This is useful for long-lived apps, identities, and data references that should remain associated with a stable friendly name. Registering a name requires spending ARIO tokens corresponding to the name's character length and purchase type. ## Name Registry The ArNS Registry is a list of registered names and their associated ANT mint addresses, managed by the `ario-arns` Solana program. Key rules embedded within the protocol include: - **Genesis Prices**: Set within the contract as starting conditions - **Dynamic Pricing**: Varies based on name length, purchase type (lease vs buy), lease duration, and current Demand Factor - **Name Records**: Include a pointer to the Ar.io Name Token (ANT) mint address, lease end time (if applicable), and undername allocation - **Reassignment**: Name registrations can be reassigned from one ANT to another - **Lease Extension**: Anyone with available ARIO Tokens can extend any name's active lease - **Lease to Permanent Buy**: Anyone with available ARIO Tokens can convert a name's lease to a permanent buy - **Undername Capacity**: Additional undername capacity can be purchased for actively registered names - **Name Removal**: Name records can only be removed from the registry if a lease expires, or a permanent name is returned to the protocol ## Name Validation Rules All names registered must meet the following criteria: 1. **Valid characters**: Only numbers 0-9, characters a-z and dashes 2. **Dash placement**: Dashes cannot be leading or trailing characters 3. **Single character domains**: Dashes cannot be used in single character domains 4. **Length limits**: 1 character minimum, 51 characters maximum 5. **43-character prohibition**: Names exactly 43 characters long are prohibited to prevent Arweave transaction ID collisions 6. **Lowercase enforcement**: Names must be lowercase at submission 7. **Reserved names**: Cannot be an invalid name predesignated to prevent unintentional use/abuse such as `www` ## Lease Management ### Lease Expirations When a lease term ends, there is a grace period where the lease can be renewed or converted to a permanent purchase before it fully expires. If this grace period elapses, the name is considered expired and returns to the protocol for public registration. Once expired, a name's associated undername registrations and capacity also expire. A recently expired name enters a **Returned Name Dutch Auction**, where it starts at a premium and decays back toward standard pricing over time. Revenue from returned name purchases is split between the protocol and the previous owner. ### Lease to Permabuy Conversions An actively leased name may be converted to a permanent registration. The price for this conversion is treated as if it were a new permanent name purchase. This functionality allows users to transition from leasing to permanent ownership based on changing needs and available resources. It generates additional protocol revenue through conversion fees, contributing to the ecosystem's financial health and reward system. ### Permanent Name Return Users have the option to "return" their permanently registered names back to the protocol. This process allows users to relinquish their ownership, returning the name to the protocol for public re-registration. Only the Owner of a name can initiate a name return. When a permanent name is returned, the name is subject to a "Returned Name Premium", similar to expired leases. A key difference is that if the name is repurchased during the premium window, the proceeds are split between the returning owner and the protocol balance. ## Primary Names The Ar.io Name System (ArNS) supports the designation of a "Primary Name" for users, simplifying how wallet addresses are displayed across applications. A Primary Name is a user-friendly alias that can replace complex wallet addresses, making interactions and profiles easier to manage and identify. Users can set one of their owned ArNS names as their Primary Name. The fee is equivalent to the cost of a single undername on a 51-character name of the same purchase type, adjusted by the current Demand Factor. Only one Primary Name can be set per wallet, and the same name cannot be the Primary Name for more than one wallet. The base name's ANT owner can remove any Primary Name set on one of its undernames. This allows applications to use a single, human-readable identifier for a wallet, improving user experience across the network. ## Next Steps Now that you understand name registration, learn about [Ar.io Name Tokens (ANTs)](/learn/arns/ants) to see how ownership and control work, or explore the [Pricing Model](/learn/arns/pricing-model) to understand how costs are calculated. # Pricing Model (/learn/arns/pricing-model) ## Addressing Variable Market Conditions The future market landscape is unpredictable, and ArNS is designed to adapt without relying on a centralized pricing oracle. Instead of fixing name prices to an external currency, the protocol adjusts pricing based on network activity and registration demand. This keeps ArNS self-contained while still allowing name prices to respond to changing market conditions over time. ArNS pricing is built from a few core ideas: 1. **Name length**: Shorter, more memorable names generally cost more. 2. **Registration type**: Names can be leased for a defined period or purchased permanently. 3. **Demand Factor**: A protocol multiplier adjusts prices based on recent ArNS activity. 4. **Returned Name Premium (RNP)**: Recently expired or returned names re-enter the market through a descending premium window. This approach lets ArNS remain predictable enough for users while still adapting to namespace demand. You can view current live pricing at [arns.ar.io](https://arns.ar.io/#/prices) to see these formulas in action, or compare name prices alongside storage costs in the [ar.io Console name price table](https://console.ar.io/pricing?type=domains). ## Key Definitions - **Demand Factor:** A protocol multiplier that adjusts prices based on recent registration activity. - **Base Fee:** The starting price for a name before dynamic adjustments. - **Lease:** A time-limited registration. - **Permabuy:** A permanent registration. - **Undername:** A subdomain-style record written with an underscore (`_`) instead of a dot (`.`). - **Protocol Revenue:** ARIO collected from ArNS actions, such as name registrations, lease extensions, and undername purchases. ## Dynamic Pricing Model ArNS uses an adaptive model to balance name availability, demand, and long-term sustainability. Prices are influenced by name length, whether the name is leased or bought permanently, undername capacity, and the current Demand Factor. The Demand Factor changes over time based on protocol activity. When demand is high, it can increase prices; when demand is low, it can decrease prices. If demand remains low for long enough, the protocol can step base fees downward so names stay accessible. For exact costs, users should rely on the live ArNS app or SDK cost simulation rather than copying formulas into their own applications. ## Returned Name Premiums (RNP) ArNS applies a **Returned Name Premium (RNP)** to names that re-enter the market after expiration or permanent return. The premium starts high and decreases over a return window until the name reaches standard pricing again. Returned name purchases split proceeds between the protocol balance and the previous owner. This discourages instant name sniping after expiry and gives owners a reason to release names they no longer need. For more detail, see [Returned Names](/learn/arns/returned-names). ## Gateway Operator ArNS Discount Gateway operators who demonstrate consistent, healthy participation in the network may be eligible for discounted ArNS interactions. This creates another incentive for gateways to provide reliable service while supporting ArNS usage. Discount eligibility requires a **Gateway Performance Ratio Weight (GPRW) of 0.9** and a **Tenure Weight (TW) of 1.0**. Eligible operators receive a **20% discount** on new ArNS name registrations, lease extensions, lease upgrades, and undername purchases. A gateway's [operations address](/build/run-a-gateway/manage/operations-address) can claim the same discount on the gateway's behalf. ## Next Steps Congratulations! You now understand the complete ArNS pricing system. Ready to get started? }> See current ArNS pricing in real-time with the live pricing chart. }> Visit arns.ar.io to register your first name and explore the pricing in action. }> Learn about ar.io gateways and how they integrate with ArNS. }> Start building applications that leverage ArNS for decentralized naming. # Returned Names (/learn/arns/returned-names) ## Overview When an ArNS name expires or is voluntarily returned to the protocol, it enters a **Returned Name Dutch Auction** before becoming available for standard registration. This mechanism prevents name squatting at expiry and provides fair pricing through a time-decaying premium. ## Dutch Auction Mechanics Returned names start at a high premium and decay to the base price over a return window: - **Starting price**: premium above the base registration price - **Ending price**: standard registration price - **Decay**: decreases over time until standard pricing resumes ```mermaid graph LR A[Name Expiredor Released] --> B[Dutch AuctionStarts at 50x] B --> C[Price DecaysOver Time] C --> D[ReachesStandard Price] D --> E[StandardRegistration] ``` ## Revenue Split When a returned name is purchased during the Dutch auction: - **50%** goes to the **protocol balance** (funds epoch rewards) - **50%** goes to the **previous owner** (the ANT holder at the time of return) This incentivizes name owners to voluntarily release names they no longer need, since they receive half the resale value. ## How Names Enter the Returned Pool ### Lease Expiration When a leased name's term ends and the grace period elapses without renewal or conversion to permanent ownership, the name enters the returned pool. ### Voluntary Release Permanent name owners can voluntarily release their name back to the protocol. This places the name in the returned pool. ## Lifecycle | Phase | Duration | Price | Action | |-------|----------|-------|--------| | **Active lease** | As registered | N/A | Name is in use | | **Grace period** | After expiry | N/A | Owner can renew or convert | | **Dutch auction** | Return window | Premium → standard price | Anyone can purchase | | **Standard registration** | Indefinite | Base price | Normal ArNS purchase | ## Querying Returned Names Use the SDK to check available returned names and their current auction price: ```typescript const ario = ARIO.mainnet(); // Get all active returned names const returnedNames = await ario.getArNSReturnedNames({ limit: 100, sortBy: 'endTimestamp', sortOrder: 'asc', }); // Get a specific returned name const name = await ario.getArNSReturnedName({ name: 'example' }); // Check current cost (includes auction premium) const cost = await ario.getTokenCost({ intent: 'Buy-Name', name: 'example', type: 'permabuy', }); ``` # Architecture (/learn/gateways/architecture) ar.io gateways are data access layers built on top of Arweave. They make permanent data easier to retrieve, cache, index, and serve through standard web interfaces. ```mermaid graph TB CLIENTS[Apps, browsers and agents] subgraph Gateway ["ar.io Gateway"] ENVOY[Envoy] subgraph Services ["Services"] API[Core ServiceGateway API and workers] OBS[Observer] SIDE[index-swarm sidecarwith trackerOptional] ENG[Torrent engineOptional] EXP[index-exportOptional, Release 85] end subgraph Local ["Data and Storage Layer"] SQL[(SQLitecore · data · bundlesmoderation · chunks)] REDIS[(RedisChain and ArNS cache)] CH[(ClickHouseOptional GraphQL history)] FS[Files on diskchunks and contiguous datadata/indexes: root-TX and L1 bands] end end subgraph External ["External Network"] ARWEAVE[(Arweave nodes)] PEERS[Other ar.io gateways] NETWORK[ar.io networkRegistry and observations] end CLIENTS -->|HTTP| ENVOY ENVOY --> API API --> SQL API --> REDIS API --> CH API --> FS SIDE -->|bands| FS SIDE -->|torrents| ENG SQL -.->|records| EXP CH -.->|records| EXP EXP -.->|bands| FS API -->|blocks, chunks, data| ARWEAVE API -->|data| PEERS OBS -->|test requests| PEERS OBS -->|reports| NETWORK SIDE -->|signed bands| PEERS ENG -->|pieces| PEERS classDef database fill:#3b82f6,stroke:#1d4ed8,stroke-width:2px,color:#fff classDef service fill:#10b981,stroke:#059669,stroke-width:2px,color:#fff classDef proxy fill:#f59e0b,stroke:#d97706,stroke-width:2px,color:#fff classDef external fill:#8b5cf6,stroke:#7c3aed,stroke-width:2px,color:#fff classDef storage fill:#ef4444,stroke:#dc2626,stroke-width:2px,color:#fff classDef optional fill:#10b981,stroke:#059669,stroke-width:2px,color:#fff,stroke-dasharray: 5 5 class SQL,REDIS,CH database class API,OBS service class SIDE,ENG,EXP optional class ENVOY proxy class CLIENTS,PEERS,ARWEAVE,NETWORK external class FS storage ``` ## Core Technology Stack ar.io gateways are built from these parts: ### Runtime and Language - **Node.js**: The primary runtime environment for all gateway services - **TypeScript**: Core services written with flexible interfaces for customization - **Event-driven architecture**: Enables efficient handling of concurrent operations ### Data Storage - **SQLite**: Five databases handle different aspects of gateway operations: - `core`: Arweave blocks and transactions - `data`: content hashes and cache metadata - `bundles`: ANS-104 data items - `moderation`: blocked content - `chunks`: chunk metadata - **Redis**: Cache for chain data and ArNS resolutions - **ClickHouse** (optional): Columnar store for GraphQL queries over long stable history - **File system storage**: Cached chunks, transaction headers and contiguous data - **CDB64 indexes**: Read-only files that map a data item to the transaction that holds it, kept current by [Index Sharing](/learn/gateways/index-sharing) when it is enabled. Shared indexes live under `data/indexes`, as root-TX bands and, from Release 85, L1 bands. From Release 85 the optional `index-export` service builds a publishing gateway's bands from its own records ### Processing Model - **Worker-based concurrency**: Specialized workers handle different background tasks - **Event-driven processing**: Loosely coupled components communicate via events - **Streaming data handling**: Minimizes memory overhead for large data operations ## Key Architectural Decisions Several important design decisions shape how ar.io gateways operate: ### Data Retrieval Strategy ar.io gateways use a **hierarchical fallback system** for data retrieval: 1. **Trusted gateways**: Prioritize data from verified, high-performance peers 2. **ar.io network**: Fetch from other ar.io gateways 3. **Chunks data items**: Reconstruct data from individual chunks when needed 4. **Transaction data**: Fall back to raw Arweave transaction data This approach improves availability while optimizing for speed and reliability. ### Verification and Trust Model - **Multi-level cryptographic verification**: Data integrity is verified at multiple points - **Trust hierarchy**: Cached verified data → trusted cached data → network streams - **Self-healing mechanisms**: Automatic recovery and re-verification of corrupted data - **Verification headers**: HTTP headers indicate the verification status of returned data ### Serving Capabilities Gateways expose a serving layer for applications and end users. This layer includes byte-range requests, signed and verifiable responses, x402 paid access, peer routing, content moderation hooks, and [Index Sharing](/learn/gateways/index-sharing), through which gateways publish and subscribe to signed indexes that locate data items. These capabilities let operators tune how data is delivered while preserving verifiability and local operator choice. ### Worker Specialization Different background workers handle specific responsibilities: - **Block synchronization workers**: Keep the gateway aligned with Arweave blocks - **Bundle processing workers**: Extract and index ANS-104 data items - **Data verification workers**: Check cached data integrity - **Maintenance workers**: Perform cleanup and optimization tasks ## Scalability and Configuration ar.io gateways are designed to scale from small personal deployments to large enterprise installations: ### Modular Architecture Gateway services can be independently configured or disabled based on operator needs: - **Data serving**: Serve cached data to applications - **Data indexing**: Index and process new Arweave data - **Bundle processing**: Handle Layer 2 bundled transactions - **ArNS routing**: Resolve ArNS names ## Core Philosophy: Builder Independence ar.io gateways are built so that builders can use Arweave without relying on any central resource. This shows in three ways: ### Extensibility Through Modularity Gateways are designed as extensible platforms that operators can customize through **[Extensions](/build/extensions/)**, sidecar services, and plugin architectures for specialized functionality. ### Data Sovereignty Operators maintain complete control through **[Data Retrieval](/learn/gateways/data-retrieval)** strategies and **[Data Verification](/learn/gateways/data-verification)** systems that ensure independence from trusted intermediaries. ### Network Resilience The modular design creates a resilient ecosystem where distributed infrastructure and customizable trust models prevent single points of failure. This architecture helps builders use Arweave without depending on a single infrastructure provider. ## Explore Gateway Capabilities } /> } /> } /> } /> # Data Retrieval (/learn/gateways/data-retrieval) Ar.io gateways retrieve and serve Arweave data from multiple sources. They prefer fast local or trusted sources when available, then fall back to broader network peers, chunks, or Arweave nodes as needed. ## How Gateways Retrieve Data When a gateway needs to serve data, it follows a hierarchical retrieval pattern, trying each source in order until the data is successfully retrieved: ```mermaid graph TD REQUEST[Data Request] --> CACHE{Local Cache?} CACHE -->|Hit| SERVE[Serve Data] CACHE -->|Miss| TRUSTED[Trusted Gateways] TRUSTED -->|Fail| NETWORK[ar.io] NETWORK -->|Fail| LOCATE[Locate the item:root transaction and offset] LOCATE --> CHUNKS[Chunk Assembly] CHUNKS -->|Fail| ARWEAVE[Arweave Nodes] LOOKUP[(Root TX lookuplocal DB · CDB64 indexespeers · GraphQL)] -.-> LOCATE TRUSTED -->|Success| VALIDATE NETWORK -->|Success| VALIDATE CHUNKS -->|Success| VALIDATE ARWEAVE -->|Success| VALIDATE VALIDATE{Valid?} -->|Yes| STORE[Cache & Serve] VALIDATE -->|No| NEXT[Try Next Source] classDef source fill:#2563eb,stroke:#1d4ed8,stroke-width:2px,color:#fff classDef process fill:#16a34a,stroke:#15803d,stroke-width:2px,color:#fff classDef database fill:#3b82f6,stroke:#1d4ed8,stroke-width:2px,color:#fff class TRUSTED,NETWORK,CHUNKS,ARWEAVE source class VALIDATE,SERVE,STORE,LOCATE process class LOOKUP database ``` ## Data Sources Ar.io gateways can retrieve data from multiple sources, each with different characteristics: ### 1. Trusted Gateways - **Purpose**: Peer-to-peer data sharing between verified ar.io gateways - **Benefits**: Distributed redundancy, load balancing, network resilience - **Trust Mechanism**: Operator-defined trust settings, observed performance, and reciprocity - **Selection**: Prioritized based on local gateway configuration ### 2. ar.io (Untrusted Peers) - **Purpose**: Broader network of ar.io gateways without established trust - **Benefits**: Geographic distribution, expanded data availability - **Selection**: Chosen based on availability, configuration, and routing strategy - **Validation**: Verification is important because the peer may not be trusted ### 3. Chunk Assembly - **Purpose**: Direct reconstruction from Arweave chunks via known offsets - **Benefits**: Data integrity guarantee, no intermediary trust required - **Process**: Fetches individual chunks efficiently and assembles them into complete data - **Optimization**: Uses offset awareness for faster chunk retrieval - **Locating**: A data item sits inside a bundle, so the gateway first looks up which transaction holds it and at what offset. Its own database answers for what it has indexed; [CDB64 indexes](/build/run-a-gateway/manage/cdb64), kept current by [Index Sharing](/learn/gateways/index-sharing), answer locally for the rest; peers and GraphQL are asked otherwise ### 4. TX Data - **Purpose**: Direct access to transaction data from Arweave nodes - **Benefits**: Authoritative data source, complete historical access - **Trade-off**: Higher latency but guaranteed availability - **Use Case**: Final fallback when other sources fail ## Retrieval Strategies Gateways employ different strategies based on the use case: ### On-Demand Retrieval Optimized for user requests with emphasis on speed: 1. **Priority order**: Trusted Gateways → Untrusted Peers (ar.io) → Chunks Assembly → Arweave 2. **Aggressive timeouts**: Quick fallback to next source 3. **Parallel attempts**: May query multiple sources simultaneously 4. **Response streaming**: Begin serving data as soon as available ### Background Retrieval Used specifically for unbundling and verification processes: 1. **Unbundling operations**: Extracting individual data items from ANS-104 bundles 2. **Data verification**: Checks the integrity of retrieved data 3. **Integrity focus**: Prefers authoritative sources for accurate processing 4. **Relaxed timeouts**: Allows for slower but reliable retrieval during verification 5. **Verification priority**: Extensive validation before caching verified data ## Trust and Validation ### Peer Trust Management Gateways can maintain trust relationships with peer gateways: ```mermaid graph TD PEER[Peer Gateway] --> METRICS[Performance Metrics] METRICS --> LATENCY[Response Time] METRICS --> SUCCESS[Success Rate] METRICS --> VALIDITY[Data Validity] LATENCY --> SCORE[Trust Score] SUCCESS --> SCORE VALIDITY --> SCORE SCORE --> SELECTION{Peer Selection} SELECTION -->|High Trust| PREFER[Preferred] SELECTION -->|Medium Trust| NORMAL[Normal] SELECTION -->|Low Trust| AVOID[Avoided] classDef metric fill:#7c3aed,stroke:#6d28d9,stroke-width:2px,color:#fff class LATENCY,SUCCESS,VALIDITY metric ``` Trust factors include: - **Response performance**: Latency and throughput metrics - **Success rates**: Percentage of successful requests - **Data validity**: Cryptographic verification results - **Reciprocity**: Mutual data sharing behavior ### Data Validation Process Every piece of retrieved data undergoes validation: 1. **Hash Verification**: Computed hash must match expected value 2. **Merkle Proof Validation**: Chunks proven against transaction root 3. **Signature Verification**: Transaction signatures validated 4. **Size Confirmation**: Data size matches header declaration ## Why Multi-Source Retrieval Matters ### For Gateway Operators - **Reduced infrastructure costs**: Use peer resources - **Improved reliability**: Multiple fallback options - **Better performance**: Optimal source selection - **Network effects**: Benefit from collective infrastructure ### For Users - **Faster access**: Data served from optimal source - **High availability**: Multiple paths to data - **Geographic optimization**: Nearby sources preferred - **Consistent experience**: Transparent source selection --- The data retrieval system is central to ar.io's mission of providing reliable, performant access to the permaweb. Multiple retrieval paths help keep permanent data accessible through a distributed gateway network. ## Related Gateway Concepts } /> } /> } /> } /> # Data Verification (/learn/gateways/data-verification) Ar.io gateways verify that retrieved and cached data matches what was committed to Arweave. Verification helps users receive authentic, uncorrupted data without trusting a single gateway operator. Gateway data verification is one layer of ar.io's broader verification architecture. For how gateway verification composes with signed response claims, client-side verification, and OIP accountability, see [Verification and Accountability](/learn/verification). ## How Gateways Verify Data Data verification uses Arweave data roots, hashes, and Merkle proofs to check that cached data matches what was originally stored. A gateway can verify data before serving it or re-import data when verification fails: ```mermaid sequenceDiagram participant Scheduler participant Worker as DataVerificationWorker participant ContigIndex as ContiguousDataIndex participant RootTxIndex as DataItemRootTxIndex participant DataRootComp as DataRootComputer participant DataSource as ContiguousDataSource participant Importer as DataImporter participant Bundler as BundleQueue Note over Scheduler,Bundler: Data Discovery Phase Scheduler->>Worker: Triggers queueRootTx() periodically Worker->>ContigIndex: getVerifiableDataIds() ContigIndex-->>Worker: Returns list of data IDs loop For each dataId Worker->>RootTxIndex: getRootTxId(dataId) RootTxIndex-->>Worker: Returns rootTxId Worker->>Worker: Enqueue rootTxId if not processed end Note over Scheduler,Bundler: Verification Phase Worker->>ContigIndex: getDataAttributes(rootTxId) ContigIndex-->>Worker: Returns attributes (indexedDataRoot, hash) alt indexedDataRoot is present Worker->>DataRootComp: computeDataRoot(rootTxId) DataRootComp->>DataSource: getData(rootTxId) DataSource-->>DataRootComp: Returns data stream DataRootComp-->>Worker: Returns computedDataRoot alt computedDataRoot matches indexedDataRoot Worker->>ContigIndex: saveVerificationStatus(rootTxId) ContigIndex-->>Worker: ✓ Verification Success Note over Worker: Data is now verified and cached for serving else computedDataRoot does NOT match Worker->>Importer: queueItem({id: rootTxId}, priority=true) Importer-->>Worker: Queued for re-import from Arweave end else indexedDataRoot is MISSING Worker->>Bundler: queueBundle({id: rootTxId}) Bundler-->>Worker: Queued for bundle unbundling end ``` **The Verification Workflow:** At a high level, verification moves through discovery, retrieval, cryptographic computation, comparison, and recovery: **1. Discovery Phase** - Periodically scan for unverified data items - Priority-based queue management (higher priority items first) - Track retry attempts for failed verifications **2. Data Retrieval** - Fetch data attributes from gateway storage - Retrieve the complete data stream - Gather metadata needed for verification **3. Cryptographic Computation** - Calculate Merkle data root from actual data stream - Generate cryptographic proofs using the same algorithm as Arweave - Create verifiable hash chains **4. Root Comparison** - Compare computed root against indexed root in database - Verify data hasn't been corrupted or altered - Validate chunk integrity against Merkle proofs **5. Action Based on Results** - **Success**: Mark data as verified with timestamp - **Failure**: Trigger re-import from Arweave or unbundle from parent - **Error**: Increment retry counter and requeue for later ## Verification Types Ar.io gateways handle different types of data verification based on the data's origin: ### Transaction Data Verification For individual Arweave transactions: - **Direct root validation** against transaction data roots stored onchain - **Complete data reconstruction** from chunks to ensure availability - **Cryptographic proof** that data matches what was originally stored ### Bundle Data Verification For ANS-104 data bundles (collections of data items): - **Bundle integrity checks** to verify the container is valid - **Individual item verification** within each bundle - **Recursive unbundling** when verification fails to re-extract items - **Nested bundle support** for bundles containing other bundles ### Chunk-Level Validation At the most granular level: - **Merkle proof validation** for individual data chunks - **Sequential integrity** ensuring chunks form complete data - **Parallel verification** of multiple chunks for performance ## Why Verification Matters ### Cryptographic Trust Foundation - **Mathematical Proof**: Merkle tree cryptography proves data integrity - **Independent Validation**: Multiple gateways verify the same data independently ### Data Integrity Guarantees - **Tamper Detection**: Any alteration to data is immediately detectable - **Corruption Recovery**: Automatic healing of corrupted data through re-import ### Gateway Reliability - **Continuous Monitoring**: Ongoing verification catches issues before users encounter them - **Self-Healing System**: Automatic recovery mechanisms maintain data availability - **Transparent Operations**: Verification status and timestamps provide audit trails ## Explore Gateway Systems } /> } /> } /> } /> # Gateway Registry (/learn/gateways/gateway-registry) ## Overview The Gateway Address Registry (GAR) is the public registry of ar.io gateways. It records which gateways have joined the network, how they can be reached, and the metadata applications need to discover and evaluate them. Gateway registry state is maintained by the ar.io Solana programs. Registered gateways are identified by Solana addresses and publish service information such as hostname, protocol settings, staking state, and operator-provided metadata. The gateways.ar.io portal displays gateways currently in the network, including stake, performance, and operational metadata. ## Joining the Registry To join the ar.io network, a gateway operator registers their gateway and locks ARIO as operator stake. Registration connects the gateway's public endpoint, operator address, observer address, and service metadata to the onchain registry. Once registered, a gateway can become eligible for network incentives, delegation, and observation through OIP. Gateway operators also need enough SOL to pay Solana transaction fees for network interactions. The registry has a maximum capacity of **3,000 gateways**. Each gateway requires a **20,000 ARIO** minimum network-join stake, and each gateway can have up to **10,000** unique delegated stakers. ## Leaving the Registry A registered gateway is always in one of three states: **joined**, **leaving**, or **gone**. A gateway operator can leave voluntarily, and a gateway removed through [pruning](/learn/oip/pruning) leaves the same way: 1. **Leaving**: The gateway stops being eligible for new rewards and enters a leave window of at least 90 days, during which any delegated stake on it still has to be withdrawn. 2. **Gone**: Once the leave window has expired and all delegated stake has been claimed, a permissionless instruction closes the gateway's registry entry and frees its slot. A gone gateway has no registry entry left; its operator has to register again to rejoin. ## What the Registry Enables ### Discovery Apps, users, and other gateways can use the registry to find gateways by endpoint, service metadata, stake, observed performance, and supported capabilities. ### Incentive Participation The registry connects gateway identity to staking, delegated stake, performance history, and reward eligibility. OIP uses this information to evaluate gateways and distribute rewards. ### Network Transparency Gateway information is publicly visible through Solana state and network tooling such as [gateways.ar.io](https://gateways.ar.io). This makes participation, performance, and configuration easier to inspect. ### Operator Choice Gateways can specialize. Some may focus on fast public access, some on indexing, some on private infrastructure, some on paid access, and some on specific moderation or compliance policies. ## Relationship to OIP The registry is the set of gateways that OIP can observe and evaluate. Registered gateways are periodically checked for availability, correctness, and ArNS resolution behavior. Reliable gateways can earn rewards, while gateways that repeatedly fail can lose eligibility and eventually be removed through pruning. ## Recap - The Gateway Address Registry is the public source of network gateway metadata. - Gateways join by registering service details and locking operator stake. - Registered gateways can participate in OIP, receive delegations, and become eligible for rewards. - Apps can use registry data to discover, filter, and route through gateways. - Poorly performing gateways can be pruned from the network after sustained failure. - A gateway moves from joined to leaving to gone as it exits the registry, freeing its slot once it is gone. ## Explore the Gateway Ecosystem } /> } /> } /> } /> # Index Sharing (/learn/gateways/index-sharing) Most data on Arweave today is uploaded as **data items**, packed many at a time into bundles. To serve one, a gateway first has to answer a question: *which Arweave transaction holds this item, and where inside it?* A gateway that can't answer locally has to ask other gateways or search the network, which is slow and often fails. **Index Sharing** lets gateways share the answer. A gateway that has built a good index publishes it; other gateways download it and answer lookups in milliseconds. Because every piece is signed and hashed, a gateway can use another gateway's index without trusting that gateway's server. For an operator, that means data items resolve from local disk instead of a round trip to other gateways or GraphQL. The answer often includes the item's byte offsets, which lets the gateway fetch and verify just that item. It is optional and off by default: a gateway turns it on with one setup script, for about 21 GB of disk. ## How It Works ```mermaid sequenceDiagram participant Sub as Your gateway participant Reg as Gateway registry participant Pub as Publishing gateway Sub->>Reg: Look up the publisher's wallet Reg-->>Sub: Its URL and observer key Sub->>Pub: GET /ar-io/indexes Pub-->>Sub: Signed publication Note over Sub: Check the signature against the registered key Sub->>Pub: Fetch each band file by its SHA-256 Pub-->>Sub: Files Note over Sub: Check every file's digest, then install ``` ### Bands An index is split into **bands**, each covering a range of block heights: older bands rarely change, while the band at the chain tip is rebuilt as new data arrives. A subscriber downloads only the bands it doesn't have, installs the newest first (most lookups are for recent data), and replaces a band without ever pausing lookups. There are two kinds of band. A **root-TX band** is a [CDB64 index](/build/run-a-gateway/manage/cdb64) that maps a data item to the transaction holding it; this is the kind a gateway loads and answers lookups from. From Release 85, which is not released yet, a publisher can also offer **L1 bands**: the Arweave base layer (blocks, transactions, tags and wallets) in Parquet. A gateway doesn't serve from them. A new gateway imports its L1 index from them instead of indexing the chain block by block, and apps query them in place for analytics. ### The Publication A publisher serves one signed **publication** at `/ar-io/indexes`, listing every band and every file in it, with each file's size and SHA-256. It is signed with the gateway's **registered observer key**, the same key it uses for [observation](/learn/oip), so anyone can check who signed it against the [gateway registry](/learn/gateways/gateway-registry). ### What a Subscriber Checks Before a single byte is used, a subscriber confirms that: 1. The publication is signed by the key the registry lists for that publisher's wallet 2. It names that same publisher 3. It isn't older than one already seen, so a cache or mirror can't roll it back 4. Every file matches the SHA-256 the signed publication names 5. Every band is a well-formed index A server in between, whether a mirror, a CDN or a cache, can delay or withhold files, but it can't change them without being caught. ### Files Are Content-Addressed Each band file can be fetched by its SHA-256 (`/ar-io/indexes/blob/`). The address can never change meaning, so those responses are safe for any cache to keep, and the same file can come from any server that has it. ## Bands over BitTorrent Because every file is checked against the signed publication, it doesn't matter who delivers the bytes. That makes peer-to-peer delivery safe, and gateways that run the optional torrent engine move bands over **BitTorrent**. Without the engine, everything moves over HTTP as shown above. ### Why BitTorrent Over HTTP, every subscriber downloads every band from the publisher, so the publisher's upload grows with each gateway that subscribes, and publishers rate-limit and charge ([x402](/learn/gateways/x402-payments)) for those downloads like any data. Over BitTorrent, subscribers also serve each other. A new band reaches the network once from the publisher and then spreads between peers, so the publisher's cost stays roughly flat as more gateways subscribe, and subscribers download from peers for free. That is what lets a large index be shared across the network without one gateway carrying all of it. ### How a Band Moves ```mermaid flowchart LR A[Signed publication] --> T[Check the .torrentagainst the signed infohash] T -- matches --> B[Fetch from peers] T -- doesn't match --> D B -- peers stall --> C[Add the publisher's WebSeed] B -- fails or times out --> D[Fetch over HTTP] C -- fails or times out --> D B -- done --> E[Re-hash every file, install, seed] C -- done --> E D -- done --> E classDef source fill:#2563eb,stroke:#1d4ed8,stroke-width:2px,color:#fff classDef process fill:#16a34a,stroke:#15803d,stroke-width:2px,color:#fff class B,C,D source class T,E process ``` 1. **Peers first.** A publisher offers each band as a torrent and seeds it. A subscriber downloads the band from whichever gateways already hold it. Nobody charges for peer transfers. 2. **The WebSeed, only when peers stall.** If nothing arrives for a while, the subscriber also turns on the publisher's **WebSeed**, an HTTP address that serves the same bytes to a torrent client. It is subject to the publisher's rate limits and payments, so it is used only when peers are not delivering. 3. **HTTP as the fallback.** If the engine is down, errors, or makes no progress for too long, the band is fetched over HTTP instead, in the same poll. 4. **Every subscriber seeds.** Once a band is installed, the subscriber seeds it too, however it arrived. The more gateways subscribe, the less any one publisher has to upload. Seeding is part of turning the engine on, and it costs the seeder upload bandwidth, so every node bounds it: by default 10 MB/s and 100 GB a day. A gateway that would rather not upload at all subscribes over HTTP only. ### Publishers of the Same Bytes Share One Swarm A band's torrent is built only from its files: their names, sizes and digests. Nothing about the publisher goes in, not even the band's id. So two gateways publishing the same bytes produce the same torrent, with the same **infohash** (the torrent's identifier), and their subscribers all join one swarm. ### What a Subscriber Trusts A `.torrent` file isn't signed, so a subscriber trusts only the infohashes in the signed publication and re-hashes every file against its signed SHA-256, exactly as over HTTP. [Bands as Torrents](/build/advanced/index-publications#bands-as-torrents) shows these checks. The torrent engine runs isolated from the rest of the gateway: it can't reach the gateway's other services, and it can't change the bands the gateway serves. A publisher also runs a **closed tracker** that helps peers find each other and answers only for the bands it offers. Peers find one another without it too, through the BitTorrent DHT and peer exchange. ## What Index Sharing Is Not - **It doesn't replace data verification.** An index says where an item lives; the gateway still [verifies the data itself](/learn/gateways/data-verification) when it serves it. - **It isn't storage.** Bands are indexes, not the data they point to: about 21 GB for the root-TX index turbo-gateway.com publishes. - **It isn't complete.** An index answers only for the bundles its publisher indexed, and the band at the chain tip is rebuilt periodically, so the newest items aren't in it yet. A miss costs nothing: the gateway falls back to its usual lookup. - **It isn't one provider.** Any registered gateway can publish. The turbo-gateway.com gateway publishes a root transaction index today, and subscribers can follow more than one publisher. - **BitTorrent isn't required.** It is an optional transport. A gateway without the torrent engine publishes and subscribes over HTTP. ## Explore Index Sharing } /> } /> } /> } /> # ar.io Gateways (/learn/gateways) ## What are Gateways? Ar.io gateways are infrastructure nodes that make Arweave data easy to access from web apps, APIs, and users. They retrieve data from Arweave, cache and index it, resolve ArNS names, and expose standard HTTP interfaces for the permaweb. Gateways do not replace Arweave. Arweave provides permanent storage and data availability guarantees; gateways make that data fast, discoverable, and practical to use. ## Core Responsibilities ### Data Access - **Retrieve permanent data** from Arweave, peer gateways, local cache, or chunk-level sources - **Serve content over HTTP** using familiar web patterns - **Optimize delivery** through caching, streaming, and source selection ### Indexing and Discovery - **Index transactions and bundled data items** so apps can search and retrieve content efficiently - **Resolve ArNS names** into the data records and routes they point to - **Expose gateway metadata** so users and applications can discover available infrastructure ### Network Participation - **Register in the Gateway Address Registry** to become discoverable as an ar.io network gateway - **Stake ARIO** to participate in network incentives and signal operational commitment - **Participate in OIP** by being observed, reporting observations when selected, and earning rewards for reliable service ## What Gateways Are Not ### Not Storage Providers - **They do not create Arweave permanence**: permanence comes from Arweave itself - **They do not need to store everything**: operators choose what to cache, index, and serve - **They do not control user data**: data ownership remains with the original publisher and underlying protocols ### Not Application Compute - **They do not run protocol logic**: ar.io protocol state, staking, ArNS, and incentives are coordinated by Solana programs - **They do not process arbitrary app logic**: gateways focus on data access, indexing, routing, and delivery ### Not a Single Service - **Anyone can run one**: the gateway network is open to independent operators - **Apps can choose among gateways**: routing can consider availability, geography, performance, policy, and payment requirements - **Operators can specialize**: gateways may differ in scale, indexing choices, moderation policy, payment setup, and extensions ## Explore Gateways } /> } /> } /> } /> } /> } /> # x402 Payments (/learn/gateways/x402-payments) ar.io gateways support the x402 payment protocol, so clients can pay for data access in cryptocurrency. Operators can recover infrastructure costs, and users who need more bandwidth can pay for it. ## What is x402? **x402** is an open-source payment protocol from Coinbase. It uses the HTTP 402 "Payment Required" status code to let clients pay for web APIs in cryptocurrency, without accounts or card payments: ### Key Features - **Zero protocol fees**: No intermediaries or payment processing costs - **Fast settlement**: Payments settle in approximately 2 seconds - **Minimal integration**: Simple HTTP status codes and headers - **Privacy-focused**: No account creation or personal information required - **Chain-agnostic**: Works with multiple blockchain networks - **Token-agnostic**: Supports various cryptocurrencies ### How x402 Works The x402 protocol uses an HTTP flow: ```mermaid sequenceDiagram participant Client participant Gateway participant Facilitator Client->>Gateway: Request data Gateway-->>Client: 402 Payment Required, with payment requirements Note over Client: Signs a payment authorization Client->>Gateway: Same request, with X-Payment header Gateway->>Facilitator: Verify payment Facilitator-->>Gateway: Valid Gateway-->>Client: Requested data Gateway->>Facilitator: Settle payment Note over Facilitator: Transfers USDC on-chain ``` The client never talks to the facilitator. The gateway verifies and settles the payment itself, and returns the settlement receipt to the client in the `X-Payment-Response` header. ## ar.io Gateway Integration ar.io gateways implement x402 with **USDC (USD Coin) on the Base blockchain**, priced per byte of data served. It works as part of the gateway's rate limiter: a client pays only once it has used its free allowance. ### Payment Flow in ar.io Gateways 1. **Free Tier Access**: Users consume regular tokens from their rate limit buckets 2. **Rate Limit Exceeded**: When limits are reached, gateway sends 402 Payment Required response 3. **Payment Processing**: Users can make USDC payments to continue access 4. **Premium Access**: Payments add paid tokens with configurable multiplier (default 10x capacity) 5. **Priority Consumption**: Regular tokens consumed first, then paid tokens 6. **Resource Bypass**: Paid requests bypass per-resource limits (only IP limits apply) ### Browser Paywall & Onramp Integration For mainnet deployments, gateways can integrate Coinbase Onramp to enable easy USDC purchases directly in the browser paywall. This requires Coinbase Developer Platform (CDP) API keys, which are separate from payment facilitator authentication. With Onramp, users can buy USDC without leaving the gateway's paywall page. **Note**: CDP keys are optional for testnet but required for mainnet when using Coinbase facilitators. These keys enable the browser-based purchasing experience and are not related to payment verification or settlement. ## Network Options ar.io gateways support two Base blockchain networks for x402 payments: | Feature | Base Sepolia (Testnet) | Base (Mainnet) | | ----------------------- | --------------------------------- | --------------------------------- | | **USDC** | Free testnet USDC (faucet) | Real USDC (costs money) | | **CDP API Key** | Not required | Required for official facilitator | | **Default Facilitator** | https://x402.org/facilitator | Must configure | | **Use Case** | Development, testing | Production monetization | | **Configuration** | `X_402_USDC_NETWORK=base-sepolia` | `X_402_USDC_NETWORK=base` | ## Rate Limited Endpoints The x402 payment system applies to data egress endpoints where gateways serve actual content: ### Data Serving Endpoints - **Transaction/Data Item requests**: `/:txid` and `/:txid/path` - **Raw data requests**: `/raw/:txid` - **ArNS resolved content**: All requests resolved through ArNS names - **Farcaster frames**: `/local/farcaster/frame/:txid` - **Chunk requests**: `GET /chunk/:offset` and `GET /chunk/:offset/data` (fixed size pricing) - **Index Sharing byte routes**: band files by name, by digest, and the WebSeed under `/ar-io/indexes`. The publication document at `/ar-io/indexes` and the `.torrent` files are free ### Not Rate Limited - GraphQL queries (`/graphql`) - Chunk POST requests (`POST /chunk`) - Administrative and informational endpoints under `/ar-io/*`, except the Index Sharing byte routes above ## Benefits for Gateway Operators ### Sustainable Infrastructure - **Cost Recovery**: Monetize data egress to cover operational expenses - **Premium Services**: Offer higher bandwidth tiers for power users - **Flexible Pricing**: Per-byte pricing scales with actual usage - **Revenue Generation**: Create sustainable business models ### Traffic Management - **Fair Access**: Free tier for casual users with reasonable limits - **Premium Access**: Power users can pay for additional capacity - **DDoS Protection**: Rate limiting prevents abuse while allowing legitimate paid access - **Resource Optimization**: Paid requests bypass resource limits for better performance ## Benefits for Users ### Payments Without Accounts - **No Account Required**: Payments work without registration or OAuth - **Fast Settlement**: 2-second payment confirmation - **Privacy Preserved**: No personal information collection - **Transparent Pricing**: Clear per-byte costs with minimum/maximum limits ### Flexible Access - **Free Tier**: Reasonable free access for casual users - **Pay-as-you-go**: Only pay when you need additional capacity - **Predictable Costs**: Clear pricing structure with configurable limits - **Multiple Payment Methods**: Browser-based and programmatic payment options ## Use Cases ### Content Delivery Networks - **Media Streaming**: Monetize high-bandwidth video and audio content - **File Downloads**: Charge for large file transfers - **API Access**: Premium API tiers with higher rate limits ### Developer Platforms - **Data APIs**: Monetize access to indexed blockchain data - **Archive Services**: Charge for historical data retrieval - **Real-time Feeds**: Premium access to live data streams ### Enterprise Services - **Bulk Data Access**: Corporate clients paying for high-volume access - **Priority Support**: Premium service tiers with guaranteed performance - **Custom Limits**: Tailored rate limits for specific use cases ## Explore Gateway Monetization } /> } /> } /> } /> # Epoch Pipeline (/learn/oip/epoch-pipeline) ## Overview On Solana, the ar.io epoch lifecycle is broken into six discrete, permissionless steps. Each step is a separate instruction that can be executed by anyone. All steps are **idempotent** (safe to run multiple times) and **permissionless** (anyone can crank them). This design ensures the protocol cannot be halted by a single point of failure. ## Pipeline Steps ```mermaid graph LR A[create_epoch] --> B[tally_weights] B --> C[prescribe_epoch] C --> D[save_observations] D --> E[distribute_epoch] E --> F[close_epoch] ``` > _Rent versus rewards:_ Observers pay SOL for both account rent and transaction fees when submitting observations. Closing their completed epoch accounts returns the account rent in SOL, but not the transaction fees. ARIO observer rewards are handled separately by `distribute_epoch`. ### 1. create_epoch **Initializes the epoch account and computes the reward rate.** - Creates the epoch account - Computes the epoch reward allocation from the protocol balance ### 2. tally_weights **Batched computation of gateway weights for observer selection.** - Computes composite weights for gateways: - **Stake weight**: Based on total stake (operator + delegated) - **Tenure weight**: Based on how long the gateway has been in the network - **Gateway performance**: Based on pass rate across recent epochs - **Observer performance**: Based on observation submission history - Batches work so large gateway sets can be processed safely on Solana ### 3. prescribe_epoch **Selects observers and prescribed ArNS names via weighted roulette.** - Selects observers using weighted random selection - Selects prescribed ArNS names that observers use as common test targets ### 4. save_observations **Observers submit their pass/fail observation reports.** - Each selected observer submits compact pass/fail results for tested gateways - This is the only step that requires a specific signer (the selected observer) - Observations are stored on the Epoch account ### 5. distribute_epoch **Batched reward distribution to gateways and their delegates.** - Functional gateways receive the Base Gateway Reward (BGR) - Functional observers receive the Base Observer Reward (BOR) - Deficient observers do not receive observer rewards - Operator rewards auto-compound into operator stake - Delegate rewards are tracked via the reward-per-share accumulator (settled lazily) - Leaving gateways receive 0 rewards ### 6. close_epoch **Closes completed per-observer epoch accounts and returns each account's remaining SOL rent to that observer's wallet.** This is separate from `distribute_epoch`, which distributes ARIO rewards. `close_epoch` is permissionless, so any cranker may submit it, but the cranker does not receive the rent returned from observer accounts. - Returns SOL rent from completed observer epoch accounts to their respective observers - Keeps onchain state lean over time ## Timing The pipeline steps can be executed as the epoch progresses: | Step | When | Batched? | |------|------|----------| | create_epoch | After previous epoch ends | No | | tally_weights | After create_epoch | Yes | | prescribe_epoch | After all weights tallied | No | | save_observations | During observation window | No (per observer) | | distribute_epoch | After observation window | Yes | | close_epoch | After epoch state is no longer needed | No | ## Who Cranks? A cranker is a permissionless actor that executes the epoch pipeline on Solana. Since Solana programs cannot execute on a timer, an external wallet must call each instruction to advance the epoch lifecycle. The key property is that cranking is **permissionless**: any wallet with SOL for transaction fees can run it. No ARIO tokens, gateway registration, or special authorization is required. Without crankers, the epoch pipeline would stall. Observations would not be prescribed, rewards would not be distributed, and completed epoch state would not be closed. Multiple independent crankers provide redundancy so the network is not dependent on a single operator. Cranking can run as a standalone bot that watches epoch state and submits whichever pipeline instruction is needed next. It can also be embedded directly in ar.io observers: ```bash # In your observer's .env file ENABLE_EPOCH_CRANKING=true ``` Multiple crankers can run at the same time without coordination. The first successful transaction advances the pipeline, and later attempts see that the step is already complete. Because the instructions are idempotent, duplicate calls do not double-distribute rewards or corrupt epoch state. # Observation & Incentive Protocol (/learn/oip) ## Overview The Observation and Incentive Protocol (OIP) helps maintain gateway quality through peer monitoring and performance-based rewards. Gateways are incentivized to serve data reliably while also acting as observers that evaluate their peers. The protocol runs in epochs. During each epoch, selected gateways observe other gateways, submit reports, and participate in an onchain pass/fail voting process. Functional gateways and observers become eligible for ARIO rewards, while deficient gateways and observers miss rewards and can lose future selection weight. ## Architecture Overview The protocol follows a repeatable flow: ```mermaid sequenceDiagram participant SC as ar.io Protocol participant OBS as Observer Gateway participant GW as Target Gateway participant AR as Arweave Network Note over SC,AR: Epoch Start SC->>SC: Select observers(weighted random) SC->>SC: Prescribe ArNS names SC->>OBS: Notify selection & provide names Note over SC,AR: Observation Phase loop For each gateway to test OBS->>GW: Test ArNS resolution GW-->>OBS: Response data OBS->>OBS: Score: Pass/Fail end Note over SC,AR: Reporting Phase OBS->>AR: Upload detailed JSON report AR-->>OBS: Confirm storage OBS->>SC: Submit compact pass/fail results SC-->>OBS: Confirm interaction Note over SC,AR: Evaluation & Rewards SC->>SC: Tally all observer votes SC->>SC: Tally observer votes SC->>SC: Determine rewards SC->>OBS: Distribute observer rewards SC->>GW: Distribute gateway rewards(if functional) ``` ## Epoch Cycle Each epoch follows a structured process with specific responsibilities for gateways and observers: ### Epoch Start - **ar.io protocol**: Selects observers using weighted random selection - **ar.io protocol**: Prescribes common ArNS names for observers to test - **Selected Observers**: Receive notification of selection and prescribed names ### Observation Phase - **Observers**: Test assigned gateways for ArNS resolution and response quality - **Observers**: Document pass/fail findings and failure reasons - **Target Gateways**: Respond to resolution requests and serve the requested content ### Reporting Phase - **Observers**: Upload detailed JSON reports to Arweave for transparency - **Observers**: Submit compact onchain pass/fail results for consensus voting ### Evaluation and Distribution - **ar.io protocol**: Tallies observer votes to classify gateways - **ar.io protocol**: Distributes rewards based on performance - **Functional Gateways/Observers**: Become eligible for ARIO rewards ## Key Features - **Decentralized Monitoring**: Peer-to-peer evaluation ensures no single point of failure - **Consensus-Based Scoring**: Observer submissions determine gateway functionality - **Performance Incentives**: Only functional gateways and observers receive ARIO token rewards - **Transparent Accountability**: All reports permanently stored on Arweave and viewable at [gateways.ar.io](https://gateways.ar.io) - **Sustainable Funding**: Protocol balance funded by ArNS name purchases, aligning rewards with network usage ## What Observers Evaluate Observer criteria can evolve over time without requiring every detail to live in the protocol itself. At a high level, observers evaluate whether gateways can: - Resolve prescribed ArNS names correctly - Return expected transaction IDs and response data - Serve data with acceptable availability and responsiveness - Produce reports that can be independently inspected by users and applications --- **View Live Data**: See current observers and performance metrics at [gateways.ar.io](https://gateways.ar.io) ## Explore the Protocol } /> } /> } /> } /> # Observer Selection (/learn/oip/observer-selection) ## Epochs and Selection Timeline The ar.io network operates in epochs, during which the observer selection and evaluation process takes place. At the start of each epoch: - Observers are selected to monitor the network - Prescribed ArNS names are chosen for all observers to test - Observers may evaluate additional names or criteria as defined by the current observation process This creates a consistent evaluation framework where all observers test the same baseline names while leaving room for the observation process to evolve over time. ## Selection Process Gateways are selected as observers using weighted random selection. The process combines randomness with gateway-specific weights so observer duties are unpredictable, but still influenced by stake, tenure, and historical performance. The entropy provides cryptographic randomness for selecting: - **Observer Gateways**: Gateways chosen to perform observations - **Prescribed ArNS Names**: Common names all observers must evaluate This approach prevents manipulation while maintaining weighted probabilities based on gateway performance and commitment. ![Current epoch observers showing their observation chance (normalized composite weight) and report status](/content/observers.png) gateways.ar.io/#/observers {" "} shows the current epoch prescribed observers and arns names, as well as their submission status ## Weighted Selection Criteria Observer selection is based on **normalized composite weights** that combine multiple performance and commitment factors. These weights determine each gateway's probability of being selected as an observer for the epoch. The selection considers four key factors that are multiplied together to create a composite weight (CW): - **Stake Weight (SW)**: Financial commitment to the network - **Tenure Weight (TW)**: Length of network participation - **Gateway Performance Ratio Weight (GPRW)**: Historical gateway performance - **Observer Performance Ratio Weight (OPRW)**: Historical observer performance These weights are then normalized across all eligible gateways to create selection probabilities. For more on how performance affects those weights, see [Performance Evaluation](/learn/oip/performance-evaluation). ## Random Selection The selection process maps random values onto normalized gateway weight ranges. Gateways with higher weights have proportionally better chances of selection, while randomness prevents predictable or easily manipulated assignments. ## Fairness and Meritocracy This system ensures: - **Meritocratic Selection**: Higher-performing gateways have better selection odds - **Fair Opportunity**: All gateways maintain non-zero selection probability - **Tamper Resistance**: Entropy prevents predictable observer assignment - **Consistent Standards**: Prescribed names create common evaluation baseline The selection is saved in the contract state at epoch start to ensure that activities during the epoch do not affect selection or reward distribution. --- ## Next Steps Ready to understand how performance is evaluated? Learn about [Performance Evaluation](/learn/oip/performance-evaluation) to see how gateways are scored, or explore [Reward Distribution](/learn/oip/reward-distribution) to understand how rewards are calculated and distributed. # Performance and Weights (/learn/oip/performance-evaluation) ## Gateway Classifications Consider the following classifications: - **Functional or Passed Gateways**: are gateways that meet or surpass the network's performance and quality standards, including ArNS resolution and chunk/offset validation (if selected). - **Deficient or Failed Gateways**: are gateways that fall short of the network's performance expectations, including failures in ArNS resolution or chunk/offset validation. - **Functional or Submitted Observers**: are selected observers who diligently perform their duties and submit observation reports and onchain observations. - **Deficient or Failed Observers**: are selected observers who do not fulfill their duty of submitting observation reports and onchain observations. ## Evaluation Process At the end of an epoch, the ar.io protocol processes observer submissions to determine gateway performance through a consensus-based vote tallying system. This evaluation transforms individual observer reports into network-wide performance assessments. ### Vote Tallying and Gateway Classification After observers submit their detailed reports (see [Reporting](/learn/oip/reporting) for submission details), the protocol performs consensus calculation: **Vote Processing:** - **Data Collection**: All observer onchain observations for each gateway are collected - **Vote Counting**: Each observer submission contributes either a PASS or FAIL vote - **Majority Determination**: If enough submitted observer interactions indicate PASS, the gateway is considered Functional - **Binary Classification**: Gateways are classified as either Functional (eligible for rewards) or Deficient (ineligible for rewards) **Consensus Mechanism:** - Multiple observers evaluate each gateway independently, ensuring reliable assessment - A majority-style threshold requires broad observer agreement for positive performance determination - Binary scoring provides clear, unambiguous performance classification - Vote tallying occurs after submissions are finalized ## Weight Impact on Gateway Performance Gateway performance directly affects multiple weighted factors that influence future observer selection and overall network participation: ### Gateway Performance Ratio Weight (GPRW) A gateway's evaluation results directly impact their Gateway Performance Ratio Weight, which affects their likelihood of being selected as an observer in future epochs. - **Functional Gateways**: Increase their passed epochs count, improving their GPRW - **Deficient Gateways**: Decrease their GPRW as participated epochs increase without corresponding passes - **Observer Selection**: Higher GPRW increases chances of being selected as an observer ### Observer Performance Ratio Weight (OPRW) For gateways selected as observers, their performance in submitting reports affects future selection. - **Functional Observers**: Who submit reports increase their OPRW - **Deficient Observers**: Who fail to submit reports see their OPRW decrease - **Future Selection**: Higher OPRW improves chances of future observer selection ### Composite Weight Performance factors combine to determine overall network influence: - **Stake Weight**: Financial commitment - **Tenure Weight**: Network longevity - **Gateway Performance Ratio Weight**: Gateway reliability history - **Observer Performance Ratio Weight**: Observation duty history **Long-term Effects:** - Consistently functional gateways accumulate higher composite weights - Poor performers see diminishing influence and selection chances - Performance history creates compounding effects on network participation ## Evaluation Timeline Rewards are distributed **at the end of each epoch** by the ar.io protocol based on the tallied observer votes. The protocol processes observer submissions and distributes rewards to functional gateways and observers based on their performance during the epoch. ## Key Features - **Majority Rule**: Gateway performance is determined by majority vote from observers - **Binary Scoring**: Simple pass/fail system for clear performance assessment - **Network Confirmation**: Evaluation happens after observer submissions are finalized - **Transparent Process**: All evaluations are based on onchain data ## Consequences of Performance ### Functional Gateways - Eligible for gateway rewards - Maintain good standing in the network - Continue to be considered for observer selection ### Deficient Gateways - Ineligible for gateway rewards - Risk being marked as deficient for multiple epochs - May face additional penalties for prolonged poor performance ### Observer Performance - Functional observers receive observer rewards - Deficient observers forfeit observer rewards - Deficient observers who are also functional gateways can have their gateway reward reduced for that epoch --- ## Next Steps Ready to understand how rewards are distributed? Learn about [Reward Distribution](/learn/oip/reward-distribution), or go back to [Observer Selection](/learn/oip/observer-selection) to review the selection process. # Gateway Pruning (/learn/oip/pruning) ## Overview Gateway pruning is the mechanism that removes gateways that continue to fail network performance checks. It helps keep the Gateway Address Registry focused on infrastructure that is reachable, useful, and aligned with the network's quality expectations. Pruning is based on OIP performance results. A gateway is not removed for a single bad result, but sustained failure can make it eligible for removal. ## How Pruning Works When a gateway remains deficient for a sustained period: 1. **The gateway becomes prunable** after repeated failed observations. 2. **A permissionless instruction can remove it** from the Gateway Address Registry. 3. **The operator's required network-join stake is slashed** to the protocol balance. 4. **Eligible excess and delegated stake follow the normal withdrawal process** instead of being slashed. ```mermaid graph TD A[Gateway participates in OIP] --> B{Performance result} B -->|Pass| C[Failure history resets] B -->|Fail| D[Failure history accumulates] D --> E{Sustained failure?} E -->|No| A E -->|Yes| F[Gateway becomes prunable] F --> G[Gateway removed from registry] G --> H[Required operator stake slashed] G --> I[Eligible excess and delegated stake withdrawable] ``` ## Impact on Operators Pruning is designed to make gateway operation economically accountable. Operators are expected to keep their gateways online, correctly configured, able to resolve ArNS names, and able to participate in observation duties when selected. When a gateway is pruned, the operator loses the required stake associated with joining the network. Any eligible excess stake follows the normal withdrawal flow. ## Impact on Delegators Delegated stake is not the target of pruning slashing. If a gateway is pruned, delegated stake follows the normal withdrawal process for leaving gateways. Delegators should still monitor gateway performance because pruning can interrupt reward eligibility and require withdrawal or redelegation decisions. ## Prevention Gateway operators can reduce pruning risk by: - Maintaining reliable uptime and public reachability - Keeping ArNS resolution working correctly - Monitoring OIP performance results and gateway health - Funding the observer wallet with SOL for required transaction fees - Reviewing gateway status in [gateways.ar.io](https://gateways.ar.io) ## Related Concepts - [Gateway Registry](/learn/gateways/gateway-registry) - [Observation and Incentive Protocol](/learn/oip) - [Staking](/learn/oip/staking) # Reporting (/learn/oip/reporting) ## Observer Responsibilities Selected observers have specific duties each epoch: test gateways, document results, and submit findings through two channels. Proper completion of these responsibilities determines observer rewards and future selection chances. ## Dual Submission Process Observers must submit their findings through both channels to fulfill their duties: ### 1. Detailed Reports to Arweave - **Format**: Comprehensive JSON reports with full evaluation data - **Purpose**: Permanent audit trail and transparency - **Content**: Complete test results, timing data, and failure details ### 2. Onchain Observations to the ar.io Protocol - **Format**: List of failed gateways - **Purpose**: Efficient vote tallying for consensus - **Content**: Binary pass/fail determinations for each gateway tested ## Observer Evaluations Observers test assigned gateways against prescribed ArNS names and document their findings: ![Observer Report Overview showing multiple gateway evaluations](/content/observer-report.png) ### Evaluation Results Passing Report: Gateway successfully resolves ArNS names with expected status codes, transaction IDs, and response data. Failing Report: Gateway fails ArNS resolution tests due to ownership issues, timeouts, or missing content. Observers evaluate gateways based on: - **Gateway Wallet Ownership**: Verifies correct wallet address - **ArNS Resolution**: Tests successful name-to-transaction resolution - **Content Hash Verification**: Ensures data integrity - **Response Times**: Measures performance within limits ## Observer Rewards and Penalties Observer performance directly impacts rewards and future participation: ### Successful Observer Performance - **Observer Reward**: Observers who submit both reports and onchain observations receive the Observer Reward - **Future Selection**: Successful reporting improves Observer Performance Ratio Weight (OPRW) - **Increased Chances**: Higher OPRW increases likelihood of future observer selection and more reward opportunities ### Failed Observer Performance - **No Observer Reward**: Observers who fail to submit required reports forfeit their Observer Reward - **Gateway Penalty**: If the deficient observer is also a functional gateway, their gateway reward can be reduced for that epoch - **Reduced Selection**: Failed submissions decrease OPRW, diminishing future observer selection chances - **Lost Opportunities**: Lower selection probability means fewer chances to earn Observer Rewards ## Observer Accountability The system tracks observer performance to ensure network quality: - **Submission Tracking**: Both Arweave reports and onchain observations must be submitted - **Performance History**: Observer submission record affects future selection probability - **Reward Impact**: Consistent reporting builds credibility and increases earning potential --- ## Next Steps Ready to understand how these reports are processed? Learn about [Performance Evaluation](/learn/oip/performance-evaluation) to see how reports become votes and determine gateway rewards, or explore [Reward Distribution](/learn/oip/reward-distribution) to understand the complete incentive structure. # Distributions (/learn/oip/reward-distribution) ## Protocol Balance and Funding The ar.io network maintains a protocol balance that funds all gateway and observer rewards. This balance is primarily funded through ArNS name purchases, ensuring sustainable network incentives aligned with usage. ## Epoch Allocation Each epoch, a portion of the protocol balance is earmarked for distribution as rewards. The protocol uses this allocation to reward functional gateways and observers. ### Funding Sources - **ArNS Name Purchases**: Primary funding mechanism - fees from ArNS name registrations and renewals - **Network Genesis Allocation**: Initial ARIO tokens allocated at network launch - **Undistributed Rewards**: Rewards not claimed due to poor performance roll forward to future epochs From this allocation, two distinct reward categories are derived: ## Base Rewards ### Base Gateway Reward (BGR) This is the portion of the epoch reward allocation distributed to functional gateways. ### Base Observer Reward (BOR) Observers, due to their additional responsibilities, have a separate reward category for successfully performing observation duties. ## Distribution Based on Performance The reward distribution is contingent on the performance classifications derived from the Performance Evaluation: - **Functional Gateways**: Gateways that meet the performance criteria receive the Base Gateway Reward. - **Deficient Gateways**: Gateways falling short in performance do not receive any gateway rewards. - **Functional Observers**: Observers that fulfilled their duty receive the Base Observer Reward. - **Deficient Observers**: Observers failing to meet their responsibilities do not receive observer rewards. If they are also functional gateways, their gateway reward can be reduced for that epoch as a consequence for not performing their observation duty. ![Epoch reward distributions showing eligible vs distributed ARIO tokens](/content/epoch-distributions.png) Epoch reward distributions showing the relationship between eligible rewards (total available) and distributed rewards (actually paid out) across epochs. The difference represents rewards not distributed due to gateway or observer deficiencies. ## Epoch Pipeline On Solana, reward distribution is driven by a permissionless 6-step [epoch pipeline](/learn/oip/epoch-pipeline) rather than a single atomic operation: 1. **create_epoch** — Initialize epoch, compute reward rate 2. **tally_weights** — Batched weight computation 3. **prescribe_epoch** — Select observers and prescribed names via weighted roulette 4. **save_observations** — Observers submit pass/fail reports 5. **distribute_epoch** — Batched reward distribution 6. **close_epoch** — Reclaim rent from completed epoch accounts All steps are permissionless and idempotent — anyone can crank them, and running them multiple times is safe. ## Operator Rewards Operator rewards always **auto-compound** into the operator's stake. There is no toggle — operators must call `decrease_operator_stake` to realize rewards as liquid tokens. ## Distribution to Delegates Delegate rewards use a **reward-per-share accumulator** pattern. Rather than distributing rewards to each delegate individually each epoch, the protocol tracks a cumulative reward-per-token on each gateway. Pending rewards are settled when the delegator interacts with the protocol (e.g., delegate more, withdraw, or claim rewards). Leaving gateways receive 0 rewards for any epoch in which they have initiated withdrawal. Delegate reward distribution considers the gateway's total reward, the gateway's delegate reward share setting, and each delegate's proportional stake. Delegated rewards are added to the delegate's existing stake for that gateway and can later be withdrawn subject to normal withdrawal rules. ## Undistributed Rewards In cases where rewards are not distributed, either due to the inactivity or deficiency of gateways or observers, the allocated tokens shall remain in the protocol balance and carry forward to the next epoch. This mechanism is in place to discourage observers from frivolously marking their peers as offline in hopes of attaining a higher portion of the reward pool. Note that if a gateway (and its delegates) leaves the network or a delegate fully withdraws stake from a gateway, they become ineligible to receive rewards within the corresponding epoch and the earmarked rewards will not be distributed. ## Handling Deficient Gateways To maintain network efficiency and reduce state bloat, gateways that remain deficient for a sustained period can be removed from the network. When this happens, their minimum network-join stake is slashed to the protocol balance, while eligible excess and delegated stake follow the standard withdrawal process. --- ## Next Steps Congratulations! You now understand the complete OIP system. Ready to learn more? - **Explore Gateways** → [Gateway Documentation](/learn/gateways/) for technical details - **Learn about ArNS** → [ArNS Documentation](/learn/arns/) for naming system details - **Back to Introduction** → [OIP Introduction](/learn/oip/) to review the basics # Staking (/learn/oip/staking) ## Overview Staking tokens within ar.io serves a dual primary purpose: it signifies a public commitment by gateway operators and qualifies them and their delegates for reward distributions. In the ar.io ecosystem, "staking" refers to the process of locking ARIO tokens into protocol-controlled vaults on Solana. This act signifies an opportunity cost for the staker, acting both as a motivator and a public pledge to uphold the network's collective interests. Once staked, tokens remain locked until the staker initiates an unstake or withdrawal action, or reaches the end of the vault's lock period. It is important to note that the ARIO Token is non-inflationary, distinguishing ar.io's staking mechanism from yield-generation tools found in other protocols. Staking in this context is about eligibility for potential rewards rather than direct token yield. By staking tokens, gateway operators (and their delegates) demonstrate their commitment to the network, thereby gaining eligibility for protocol-driven rewards and access to the network's shared resources. SOL is required for Solana transaction fees when staking, delegating, redelegating, or withdrawing ARIO. ## Gateway Staking A gateway operator must stake ARIO to join their gateway to the network. This makes them eligible for protocol rewards and promotes network reliability by giving operators an economic commitment to the network's performance. The minimum network-join stake is **20,000 ARIO**. The gateway registry can hold up to **3,000 gateways**; when the registry is full, new gateways cannot join until existing gateways leave. Operators may increase their stake above the minimum required to join the network. This additional amount is known as **excess stake**. A gateway's total stake impacts the following epoch once excess stake is added or removed. Operator rewards auto-compound into operator stake. To realize rewards, operators must withdraw excess stake. If an excess stake withdrawal is still within its withdrawal period, the operator can cancel it and immediately restake the amount to the original gateway. Operators cannot self-delegate. Operator stake and delegated stake are managed through separate mechanisms. ### Gateway Pruning Gateways that repeatedly fail observation can be pruned from the network. When a gateway is removed for sustained poor performance, its minimum operator stake is slashed to the protocol balance. Any excess operator stake and delegated stake enter the standard withdrawal process. ## Delegated Staking To promote participation from a wider audience, the network allows anyone with available ARIO tokens to partake in delegated staking. Users can choose to take part in the risk and rewards of gateway operations by staking their tokens with an active gateway (or multiple gateways) through an act known as delegating. Delegators can select which gateways to stake with in gateways.ar.io — maximize their potential rewards based on operator performance, stakes, and weights ### How Delegated Staking Works **Delegated staking allows you to participate in ar.io's reward system without running your own gateway.** By staking your ARIO tokens on existing gateways, you can earn rewards while supporting network infrastructure. When you delegate stake to a gateway, you're increasing that gateway's total stake. This can improve the gateway's chances of being selected as an observer, which means more potential rewards for both the gateway operator and you as a delegator. Delegate rewards use a **reward-per-share accumulator** pattern. Rather than distributing rewards to each delegate individually each epoch, the protocol tracks a cumulative reward-per-token on each gateway. Your pending rewards are settled when you interact with the protocol (e.g., delegate more, withdraw, or claim rewards). Gateway operators can choose whether to accept delegated staking and can configure their own delegation settings within protocol limits. They may also offer non-protocol incentives, such as faster access, privileged access, or other off-chain perks. Delegated staking has these protocol limits: - Global minimum delegated stake: **10 ARIO** - Gateway-selected delegate reward share: **0% to 95%** - Maximum unique delegates per gateway: **10,000** - Delegated stake withdrawal duration: **30 days** ### Benefits - **Potential Rewards**: Become eligible for rewards without running infrastructure - **Network Participation**: Support ar.io's growth - **Flexibility**: Redelegate to different gateways as conditions change - **Low Barrier to Entry**: No technical expertise required - **Transparent Rewards**: Clear visibility into reward distribution ### Getting Started **Get ARIO Tokens** You'll need ARIO tokens in a Solana wallet to delegate. See our comprehensive guide on [How to Get ARIO Tokens](/learn/token/get-the-token) for detailed information about acquiring tokens through exchanges, swaps, and network participation. **Choose a Gateway** Research gateways on the [Gateway Portal](https://gateways.ar.io/#/staking) to find one that matches your preferences for reward sharing and performance. Look for gateways with strong uptime, competitive reward sharing percentages, and reliable operation history. **Delegate Your Stake** Use the [Gateway Portal](https://gateways.ar.io/#/staking) to delegate your tokens. Connect your Solana wallet (Phantom, Solflare, or Backpack) and the process is straightforward. **Monitor Your Rewards** Track your delegation performance and rewards through the portal's dashboard. Your rewards accumulate via the reward-per-share accumulator and are settled when you interact with the protocol. ### Important Considerations - **Gateway Performance**: Your rewards depend on the gateway's performance and observer selection - **Reward Sharing**: Gateway operators set the percentage of rewards shared with delegators - **Delegation Settings**: Gateway operators can update delegation settings; changes take effect in the following epoch - **Redelegation**: You can move your stake between gateways as network conditions change - **Withdrawal Delays**: There is a 30 day standard delay when withdrawing your delegated stake - **Gateway Exit or Removal**: If a gateway leaves or is removed from the network, delegates must claim their stake through the protocol, subject to withdrawal delays ## Stake Redelegation This feature enables existing stakers to reallocate their staked tokens between gateways, known as redelegation. Both delegated stakers and gateway operators with excess stake (stake above the minimum network-join requirement) can take advantage of this feature. Redelegation is intended to offer users flexibility and the ability to respond to changing network conditions. To balance flexibility with network stability, users are entitled to one free redelegation during each seven-day window from their last redelegation action. Additional redelegations within that window incur increasing protocol fees, capped at 60%, and fees are sent to the protocol balance. All redelegations must respect minimum stake requirements at both the source and destination gateways. Gateway minimum network-join stakes are not eligible for redelegation, but operator excess stake and eligible withdrawal vaults may be redelegated. ## Redeeming Stake for ArNS Staked tokens generally have restricted liquidity to maintain a healthy degree of stability in the network. However, an exception allows delegated stakers and gateway operators to redeem eligible staked tokens to fund specific ArNS-related services. By leveraging their stake, participants can further engage with ArNS, strengthening the name system's utilization and impact across the network. Eligible ArNS interactions include: - Purchasing a name - Extending a lease - Upgrading a lease to a permabuy - Increasing undername capacity - Requesting or setting a primary name Each ArNS purchase is funded from a single source selected by the user: liquid balance, delegated stake, a delegated stake withdrawal vault, or eligible operator excess stake. If the selected source does not cover the full purchase, the transaction fails and no balances or stakes are modified. Using eligible staked tokens for ArNS interactions does not impose an additional fee, but source gateway minimum stake requirements still apply. ## Expedited Withdrawal Fees Gateway operators and delegated stakers can shorten the standard withdrawal delay period after initiating a withdrawal or being placed into an automatic withdrawal by protocol mechanisms. This action is subject to a dynamic fee paid to the protocol balance. At any point during the withdrawal delay, users can expedite access to pending withdrawal tokens by paying a fee based on how much sooner they want to receive their funds. Once triggered, the tokens are returned immediately to the user's wallet. A gateway's minimum operator stake is not withdrawable while the gateway is active; only excess stake can be placed into withdrawal during normal operation. If a gateway voluntarily leaves the network, the minimum stake becomes an exit vault and can follow the same expedited withdrawal flow. If a gateway is removed for sustained poor performance, the minimum stake is slashed and cannot be recovered. ## Explore Staking } /> } /> } /> } /> # Get the Token (/learn/token/get-the-token) ## Acquiring ARIO Tokens ARIO exists on multiple networks: - **Solana ARIO (canonical):** The native SPL Token that powers ar.io — staking, rewards, ArNS, and governance. - **Base ARIO (bridged):** A representation of Solana ARIO on the Base L2 for EVM-based liquidity, tools, and integrations. Choose between mainnet and testnet tokens based on your requirements. ## Mainnet Tokens #### Market Trackers - **[CoinGecko](https://www.coingecko.com/en/coins/ar-io-network)** - **[CoinMarketCap](https://coinmarketcap.com/currencies/ar-io-network/)** Pick the network you want to use: ### Token Details - **Mint Address:** `DcNnMuFxwhgV4WY1HVSaSEgr92bv2b1vUvEKiNxWqHdF` - **Token Standard:** SPL Token - **Decimals:** 6 - **Explorer:** [View on Solscan](https://solscan.io/token/DcNnMuFxwhgV4WY1HVSaSEgr92bv2b1vUvEKiNxWqHdF) --- ### Exchanges #### Decentralized Exchanges (DEXs) You can swap ARIO on Solana DEXs: - **[Jupiter](https://jup.ag)** — Solana's leading DEX aggregator - **[Raydium](https://raydium.io)** — AMM and liquidity on Solana When trading on a DEX, always confirm that the token mint address matches the official ARIO mint above. --- ### Wallets ARIO is a standard SPL Token — any Solana-compatible wallet can hold it: - **[Phantom](https://phantom.app)** — Most popular Solana wallet - **[Solflare](https://solflare.com)** — Full-featured Solana wallet with staking support - **[Backpack](https://backpack.app)** — Multi-chain wallet with Solana support Most Solana wallets auto-detect ARIO once you receive tokens. To add manually, import the mint address above. See our [Wallet Setup Guide](/learn/token/wallets) for detailed instructions on configuring your wallet for ARIO. ### Token Details - **Contract ID:** `0x138746adfA52909E5920def027f5a8dc1C7EfFb6` - **Explorer:** https://basescan.org/token/0x138746adfa52909e5920def027f5a8dc1c7effb6 --- You can obtain ARIO on Base by swapping on a DEX or by bridging it from Solana. ### Bridge Bridge from Solana to Base (and vice versa) via the **[ARIO Bridge](https://swap.ar.io)**. ### Decentralized Exchanges (DEXs) Acquire Base ARIO on Base-native DEXs: - **[Hydrex](https://www.hydrex.fi/swap?tokenIn=0x833589fcd6edb6e08f4c7c32d4f71b54bda02913&tokenOut=0x138746adfa52909e5920def027f5a8dc1c7effb6)** - **[Coinbase DEX](https://www.coinbase.com/dex)** (via Coinbase mobile app or Base integration) Important: Always verify the contract address before swapping. DEXs are permissionless, and imposters can appear. ### Wallets Use your favorite Base network-enabled EVM wallet to interact with ARIO on Base. --- ### Network Participation ARIO on Solana can be earned by supporting the network through direct operation or delegated staking. **Note:** Only ARIO on Solana can be used for joining a gateway to the network and delegated staking. SOL is also required for Solana transaction fees. #### Gateway Operation Operate an ar.io gateway and earn protocol rewards: 1. **Deploy a Gateway:** Follow open-source setup instructions. 2. **Stake ARIO:** Meet the minimum stake requirement (20,000 ARIO). 3. **Serve the Network:** Provide reliable data retrieval, caching, and indexing services. 4. **Earn Rewards:** Receive ARIO for contributing to network performance and uptime. #### Delegated Staking Support a gateway operator without running infrastructure: 1. **Select a Gateway:** Use the Gateway Address Registry (GAR) to find a trusted operator. 2. **Delegate Tokens:** Stake your ARIO with them to increase their total stake. 3. **Earn Rewards:** Share in their rewards proportionally to your delegation. 4. **Unstake Anytime:** Withdraw after the standard protocol delay period. --- ### Ecosystem and Community Programs Earn ARIO through ecosystem initiatives that reward contribution and participation. #### Grants and Bounties The **ar.io Foundation** funds and supports community builders through: - **Developer Grants:** Build tools, integrations, and applications for the network. - **Bug Bounties:** Help improve network security by identifying vulnerabilities. - **Community Programs:** Contribute to education, content creation, and outreach. #### Ecosystem Growth From app development to creative projects, contributors across the permaweb can earn ARIO for helping expand and improve the network ecosystem. ## Testnet Tokens For building and testing, ar.io runs a free **[Testnet Sandbox](/build/testnet)** on Solana devnet using a **staging ARIO** token — no mainnet, no real money. - **Staging ARIO** is dispensed by the **[ar.io faucet](https://faucet.services.ar-io.dev)** (GitHub sign-in, 2,500 ARIO per claim) on Solana devnet — mint `6vTw5CysRXQ4ybbHkDUiisHWVsBeMtUzYvJqs2iqHyaN`. - You'll also need a little **devnet SOL** for transaction fees, from [faucet.solana.com](https://faucet.solana.com) or `solana airdrop 1 --url devnet`. Use staging ARIO to pay for uploads and buy devnet ArNS names in the sandbox. See **[Funds & Faucet](/build/testnet/funds-and-faucet)** to get started. --- ## Summary - **ARIO powers ar.io** — enabling staking, naming, and incentives. - **Get ARIO** on DEXs or Bridging. - **Earn ARIO** by running gateways, delegating, or contributing to ecosystem growth. # Token (/learn/token) ## Overview ARIO is the native token of the [ar.io network](https://ar.io), implemented as an [SPL Token](https://spl.solana.com/token) on Solana. It powers the network's decentralized gateway infrastructure, ArNS naming system, and incentive mechanisms. The token uses 6 decimal places (1 ARIO = 1,000,000 mARIO). ## Mainnet Addresses Use these canonical Solana mainnet addresses when verifying ARIO in wallets, explorers, integrations, or gateway configuration. ### Token | Item | Address | | --- | --- | | ARIO SPL Token mint | `DcNnMuFxwhgV4WY1HVSaSEgr92bv2b1vUvEKiNxWqHdF` | ### ar.io Programs | Program | Program ID | | --- | --- | | `ario-core` | `73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh` | | `ario-gar` | `89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf` | | `ario-arns` | `2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ` | | `ario-ant` | `2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5` | | `ario-ant-escrow` | `5HZhe9UqKL5zAsdz81nuuaxV41h8bFhudzxxBigAQndM` | ## How ARIO Is Used - **Gateway participation**: Gateway operators stake ARIO to join the network and become eligible for protocol rewards. - **Delegated staking**: Token holders can delegate ARIO to gateway operators and participate in network incentives without running infrastructure. - **ArNS names**: ARIO is used for ArNS name registration, lease extensions, undername capacity, and related naming operations. - **Protocol incentives**: ARIO rewards help coordinate gateway performance, observation, and network reliability. ## Solana Compatibility As an SPL Token, ARIO works with Solana-compatible wallets and applications. This makes it easier to hold, transfer, swap, and integrate ARIO across the Solana ecosystem while the ar.io protocol continues to serve Arweave's permanent data layer. Common places users interact with ARIO include: - Solana wallets such as Phantom, Solflare, and Backpack - Solana exchanges and liquidity tools such as Jupiter and Raydium - ar.io applications such as the Gateway Portal and ArNS app ## Network Incentives ARIO is non-inflationary. Staking is not a generic yield product; it is a protocol mechanism for aligning gateway operators, delegators, and users around reliable permanent cloud infrastructure. For the deeper mechanics, see [Staking & Delegation](/learn/oip/staking), [Protocol Architecture](/learn/protocol-architecture), and the [Observation and Incentive Protocol](/learn/oip). ## Explore the Token } /> } /> } /> } /> # Wallet Setup (/learn/token/wallets) ## Adding ARIO to Your Wallet ARIO is a standard SPL Token on Solana. Most Solana wallets auto-detect ARIO once you receive tokens, but you can also add it manually using the mint address. ## Supported Wallets ARIO tokens are support by any Solana-compatible wallet - below are some examples: ### Phantom [Phantom](https://phantom.app) is the most popular Solana wallet, available as a browser extension and mobile app. ### Install Phantom Download from [phantom.app](https://phantom.app) for your browser or mobile device. Create a new wallet or import an existing one. ### Receive or Add ARIO ARIO will appear automatically once you receive tokens. To add manually: 1. Open your Phantom wallet 2. Click **"Manage Token List"** or search for tokens 3. Search for **"ARIO"** or paste the mint address: `DcNnMuFxwhgV4WY1HVSaSEgr92bv2b1vUvEKiNxWqHdF` 4. Toggle the token on ### Verify Your Balance Your ARIO balance appears alongside your other Solana tokens. You can send, receive, and swap ARIO directly within Phantom. ### Solflare [Solflare](https://solflare.com) is a full-featured Solana wallet. ### Install Solflare Download from [solflare.com](https://solflare.com) as a browser extension, web app, or mobile app. ### Add ARIO Token 1. Navigate to your token list 2. Search for **"ARIO"** or use the mint address: `DcNnMuFxwhgV4WY1HVSaSEgr92bv2b1vUvEKiNxWqHdF` 3. Add the token to your portfolio ## Next Steps 1. **Stake ARIO**: Delegate your tokens to gateway operators at [gateways.ar.io](https://gateways.ar.io/#/staking) 2. **Buy an ArNS Name**: Register a permanent name at [arns.ar.io](https://arns.ar.io) 3. **Join the Network**: [Run a gateway](/build/run-a-gateway) and earn protocol rewards # Claims, Proofs, and Accountability (/learn/verification/claims-proofs-accountability) Not every verification signal has the same meaning. Ar.io uses claims, signed claims, cryptographic proofs, and economic accountability together because each one catches a different failure mode. ## Quick Reference | Mechanism | Type | Meaning | | --- | --- | --- | | `X-AR-IO-Verified` | Claim | The gateway says its local cache has verified the data. | | RFC 9421 `Signature` | Signed claim | The gateway is accountable for the response headers it signed. | | `Content-Digest` | Verifiable digest | The client can hash the response body and compare it to the header. | | Arweave signature verification | Proof | The client verifies the bytes match the signed commitment for a known ID. | | Observer reports and slashing | Economic accountability | Persistent network-visible failure can affect rewards or stake. | ## Claims A claim is a statement from a gateway. For example, `X-AR-IO-Verified: true` says the gateway believes the returned data was verified in its local system. That is useful operational metadata, but it is not a cryptographic proof. A malicious gateway controls its own software and database. It can set false headers. ## Signed Claims Signed response headers make claims attributable. If a gateway signs `X-AR-IO-Verified`, `X-AR-IO-Stable`, ArNS resolution headers, or a `Content-Digest`, the gateway cannot later deny that it made those claims. Signed claims are evidence. They are not automatically truth. A false signed claim is still possible, but it becomes attributable to a signing key bound to a gateway operator identity. ## Proofs A proof is something the client can verify independently. For known transaction or data item IDs, the strongest path is Arweave signature verification: 1. Fetch the data and the signed metadata needed for verification. 2. Recompute the data root or ANS-104 deep hash. 3. Verify the transaction or data item signature. 4. Confirm that `SHA-256(signature)` matches the requested ID. If this chain verifies, the bytes match the commitment for that ID. The result does not depend on trusting the gateway's local verification header. ## Accountability Network accountability is handled through OIP. Observers sample gateway behavior, upload detailed reports to Arweave, and submit compact results to the protocol. Gateways that perform poorly can miss rewards and, after sustained failure, become eligible for pruning. OIP is intentionally not duplicated here. For mechanics, see: - [Observation & Incentive Protocol](/learn/oip) - [Observer Selection](/learn/oip/observer-selection) - [Reporting](/learn/oip/reporting) - [Performance Evaluation](/learn/oip/performance-evaluation) - [Pruning](/learn/oip/pruning) ## How They Compose These mechanisms are not redundant: - Gateway local verification catches cache corruption and stale imports. - Signed claims create portable evidence about what a gateway said. - Client-side verification detects bad data at request time. - OIP handles long-running public behavior across the network. For high-integrity applications, use client-side verification for request-time correctness and signed claims or observer reports for later accountability. # Client-Side Verification (/learn/verification/client-side-verification) Client-side verification lets applications decide how much trust they place in a gateway response. In ar.io, this is primarily exposed through Wayfinder verification strategies. This page focuses on the verification model. For integration examples and SDK usage, see [Wayfinder](/learn/wayfinder). ## Verification Strategies | Strategy | What it checks | Trust profile | | --- | --- | --- | | Remote verification | Accepts the serving gateway's verification claim. | Trusts the serving gateway. | | Hash verification | Hashes the received data and compares it to hashes from reference gateways. | Trusts the reference set. | | Data root verification | Computes the Merkle data root and compares it to a known or referenced root. | Stronger for L1 transactions when root data is available. | | Signature verification | Reconstructs the signed commitment and verifies the transaction or data item signature. | Strongest path for known IDs. | ## Known IDs For requests by known transaction ID or data item ID, signature verification can establish data integrity without trusting the serving gateway. For ANS-104 data items, the verifier reconstructs the deep hash over the item fields and data, verifies the signature, and confirms that the item ID is derived from that signature. For Arweave L1 transactions, the verifier computes the data root from the served bytes, verifies the transaction signature over the committed fields, and confirms that the transaction ID is derived from the signature. ## ArNS Names Name resolution adds a separate question: did the name resolve to the correct ID? After a name resolves to a transaction ID, the data can be verified against that ID. But content integrity does not by itself prove that the gateway resolved the ArNS name correctly or freshly. Applications that need stronger ArNS guarantees can compare resolution across gateways, check protocol state directly, or require signed ArNS resolution headers from gateways that support response signing. ## Failure Handling Applications need to choose whether verification failures are fail-open or fail-closed: - **Fail-closed**: reject the response and surface an error. - **Fail-open**: return the response but emit a warning or telemetry event. High-integrity workflows should prefer fail-closed behavior. General browsing experiences may choose fail-open behavior for compatibility and performance. Verification is a policy choice. Ar.io provides the mechanisms, but applications decide the verification strategy, reference set, and failure behavior that match their risk model. ## Related Docs - [Wayfinder Protocol](/learn/wayfinder) - [Wayfinder Integration](/learn/wayfinder/integration) - [Gateway Data Verification](/learn/gateways/data-verification) - [Signed Gateway Claims](/learn/verification/signed-gateway-claims) # Verification and Accountability (/learn/verification) Arweave stores data permanently. Ar.io focuses on the delivery layer: retrieving that data, serving it through independent gateways, and giving clients ways to detect or hold gateways accountable when delivery is wrong. The key distinction is that **storage integrity** and **delivery integrity** are different problems. Arweave can prove that data was committed to the storage layer. A client still needs to know whether a gateway served the right bytes, made honest claims about those bytes, and resolved mutable names correctly. ## The Five Layers | Layer | What it does | Where to learn more | | --- | --- | --- | | Arweave base commitments | Bind transaction and data item IDs to signed data commitments. | [What is Arweave?](/learn/what-is-arweave) | | Gateway local verification | Recompute data roots, verify bundles, and re-import corrupted data. | [Gateway data verification](/learn/gateways/data-verification) | | Signed gateway claims | Sign trust-relevant response headers so gateway claims are attributable. | [Signed Gateway Claims](/learn/verification/signed-gateway-claims) | | Client-side verification | Let applications verify responses instead of trusting one gateway. | [Client-Side Verification](/learn/verification/client-side-verification) | | Network observation and incentives | Sample gateway behavior, record reports, and apply economic consequences. | [Observation & Incentive Protocol](/learn/oip) | ## The Core Property When a client knows the transaction ID or data item ID it wants, the strongest verification path is cryptographic: verify the signature chain from the served bytes back to the requested identifier. For Arweave L1 transactions, the identifier is derived from the transaction signature, and the signature commits to the transaction data root. For ANS-104 data items, the item ID is derived from the data item signature, and the signature commits to the item fields and data. This means a gateway cannot fabricate different valid bytes for a known ID without the original signer's private key. Other parts of the ar.io architecture make that property easier to use in production. ## What This Does Not Prove Verification is powerful, but the limits matter: - A signed gateway response proves that the gateway made a claim, not that the claim was true. - There is no universal proof that a specific client received a complete response body. - Client-side verification is opt-in and can be configured fail-open or fail-closed. - Observer-based accountability is sampling-based and is designed for persistent public misbehavior, not every targeted attack. - ArNS name correctness is separate from content integrity. A client can verify bytes for a transaction ID while still needing to verify that a name resolved to the right ID. ## Explore } /> } /> } /> } /> } /> # Signed Gateway Claims (/learn/verification/signed-gateway-claims) Gateway verification headers are useful, but unsigned headers can be modified, stripped, or fabricated. Ar.io gateways can opt in to signing trust-relevant response headers with [RFC 9421 HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html). ## What Gets Signed A gateway signs the trust-relevant headers that are present on a response. These can include: | Category | Examples | | --- | --- | | Data identity and status | `x-ar-io-data-id`, `x-ar-io-verified`, `x-ar-io-stable`, `x-ar-io-trusted` | | Body integrity | `content-type`, `content-digest` | | ArNS resolution | `x-arns-name`, `x-arns-resolved-id`, `x-arns-ttl-seconds`, `x-arns-process-id` | | Bundled data | `x-ar-io-root-transaction-id` | When gateways chain requests through one another, each gateway signs its own assessment. Upstream signature headers are stripped before the serving gateway signs its response. ## Signing Keys and Operator Identity The response-signing key is a hot Ed25519 key used for per-response signatures. It is separate from the operator wallet. The operator wallet signs an attestation document that binds the Ed25519 response-signing key to the operator's gateway identity. This creates a chain: 1. The gateway signs response headers with Ed25519. 2. The Ed25519 public key is attested by the operator wallet. 3. The operator identity is connected to network state through the Gateway Address Registry. That chain makes signed claims attributable to a gateway operator for the time period in which the attestation and registry state are valid. ## Body Integrity HTTP signatures sign headers, not necessarily the full response body. Body integrity depends on whether a signed `Content-Digest` is present. The architecture is designed around a performance tradeoff: - Cached responses, `HEAD` requests, and buffered responses under the gateway's body-digest threshold can include `Content-Digest`. - Large streamed responses may omit body binding to preserve streaming performance. - When no body digest is signed, clients that need integrity should use client-side Arweave verification. Signed headers provide non-repudiation for gateway claims. They do not by themselves prove that the response body was correct. ## Limitations - A malicious gateway can still sign a false claim. - Missing signatures should be treated as unsigned mode, not as proof of failure. - If a response-signing key is compromised, historical signatures from that key remain cryptographically valid until clients apply rotation or revocation policy. - Partial content responses need special care because signing the data ID does not necessarily bind a specific byte range. ## Related Docs - [Claims, Proofs, and Accountability](/learn/verification/claims-proofs-accountability) - [Gateway Registry](/learn/gateways/gateway-registry) - [Client-Side Verification](/learn/verification/client-side-verification) # Wayfinder Protocol (/learn/wayfinder) ## The Problem: Centralized Gateway Reliance Today, most Arweave content is accessed through a single gateway: `arweave.net`. This creates a critical centralization risk: - **Single point of failure** - If arweave.net goes down, content becomes inaccessible - **Censorship vulnerability** - A single gateway can block or filter content - **Performance bottlenecks** - All traffic flows through one gateway - **No content verification** - Users must trust the gateway to serve authentic content ## What is Wayfinder? The Wayfinder protocol solves these problems by enabling **decentralized access** to Arweave content through any gateway in the ar.io network. It's a [URI scheme](https://wikipedia.org/wiki/Uniform_Resource_Identifier) that transforms centralized URLs like `https://arweave.net/txid` into decentralized `ar://` URLs that can be resolved by any participating gateway. Key capabilities: - **Multi-gateway routing** - Access content through any ar.io gateway - **Built-in verification** - Verify content authenticity regardless of which gateway serves it - **Automatic failover** - If one gateway is down, requests route to another - **User control** - Choose routing strategies based on speed, trust, or randomization ## How Wayfinder Works The Wayfinder protocol consists of three core components that work together to resolve and serve Arweave content: ```mermaid graph TB subgraph "Centralized Access" U1[Users] --> AW[arweave.net] AW --> AR1[Arweave] style AW fill:#ffcccc,stroke:#ff0000,stroke-width:3px,color:#333 style U1 fill:#fff,stroke:#333,stroke-width:2px,color:#333 style AR1 fill:#fff,stroke:#333,stroke-width:2px,color:#333 end ``` vs. ```mermaid graph TB subgraph "Decentralized Access" U2[Users] --> WF[ar:// Protocol] WF --> G1[Gateway 1] WF --> G2[Gateway 2] WF --> G3[Gateway 3] WF --> GN[Gateway N] G1 --> AR2[Arweave] G2 --> AR2 G3 --> AR2 GN --> AR2 end style U2 fill:#fff,stroke:#333,stroke-width:2px,color:#333 style AR2 fill:#fff,stroke:#333,stroke-width:2px,color:#333 style WF fill:#b3d9ff,stroke:#333,stroke-width:3px,color:#333 style G1 fill:#b3d9ff,stroke:#333,stroke-width:2px,color:#333 style G2 fill:#b3d9ff,stroke:#333,stroke-width:2px,color:#333 style G3 fill:#b3d9ff,stroke:#333,stroke-width:2px,color:#333 style GN fill:#b3d9ff,stroke:#333,stroke-width:2px,color:#333 ``` Wayfinder enables: 1. **Decentralized Routing**: Select from multiple gateways instead of relying on arweave.net 2. **Redundant Retrieval**: If one gateway fails, automatically failover to another 3. **Trust-minimized Verification**: Verify content authenticity regardless of which gateway serves it ### Transaction ID Resolution To access content tied to an Arweave Transaction ID (TxId), simply append the TxId to `ar://`: ``` ar://qI19W6spw-kzOGl4qUMNp2gwFH2EBfDXOFsjkcNyK9A ``` Inputting this into a WayFinder-equipped browser will route your request through the right ar.io Gateway, translating it as per your `Routing Method` settings. ### ArNS Name Resolution Fetching content via an Ar.io Name System (ArNS) name is straightforward. Attach the ArNS name to `ar://`: ``` ar://good-morning ``` The Wayfinder protocol, along with the Wayfinder app, distinguishes between TxIDs and ArNS names. Once the suitable `https://` request is formulated, the chosen gateway resolves the ArNS name through the current ar.io network state. ## Detailed Flow ```mermaid sequenceDiagram participant User participant Wayfinder participant Gateway as ar.io Gateway participant Arweave User->>Wayfinder: ar://ardrive or ar://txid Wayfinder->>Wayfinder: Select a gateway alt ArNS Name (ar://ardrive) Wayfinder->>Gateway: Request content via ArNS name Gateway->>Gateway: Resolve ArNS to TxID Gateway->>Gateway: Check cache for content alt Content not cached Gateway->>Arweave: Fetch from network Arweave-->>Gateway: Return content end Gateway-->>Wayfinder: Return content else Transaction ID (ar://txid) Wayfinder->>Gateway: Request content via TxID Gateway->>Gateway: Check cache for content alt Content not cached Gateway->>Arweave: Fetch from network Arweave-->>Gateway: Return content end Gateway-->>Wayfinder: Return content end Wayfinder->>Wayfinder: Verify content integrity Wayfinder-->>User: Deliver verified content ``` ## Why Decentralized Access Matters ### Resilience Against Censorship With centralized gateways like arweave.net, content can be blocked or filtered at a single point. Wayfinder distributes access across multiple independent gateways, making censorship significantly more difficult. ### Always-Available Content When arweave.net experiences downtime or congestion, all content becomes inaccessible. Wayfinder automatically routes around failed gateways, ensuring your content remains available. ### Trust Through Verification Centralized gateways require blind trust - you can't verify if the content served matches what's stored on Arweave. Wayfinder includes built-in verification capabilities, allowing clients to cryptographically verify content authenticity from any gateway. ### Performance Through Competition Multiple gateways create a competitive ecosystem where gateways optimize for speed and reliability. Users benefit from automatic routing to the fastest available gateway. ## Verification: Trust but Verify Wayfinder supports content verification at multiple levels: 1. **Transaction verification** - Verify that content matches the requested transaction ID 2. **Data integrity checks** - Ensure content hasn't been tampered with during transmission 3. **Manifest validation** - For bundled content, verify all components are authentic 4. **ArNS resolution verification** - Confirm ArNS names resolve to the correct transaction IDs This verification happens transparently, giving users confidence that they're receiving authentic Arweave content regardless of which gateway serves it. For the trust model behind these strategies, including the difference between gateway claims, signed claims, and cryptographic proofs, see [Verification and Accountability](/learn/verification). ## Explore Wayfinder } /> } /> } /> } /> # Integration (/learn/wayfinder/integration) ## Getting Started **Get the Extension** The easiest way to use Wayfinder is the [Wayfinder Extension](https://chromewebstore.google.com/detail/ario-wayfinder/hnhmeknhajanolcoihhkkaaimapnmgil?hl=en-US), available in the Chrome Web Store. ### Wayfinder Extension The wayfinder-extension is a simple Chrome extension that supports the ar:// routing protocol and allows you to: - **Navigate ar:// URLs directly** in your browser - **Configure routing strategies** - Choose how requests are routed to gateways - **Set verification preferences** - Control content verification levels - **Monitor gateway performance** - See which gateways are serving your requests No coding required - just install the extension and start browsing ar:// URLs! ## Developer Integration Options For developers who want to integrate Wayfinder into their applications: ### Wayfinder Core The [wayfinder-core](/sdks/wayfinder/wayfinder-core) library is the core protocol implementation that accepts various configuration options for setting up Wayfinder. It provides: - **Gateway selection strategies** - Choose how to route requests - **Content verification** - Optionally verify content authenticity - **Telemetry collection** - Understand gateway performance across routing strategies - **Custom configurations** - Fine-tune behavior for your use case ### Wayfinder React Web developers will likely be interested in [wayfinder-react](/sdks/wayfinder/wayfinder-react), which provides: - **React Context Provider** - Easy integration with React apps - **Custom hooks** - Simplified data fetching and state management - **Component library** - Pre-built UI components for common patterns - **TypeScript support** - Full type safety out of the box ### Wayfinder Router For teams who want a **ready-to-deploy proxy service**, [Wayfinder Router](/build/run-wayfinder-router) provides: - **Single trusted endpoint** - Give your users one reliable URL - **Server-side verification** - All content verified before serving - **Built-in caching** - LRU cache with optional disk persistence - **Admin dashboard** - Web UI for monitoring and moderation - **Standalone binary** - No runtime dependencies, runs anywhere This is ideal when you want infrastructure-level Wayfinder without embedding the SDK in your application. ## Common Integration Pattern: Preferred with Fallback For most builds, teams use the "preferred with fallback" pattern. This routing strategy prioritizes your preferred gateway but automatically falls back to other gateways in the network if needed: ```typescript const wayfinder = new createWayfinderClient({ ario: ARIO.mainnet(), verification: 'hash', routing: 'preferred', preferredGateway: 'ar-io.net', }); ``` This pattern ensures: 1. **Primary traffic** goes to your preferred gateway (for example, one you run or trust) 2. **Automatic failover** if your gateway can't serve the data 3. **Network resilience** by finding another gateway that can serve the content For detailed routing strategy options, see the [routing strategies documentation](/sdks/wayfinder/wayfinder-core/routing-strategies). ## Verification: Optional but Encouraged While verification is optional, it's strongly encouraged when fetching from gateways: ```typescript const wayfinder = createWayfinderClient({ //...other settings, verification: 'hash', // hash based verification }); ``` Verification ensures you're receiving authentic content regardless of which gateway serves it. ## Telemetry for Performance Insights Enable telemetry to understand how Wayfinder performs across gateways: ```typescript const wayfinder = createWayfinderClient({ //...other settings, telemetry: { enabled: true, sampleRate: 0.1, // sample 10% of requests apiKey: 'your-api-key', // optional clientName: 'my-app', clientVersion: '1.0.0' } }); ``` This helps teams make data-driven decisions about gateway selection and optimization. ## React Integration Example Here's a complete example using wayfinder-react: ```tsx // Configure Wayfinder const wayfinderConfig = { gatewaysProvider: new NetworkGatewaysProvider({ ario: ARIO.mainnet(), }), routingSettings: { // use the fastest pinging strategy to select the fastest gateway for requests strategy: new FastestPingRoutingStrategy({ timeoutMs: 1000, }), } verificationSettings: { enabled: false } }; // Wrap your app function App() { return ( ); } // Use in components function WayfinderImage({ txId }: { txId: string }) { const { resolvedUrl, isLoading, error } = useWayfinderUrl({ txId }); if (error) { return Error resolving URL: {error.message}; } if (isLoading) { return Resolving URL...; } return ( ); } ``` ## Next Steps } /> } /> } /> } /> # Use Cases (/learn/wayfinder/use-cases) ## Decentralized Web Hosting with Flexible Access With Wayfinder, not only can websites be hosted on the Arweave network, but their accessibility is also enhanced. By using the Wayfinder Protocol, web developers can ensure that if a specific ar.io Gateway is down, the content can still be accessed through another gateway, offering a more reliable and resilient user experience. This is valuable for: - **Personal websites** that need to remain accessible - **Documentation sites** that must be always available - **Portfolio sites** for professionals and creators ## Digital Archives and Preservation Digitally archiving public domain works becomes easier to share with Wayfinder. Historical institutions, researchers, and enthusiasts can link directly to permanent documents or media without depending on a single gateway URL. Unlike hardcoded links which might break if a specific gateway goes offline, Wayfinder ensures that the content remains consistently accessible. This is ideal for: - **Historical documents** and public domain works - **Academic research** and scholarly articles - **Cultural preservation** projects - **Legal documents** that need permanent access ## Media Sharing Platforms with Consistent Content Delivery For platforms hosting user-generated content, the Wayfinder Protocol provides not just decentralized hosting but also a guarantee of content delivery. Even if a content piece becomes viral and one gateway gets congested, Wayfinder ensures that users can still access the content through another gateway, providing a seamless experience. Perfect for: - **Social media platforms** with user-generated content - **Video sharing sites** with viral content - **Image galleries** and art platforms - **Podcast hosting** and audio content ## Decentralized Applications (DApps) with Reliable Front-End Accessibility DApps, while benefiting from Arweave's permanent hosting, can further ensure their front-end remains consistently accessible to users by using Wayfinder. If a DApp's front-end is accessed frequently, causing strain on one gateway, Wayfinder can help ensure the load is distributed, and the DApp remains online and functional. This is essential for: - **DeFi applications** that need high availability - **NFT marketplaces** with high traffic - **Gaming platforms** with real-time requirements - **Collaborative tools** and productivity apps ## Branded Content Access Companies and individuals can brand their permaweb content, making it accessible through their domain, enhancing brand visibility and user trust. This is achieved through DNS TXT records that link domain names to Arweave content. ## Dynamic Content Updates Domain owners can easily update what Permaweb content their `ar://` URL resolves to, which is ideal for frequently updated resources like documents, blogs, and application interfaces. ## Educational and Informational Resources Educational institutions and information providers can make their resources permanently available on the permaweb, accessible through simple, memorable URLs. ## Next Steps Ready to get started with Wayfinder? Explore [Integration Methods](/learn/wayfinder/integration) to see how to implement Wayfinder in your applications, or go back to the [Overview](/learn/wayfinder) to review the basics. # Husky (Developers Only) (/sdks/(clis)/ardrive-cli/(build-and-run-from-source)/husky-developers-only) We use husky 6.x to manage the git commit hooks that help to improve the quality of our commits. Please run: ```shell yarn husky install ``` to enable git hooks for your local checkout. Without doing so, you risk committing non-compliant code to the repository. # Install Yarn 3 (/sdks/(clis)/ardrive-cli/(build-and-run-from-source)/install-yarn-3) Both the ArDrive CLI and ArDrive Core JS use Yarn 3 to manage dependencies and initiate workflows, so follow the [yarn installation instructions][yarn-install] in order to get the latest version. In most cases: ```shell brew install yarn npm install -g yarn ``` # Installing and Starting the CLI From Source (/sdks/(clis)/ardrive-cli/(build-and-run-from-source)/installing-and-starting-the-cli-from-source) Now that your runtime and/or development environment is set up, to install the package simply run: ```shell yarn && yarn build ``` And then start the CLI (always from the root of this repository): ```shell yarn ardrive ``` For convenience in the **non-developer case**, you can install the CLI globally on your system by performing the following step: ```shell yarn pack npm install i -g /path/to/package.tgz ardrive ``` # Recommended Visual Studio Code extensions (Developers Only) (/sdks/(clis)/ardrive-cli/(build-and-run-from-source)/recommended-visual-studio-code-extensions-developers-only) To ensure your environment is compatible, we also recommend the following VSCode extensions: - [ES-Lint][eslint-vscode] - [Editor-Config][editor-config-vscode] - [Prettier][prettier-vscode] - [ZipFS][zipfs-vscode] # Using a custom ArDrive-Core-JS (Optional) (/sdks/(clis)/ardrive-cli/(build-and-run-from-source)/using-a-custom-ardrive-core-js-optional) To test a with a custom version of the `ardrive-core-js` library on your local system, change the `"ardrive-core-js"` line in `package.json` to the root of your local `ardrive-core-js` repo: ```diff - "ardrive-core-js": "1.0.0" + "ardrive-core-js": "../ardrive-core-js/" ``` # Dealing With Network Congestion (/sdks/(clis)/ardrive-cli/(other-utility-operations)/dealing-with-network-congestion) Currently, Arweave blocks hold up to 1000 transactions per block. The "mempool", where pending transactions reside until they've been included into a block, will only hold a transaction for 50 blocks (~100-150 minutes) before it's discarded by the network resulting in no fees or data being transacted. During periods of network congestion (i.e. those where the mempool contains 1000 or more pending transactions), it may make sense to either: a) wait for congestion to dissipate before attempting your transactions. b) apply the fee boost multiplier to your transactions rewards with the --boost parameter during write operations in order to front-run some of the congestion. #### Check for network congestion before uploading ```shell ardrive get-mempool ardrive get-mempool | jq 'length' ``` #### Front-run Congestion By Boosting Miner Rewards ```shell ardrive upload-file --wallet-file /path/to/my/wallet.json --parent-folder-id "f0c58c11-430c-4383-8e54-4d864cc7e927" --local-path ./helloworld.txt --boost 1.5 ``` #### Send AR Transactions From a Cold Wallet The best cold wallet storage never exposes your seed phrase and/or private keys to the Internet or a compromised system interface. You can use the ArDrive CLI to facilitate cold storage and transfer of AR. If you need a new cold AR wallet, generate one from an air-gapped machine capable of running the ArDrive CLI by following the instructions in the [Wallet Operations](#wallet-operations) section. Fund your cold wallet from whatever external sources you'd like. NOTE: Your cold wallet won't appear on chain until it has received AR. The workflow to send the AR out from your cold wallet requires you to generate a signed transaction with your cold wallet on your air-gapped machine via the ArDrive CLI, and then to transfer the signed transaction (e.g. by a file on a clean thumb drive) to an Internet-connected machine and send the transaction to the network via the ArDrive CLI. You'll need two inputs from the Internet-connected machine: - the last transaction sent OUT from the cold wallet (or an empty string if none has ever been sent out) - the base fee for an Arweave transaction (i.e. a zero bye transaction). Note that this value could change if a sufficient amount of time passes between the time you fetch this value, create the transaction, and send the transaction. To get the last transaction sent from your cold wallet, use the `last-tx` command and specify your wallet address e.g.: ``` ardrive last-tx -a \ ``` To get the base transaction reward required for an AR transaction, use the `base-reward` function, optionally applying a reward boost multiple if you're looking to front-run network congestion: ``` ardrive base-reward --boost 1.5 ``` Write down or securely copy the values you derived from the Internet-connected machine and run the following commands on the airgapped machine, piping the outputted signed transaction data to a file in the process, e.g. `sendme.json` (if that's your signed transaction transfer medium preference): ``` ardrive create-tx -w /path/to/wallet/file.json -d \ -a \ --last-tx \ --reward "\" > sendme.json ``` Transport your signed transaction to the Internet-connected machine and run the following command to send your transaction to the Arweave network: ``` ardrive send-tx -x /path/to/sendme.json ``` # Monitoring Transactions (/sdks/(clis)/ardrive-cli/(other-utility-operations)/monitoring-transactions) Block time on Arweave is typically between 2-3 minutes in duration, so transactions can be mined within that time frame when [network congestion](#dealing-with-network-congestion) is low. Transactions, in the general case, proceed through the following set of states: - Pending: the transaction is waiting the "mempool" to be mined - Confirming: the transaction was mined on an Arweave Node, but has not yet been confirmed by at least 15 total nodes on the network - Confirmed: the transaction was mined on an Arweave Node and confirmed by at least 15 total nodes on the network - Not Found: the transaction is not available for any of the following reasons: - Insufficient reward to join the mempool - Insufficient reward to be mined within 50 blocks during a period of network congestion - Transaction is transitioning between states - Transaction ID is invalid Monitor any Arweave transaction's status via its transaction ID by performing: ```shell ardrive tx-status -t "ekSMckikdRJ8RGIkFa-X3xq3427tvM7J9adv8HP3Bzs" ``` Example output: ```shell ekSMckikdRJ8RGIkFa-X3xq3427tvM7J9adv8HP3Bzs: Mined at block height 775810 with 22439 confirmations ``` ```shell watch -n 10 ardrive tx-status -t "ekSMckikdRJ8RGIkFa-X3xq3427tvM7J9adv8HP3Bzs" ``` # Persistent Caching of ArFS Entity Metadata (/sdks/(clis)/ardrive-cli/(other-utility-operations)/persistent-caching-of-arfs-entity-metadata) To avoid redundant requests to the Arweave network for immutable ArFS entity metadata, a persistent file cache is created and maintained at: ``` Windows: /ardrive-caches/metadata Non-Windows: /.ardrive/caches/metadata ``` The `XDG_CACHE_HOME` environment variable is honored, where applicable, and will be used in place of `os.homedir()` in the scenarios described above. Metadata cache logging to stderr can be enabled by setting the `ARDRIVE_CACHE_LOG` environment variable to `1`. Cache performance is UNDEFINED for multi-process scenarios, but is presumed to be generally usable. The cache can be manually cleared safely at any time that any integrating app is not in operation. ```shell █████╗ ██████╗ ██████╗ ██████╗ ██╗██╗ ██╗███████╗ ██╔══██╗██╔══██╗██╔══██╗██╔══██╗██║██║ ██║██╔════╝ ███████║██████╔╝██║ ██║██████╔╝██║██║ ██║█████╗ ██╔══██║██╔══██╗██║ ██║██╔══██╗██║╚██╗ ██╔╝██╔══╝ ██║ ██║██║ ██║██████╔╝██║ ██║██║ ╚████╔╝ ███████╗ ╚═╝ ╚═╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═══╝ ╚══════╝ ██████╗██╗ ██╗ ██╔════╝██║ ██║ ██║ ██║ ██║ ██║ ██║ ██║ ╚██████╗███████╗██║ ╚═════╝╚══════╝╚═╝ Write ArFS =========== create-drive create-folder upload-file create-manifest pin-file create-snapshot hide-file unhide-file hide-folder unhide-folder move-file move-folder retry-tx Read ArFS =========== file-info folder-info drive-info list-folder list-drive list-all-drives download-file download-folder download-drive Wallet Ops =========== generate-seedphrase generate-wallet get-address get-balance send-ar get-drive-key get-file-key last-tx Arweave Ops =========== base-reward get-mempool create-tx send-tx tx-status ardrive \ --help ``` [ArDrive Community Discord][ardrive-discord] [ardrive]: https://ardrive.io [arweave]: https://ardrive.io/what-is-arweave/ [ardrive-github]: https://github.com/ardriveapp/ [arfs]: https://ardrive.atlassian.net/l/c/m6P1vJDo [ardrive-web-app]: https://app.ardrive.io [ardrive-core]: https://github.com/ardriveapp/ardrive-core-js [yarn-install]: https://yarnpkg.com/getting-started/install [nvm-install]: https://github.com/nvm-sh/nvm#installing-and-updating [wsl-install]: https://code.visualstudio.com/docs/remote/wsl [editor-config-vscode]: https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig [prettier-vscode]: https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode [zipfs-vscode]: https://marketplace.visualstudio.com/items?itemName=arcanis.vscode-zipfs [eslint-vscode]: https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint [viewblock blockchain explorer]: https://viewblock.io/arweave/ [ardrive-discord]: https://discord.com/invite/ya4hf2H [arconnect]: https://arconnect.io/ [kb-wallets]: https://ardrive.atlassian.net/l/c/FpK8FuoQ [arweave-manifests]: https://github.com/ArweaveTeam/arweave/wiki/Path-Manifests [example-manifest-webpage]: https://arweave.net/qozq9YIUPEHfZhoTp9DkBpJuA_KNULBnfLiMroj5pZI [arlocal]: https://github.com/textury/arlocal [mozilla-mime-types]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types [viewblock]: https://viewblock.io/arweave/ [tx_anchors]: https://docs.arweave.org/developers/server/http-api#field-definitions [gql-guide]: https://gql-guide.vercel.app/#owners [ardrive-turbo]: https://ardrive.io/turbo/ # Using a Custom Arweave Gateway (/sdks/(clis)/ardrive-cli/(other-utility-operations)/using-a-custom-arweave-gateway) On each command that uses a gateway, it is possible to supply your own custom Arweave gateway using the flag `--gateway` or by setting an environment variable named `ARWEAVE_GATEWAY`. For example, you could test out that your ArFS transactions are working as expected on a local test network such as [ArLocal] with this flow: ```shell npx arlocal curl http://localhost:1984/mint/{ your public wallet address }/99999999999999 ardrive create-drive --gateway http://127.0.0.1:1984 -w /path/to/wallet -n 'my-test-drive' curl "$ARWEAVE_GATEWAY/mine" ardrive upload-file -F { root folder id from create drive } -l /path/to/file -w /path/to/wallet curl "$ARWEAVE_GATEWAY/mine" ardrive list-drive -d { drive id from create drive } ardrive download-file -f { file id from upload file } ``` # Git (/sdks/(clis)/ardrive-cli/(prerequisites)/git) Some of ArDrive's dependencies are transitively installed via Git. Install it, if necessary, and ensure that it's available within your terminal environment: [Download Git](https://git-scm.com/downloads) # NVM (Optional - Recommended) (/sdks/(clis)/ardrive-cli/(prerequisites)/nvm-optional-recommended) This project uses the Node Version Manager (NVM) and an `.nvmrc` file to lock the recommended Node version used by the latest version of `ardrive-core-js`. **Note for Windows: We recommend using WSL for setting up NVM on Windows using the [instructions described here][wsl-install]** Follow these steps to get NVM up and running on your system: 1. Install NVM using [these installation instructions][nvm-install]. 2. Navigate to this project's root directory 3. Ensure that the correct version of Node is installed by performing: `nvm install` 4. Use the correct version of Node, by performing: `nvm use` **IT IS STRONGLY RECOMMENDED THAT YOU AVOID GENERATING WALLETS VIA SEED PHRASE WITH THE CLI USING ANY NODE VERSION OTHER THAN THE ONE SPECIFIED IN `.nvmrc`.** # Creating Drives (/sdks/(clis)/ardrive-cli/(working-with-drives)/creating-drives) ```shell ardrive create-drive --wallet-file /path/to/my/wallet.json --drive-name "My Public Archive" ardrive create-drive --wallet-file /path/to/my/wallet.json --drive-name "Teenage Love Poetry" -P ``` # List Drive Pipeline Examples (/sdks/(clis)/ardrive-cli/(working-with-drives)/list-drive-pipeline-examples) You can utilize `jq` and the list commands to reshape the commands' output data into useful forms and stats for many use cases. Here are a few examples: ```shell ardrive list-drive -d a44482fd-592e-45fa-a08a-e526c31b87f1 | jq '.[] | select(.entityType == "file") | "https://app.ardrive.io/#/file/" + .entityId + "/view"' ``` Example output: ```shell "https://app.ardrive.io/#/file/1337babe-f000-dead-beef-ffffffffffff/view" "https://app.ardrive.io/#/file/cdbc9ddd-1cab-41d9-acbd-fd4328929de3/view" "https://app.ardrive.io/#/file/f19bc712-b57a-4e0d-8e5c-b7f1786b34a1/view" "https://app.ardrive.io/#/file/4f8e081b-42f2-442d-be41-57f6f906e1c8/view" "https://app.ardrive.io/#/file/0e02d254-c853-4ff0-9b6e-c4d23d2a95f5/view" "https://app.ardrive.io/#/file/c098b869-29d1-4a86-960f-a9e10433f0b0/view" "https://app.ardrive.io/#/file/4afc8cdf-4d27-408a-bfb9-0a2ec21eebf8/view" "https://app.ardrive.io/#/file/85fe488d-fcf7-48ca-9df8-2b39958bbf15/view" ... ``` ```shell ardrive list-drive -d 13c3c232-6687-4d11-8ac1-35284102c7db | jq ' map(select(.entityType == "file") | .size) | add' ``` ```shell ardrive list-drive -d 01ea6ba3-9e58-42e7-899d-622fd110211c | jq '[ .[] | select(.entityType == "file") ] | length' ``` # Listing Drives for an Address (/sdks/(clis)/ardrive-cli/(working-with-drives)/listing-drives-for-an-address) You can list all the drives associated with any Arweave wallet address, though the details of private drives will be obfuscated from you unless you provide the necessary decryption data. ```shell ardrive list-all-drives -w /path/to/my/wallet.json -P ardrive list-all-drives --address "HTTn8F92tR32N8wuo-NIDkjmqPknrbl10JWo5MZ9x2k" ``` # Listing Every Entity in a Drive (/sdks/(clis)/ardrive-cli/(working-with-drives)/listing-every-entity-in-a-drive) Useful notes on listing the contents of drives: - Listing a drive is effectively the same as listing its root folder. - You can control the tree depth of the data returned. - path, txPath, and entityIdPath properties on entities can provide useful handholds for other forms of data navigation ```shell ardrive list-drive -d "c7f87712-b54e-4491-bc96-1c5fa7b1da50" -w /path/to/my/wallet.json -P ardrive list-drive -d "c7f87712-b54e-4491-bc96-1c5fa7b1da50" -w /path/to/my/wallet.json -P --with-keys ardrive list-drive -d "c7f87712-b54e-4491-bc96-1c5fa7b1da50" --max-depth 2 ``` # Managing Drive Passwords (/sdks/(clis)/ardrive-cli/(working-with-drives)/managing-drive-passwords) The ArDrive CLI's private drive and folder functions all require either a drive password OR a drive key. Private file functions require either the drive password or the file key. **Keys and passwords are sensitive data, so manage the entry, display, storage, and transmission of them very carefully.** Drive passwords are the most portable, and fundamental, encryption facet, so a few options are available during private drive operations for supplying them: - Environment Variable - STDIN - Secure Prompt #### Supplying Your Password: Environment Variable ```shell read -rs TMP_ARDRIVE_PW ardrive \ -w /path/to/wallet.json -P ``` #### Supplying Your Password: STDIN ```shell cat /path/to/my/drive/password.txt | ardrive \ -w /path/to/wallet.json -P ardrive \ -w /path/to/wallet.json -P -w /path/to/wallet.json -P ? Enter drive password: › ******** ``` # Understanding Drive and File Keys (/sdks/(clis)/ardrive-cli/(working-with-drives)/understanding-drive-and-file-keys) Private Drives achieve privacy via end-to-end encryption facilitated by hash-derived "Keys". Drive Keys encrypt/decrypt Drive and Folder data, and File Keys encrypt/decrypt File Data. The relationships among your data and their keys is as follows: - Drive Key = functionOf(Wallet Signature, Randomly Generated Drive ID, User-specified Drive Password) - File Key = functionOf(Randomly Generated File ID, Drive Key) When you create private entities, the returned JSON data from the ArDrive CLI will contain the keys needed to decrypt the encrypted representation of your entity that is now securely and permanently stored on the blockweave. To derive the drive key again for a drive, perform the following: ```shell ardrive get-drive-key -w /path/to/my/wallet.json -d "6939b9e0-cc98-42cb-bae0-5888eca78885" -P ``` To derive the file key again for a file, perform the following: ```shell ardrive get-file-key --file-id "bd2ce978-6ede-4b0d-8f79-2d7bc235a0e0" --drive-id "6939b9e0-cc98-42cb-bae0-5888eca78885" --drive-key "yHdCjpCK3EcuhQcKNx2d/NN5ReEjoKfZVqKunlCnPEo" ``` # Understanding Drive Hierarchies (/sdks/(clis)/ardrive-cli/(working-with-drives)/understanding-drive-hierarchies) At the root of every data tree is a "Drive" entity. When a drive is created, a Root Folder is also created for it. The entity IDs for both are generated and returned when you create a new drive: ```shell ardrive create-drive --wallet-file /path/to/my/wallet.json --drive-name "Teenage Love Poetry" | tee created_drive.json | jq '[.created[] | del(.metadataTxId, .entityName, .bundledIn)]' [ { "type": "drive", "entityId": "6939b9e0-cc98-42cb-bae0-5888eca78885" } { "type": "folder", "entityId": "d1535126-fded-4990-809f-83a06f2a1118" } ] ``` The relationship between the drive and its root folder is clearly visible when retrieving the drive's info: ```shell ardrive drive-info -d "6939b9e0-cc98-42cb-bae0-5888eca78885" | jq '{driveId, rootFolderId}' { "driveId": "6939b9e0-cc98-42cb-bae0-5888eca78885", "rootFolderId": "d1535126-fded-4990-809f-83a06f2a1118" } ``` All file and folder entities in the drive will be anchored to it by a "Drive-ID" GQL Tag. And they'll each be anchored to a parent folder ID, tracked via the "Parent-Folder-ID" GQL tag, forming a tree structure whose base terminates at the Root Folder. # Dry Run (/sdks/(clis)/ardrive-cli/(working-with-entities)/dry-run) An important feature of the ArDrive CLI is the `--dry-run` flag. On each command that would write an ArFS entity, there is the option to run it as a "dry run". This will run all of the steps and print the outputs of a regular ArFS write, but will skip sending the actual transaction: ```shell ardrive \ \ --dry-run ``` This can be very useful for gathering price estimations or to confirm that you've copy-pasted your entity IDs correctly before committing to an upload. # Uploading to Turbo (BETA) (/sdks/(clis)/ardrive-cli/(working-with-entities)/uploading-to-turbo-beta) Users can optionally choose to send each ArFS entities created to [ArDrive Turbo][ardrive-turbo] using the `--turbo` flag. Instead of using AR from an Arweave wallet, you can use Turbo Credits or take advantage of free/discounted upload promotions. ```shell ardrive \ \ --turbo ``` This flag will skip any balance check on the CLI side. Turbo will check a user's balance and accept/reject a data item at the time of upload. The `--turbo` flag by default will send your files to `upload.ardrive.io` to be bundled. To change the Turbo destination, users can use the `--turbo-url` flag. # Creating a Snapshot (/sdks/(clis)/ardrive-cli/(working-with-files)/creating-a-snapshot) A **snapshot** is a single Arweave transaction, tagged `Entity-Type: snapshot`, `Drive-Id`, `Block-Start`, and `Block-End`, whose body is a JSON index of every ArFS entity metadata transaction (drive, folder, and file revisions) mined for that drive across the block range it covers. It exists purely as a read-path optimization: a client that wants to list a drive's full entity history can read the snapshot's JSON body directly instead of paginating through and re-fetching every individual metadata transaction the drive has ever produced. `create-snapshot` builds this snapshot for you and posts it to Arweave. Some important things to know: - **Costs to post, like any other data transaction.** For a drive with a long entity history the snapshot body can be large, so posting it is not free -- `create-snapshot` estimates the cost up front, asserts your wallet can cover it, and prints the cost before sending. - **When to use it.** Snapshotting is most useful for drives with a large number of files/folders/revisions, where clients that support snapshot-accelerated listing would otherwise have to replay a long transaction history on every listing. It's a maintenance operation you run occasionally (e.g. periodically, or before publishing a drive expected to see heavy read traffic) -- not something every drive needs. - **Public drives only (for now).** Private drive snapshots are not yet supported. - **Idempotent-ish, not automatic.** Each run creates a NEW snapshot transaction covering the drive's entity history at that point in time; it does not update or replace a previous snapshot. ```shell ardrive create-snapshot --drive-id "bc9af866-6421-40f1-ac89-202bddb5c487" -w "/path/to/wallet" ``` Use `--dry-run` to see the block range, entity count, byte size, and estimated cost without posting anything: ```shell ardrive create-snapshot --drive-id "bc9af866-6421-40f1-ac89-202bddb5c487" -w "/path/to/wallet" --dry-run ``` Like other write commands, `create-snapshot` supports `--boost`, `--turbo`/`--turbo-url`, and `--gateway`. See `ardrive create-snapshot --help` for the full flag list. # Download a Single file (BETA) (/sdks/(clis)/ardrive-cli/(working-with-files)/download-a-single-file-beta) By using the `download-file` command you can download a file on chain to a folder in your local storage specified by --local-path (or to your current working directory if not specified): ```shell ardrive download-file -w /path/to/wallet.json --file-id "ff450770-a9cb-46a5-9234-89cbd9796610" --local-path /my_ardrive_downloads/ ``` Specify a filename in the --local-path if you'd like to use a different name than the one that's used in your drive: ```shell ardrive download-file -w /path/to/wallet.json --file-id "ff450770-a9cb-46a5-9234-89cbd9796610" --local-path /my_ardrive_downloads/my_pic.png ``` # Downloading a Drive (/sdks/(clis)/ardrive-cli/(working-with-files)/downloading-a-drive) To download the whole drive you can use the `download-drive` command. ```shell ardrive download-drive -d "c0c8ba1c-efc5-420d-a07c-a755dc67f6b2" ``` This is equivalent to running the `download-folder` command against the root folder of the drive. # Downloading a Folder with Files (/sdks/(clis)/ardrive-cli/(working-with-files)/downloading-a-folder-with-files) You can download a folder from ArDrive to your local machine with the `download-folder` command. In the following examples, assume that a folder with ID "47f5bde9-61ba-49c7-b409-1aa4a9e250f6" exists in your drive and is named "MyArDriveFolder". ```shell ardrive download-folder -f "47f5bde9-61ba-49c7-b409-1aa4a9e250f6" ``` By specifying the `--local-path` option, you can choose the local parent folder into which the on-chain folder will be downloaded. When the parameter is omitted, its value defaults to the current working directory (i.e. `./`). ```shell ardrive download-folder -f "47f5bde9-61ba-49c7-b409-1aa4a9e250f6" --local-path /my_ardrive_downloads/ ``` The `--max-depth` parameter lets you to choose a custom folder depth to download. When omitted, the entire subtree of the folder will be downloaded. In the following example, only the immediate children of the folder will be downloaded: ```shell ardrive download-folder -f "47f5bde9-61ba-49c7-b409-1aa4a9e250f6" --max-depth 0 ``` The behaviors of `--local-path` are similar to those of `cp` and `mv` in Unix systems, e.g.: ```shell ardrive download-folder -f "47f5bde9-61ba-49c7-b409-1aa4a9e250f6" --local-path "/existing_folder" ardrive download-folder -f "47f5bde9-61ba-49c7-b409-1aa4a9e250f6" --local-path "/existing_folder/MyArDriveFolder" ardrive download-folder -f "47f5bde9-61ba-49c7-b409-1aa4a9e250f6" --local-path "/existing_folder/non_existent_folder" ardrive download-folder -f "47f5bde9-61ba-49c7-b409-1aa4a9e250f6" --local-path "/non_existent_folder_1/non_existent_folder_2" ``` # Fetching the Metadata of a File Entity (/sdks/(clis)/ardrive-cli/(working-with-files)/fetching-the-metadata-of-a-file-entity) Simply perform the file-info command to retrieve the metadata of a file: ```shell ardrive file-info --file-id "e5ebc14c-5b2d-4462-8f59-7f4a62e7770f" ``` Example output: ```shell { "appName": "ArDrive-Web", "appVersion": "0.1.0", "arFS": "0.11", "contentType": "application/json", "driveId": "51062487-2e8b-4af7-bd81-4345dc28ea5d", "entityType": "file", "name": "2_depth.png", "txId": "CZKdjqwnmxbWchGA1hjSO5ZH--4OYodIGWzI-FmX28U", "unixTime": 1633625081, "size": 41946, "lastModifiedDate": 1605157729000, "parentFolderId": "a2c8a0cb-0ca7-4dbb-8bf8-93f75f308e63", "entityId": "e5ebc14c-5b2d-4462-8f59-7f4a62e7770f", "fileId": "e5ebc14c-5b2d-4462-8f59-7f4a62e7770f", "dataTxId": "Jz0WsWyAGVc0aE3UzACo-YJqG8OPrN3UucmDdt8Fbjc", "dataContentType": "image/png" } ``` # Hiding and Unhiding a File or Folder (/sdks/(clis)/ardrive-cli/(working-with-files)/hiding-and-unhiding-a-file-or-folder) The `hide-file`, `unhide-file`, `hide-folder`, and `unhide-folder` commands let you toggle whether a file or folder entity is flagged as hidden, without touching its data or metadata otherwise. Hiding writes a new metadata revision with an `isHidden` flag set to `true`; clients that respect this flag (e.g. the ArDrive web/desktop apps) omit the entity from their normal drive listings, while it remains fully present on-chain. Unhiding writes another revision flipping the flag back to `false`. Some important things to know: - **Reversible.** Hiding never deletes or re-uploads data -- it's a metadata-only toggle, and `unhide-file`/`unhide-folder` fully restores visibility at any time. - **Works on both public and private entities.** Pass `--drive-key` or (`--wallet-file`/`--seed-phrase` plus `--unsafe-drive-password`) to target a private file/folder; omit them to target a public one, exactly like `rename-file`/`rename-folder`. - **Costs a small metadata fee.** Like a rename, hiding/unhiding writes a new metadata revision to Arweave, so it isn't free, but it's the same tiny metadata-only cost as any other rename/move operation -- no file data is re-uploaded. - **Not recursive.** Hiding a folder flags only that folder's own metadata; it does not walk its contents and hide child files/folders individually. ```shell ardrive hide-file --file-id "290a3f9a-37b2-4f0f-a899-6fac983833b3" -w "/path/to/wallet.json" ardrive unhide-file --file-id "290a3f9a-37b2-4f0f-a899-6fac983833b3" -w "/path/to/wallet.json" ardrive hide-file --file-id "290a3f9a-37b2-4f0f-a899-6fac983833b3" -w "/path/to/wallet.json" --unsafe-drive-password "p4ssw0rd" ardrive hide-folder --folder-id "568d5eba-dbf3-4a49-8129-1c58f7fd35bc" -w "/path/to/wallet.json" ardrive unhide-folder --folder-id "568d5eba-dbf3-4a49-8129-1c58f7fd35bc" -w "/path/to/wallet.json" --drive-key "base64EncodedDriveKey" ``` Like other write commands, `hide-file`/`unhide-file`/`hide-folder`/`unhide-folder` support `--dry-run`, `--boost`, `--turbo`/`--turbo-url`, and `--gateway`. See `ardrive hide-file --help` (and `unhide-file`/`hide-folder`/`unhide-folder --help`) for the full flag list. # IPFS CID Tagging (/sdks/(clis)/ardrive-cli/(working-with-files)/ipfs-cid-tagging) Certain nodes on the Arweave network may be running the [IPFS+Arweave bridge](https://arweave.medium.com/arweave-ipfs-persistence-for-the-interplanetary-file-system-9f12981c36c3). Tagging your file upload transaction with its IPFS v1 CID value in the 'IPFS-Add' tag may allow you to take advantage of this system. It can also be helpful for finding data on Arweave via GQL based on its CID. To include the CID tag on your **PUBLIC** file uploads, you may use the '--add-ipfs-tag' flag: ```shell ardrive upload-file --add-ipfs-tag --local-path /path/to/file.txt --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json ``` # Moving Files (/sdks/(clis)/ardrive-cli/(working-with-files)/moving-files) Files can be moved from one folder to another within the same drive. Moving a file is simply the process of uploading a new file metadata revision with an updated File ID Parent Folder ID relationship. The following command will move a file from its current location in a public drive to a new parent folder in that drive: ```shell ardrive move-file --file-id "e5ebc14c-5b2d-4462-8f59-7f4a62e7770f" --parent-folder-id "a2c8a0cb-0ca7-4dbb-8bf8-93f75f308e63" ``` # Name Conflict Resolution on Upload (/sdks/(clis)/ardrive-cli/(working-with-files)/name-conflict-resolution-on-upload) By default, the `upload-file` command will use the upsert behavior if existing entities are encountered in the destination folder tree that would cause naming conflicts. Expect the behaviors from the following table for each of these resolution settings: | Source Type | Conflict at Dest | `skip` | `replace` | `upsert` (default) | | ----------- | ---------------- | ------ | --------- | ------------------ | | File | None | Insert | Insert | Insert | | File | Matching File | Skip | Update | Skip | | File | Different File | Skip | Update | Update | | File | Folder | Skip | Fail | Fail | | Folder | None | Insert | Insert | Insert | | Folder | File | Skip | Fail | Fail | | Folder | Folder | Re-use | Re-use | Re-use | The default upsert behavior will check the destination folder for a file with a conflicting name. If no conflicts are found, it will insert (upload) the file. In the case that there is a FILE to FILE name conflict found, it will only update it if necessary. To determine if an update is necessary, upsert will compare the last modified dates of conflicting file and the file being uploaded. When they are matching, the upload will be skipped. Otherwise the file will be updated as a new revision. To override the upsert behavior, use the `--replace` option to always make new revisions of a file or the `--skip` option to always skip the upload on name conflicts: ```shell ardrive upload-file --replace --local-path /path/to/file.txt --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json ``` ```shell ardrive upload-file --skip --local-path /path/to/file.txt --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json ``` Alternatively, the upload-file commands now also supports the `--ask` conflict resolution option. This setting will always provide an interactive prompt on name conflicts that allows users to decide how to resolve each conflict found: ```shell ardrive upload-file --ask --local-file-path /path/to/file.txt --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json Destination folder has a file to file name conflict! File name: 2.png File ID: efbc0370-b69f-44d9-812c-0d272b019027 This file has a DIFFERENT last modified date Please select how to proceed: › - Use arrow-keys. Return to submit. ❯ Replace as new file revision Upload with a different file name Skip this file upload ``` # Pinning a File (/sdks/(clis)/ardrive-cli/(working-with-files)/pinning-a-file) Pinning lets you reference an **existing** Arweave data transaction as a new file entity in one of your PUBLIC drives, without re-uploading any data. This is useful for adopting data that already lives permanently on Arweave (e.g. a transaction uploaded outside of ArDrive, or one belonging to someone else) into your drive's folder structure, so it shows up alongside your other files with its own name, metadata, and location. Because a pinned file's metadata transaction only references the existing `--tx-id` (it doesn't touch the underlying data bytes), pinning a small file costs the same tiny metadata-only fee as any other file operation -- there is no data-upload cost, regardless of the size of the original file. Some important constraints: - **Public drives only.** Pinning writes a plaintext ArFS metadata transaction that points at the referenced data. Private drives are not supported -- targeting a private `--parent-folder-id` fails with a clear error. - **The referenced transaction is never re-uploaded or modified.** Only a new file metadata entity is created; `--tx-id` is reused as-is as the new file's data transaction. - **Name conflicts throw by default.** If `--dest-file-name` already exists in the destination folder, the command fails unless `--skip` is provided, in which case the command exits successfully having made no changes. ```shell ardrive pin-file --parent-folder-id "a2c8a0cb-0ca7-4dbb-8bf8-93f75f308e63" --tx-id "Y7GFF8r9y0MEU_oi1aZeD87vrmai97JdRQ2L0cbGJ68" --dest-file-name "hello_world.txt" -w "/path/to/wallet" ``` `--drive-id` is optional -- the destination drive is normally resolved automatically from `--parent-folder-id`. Supply it only if you want the command to assert that the folder belongs to the drive you expect (the command fails if it doesn't): ```shell ardrive pin-file --parent-folder-id "a2c8a0cb-0ca7-4dbb-8bf8-93f75f308e63" --drive-id "bc9af866-6421-40f1-ac89-202bddb5c487" --tx-id "Y7GFF8r9y0MEU_oi1aZeD87vrmai97JdRQ2L0cbGJ68" --dest-file-name "hello_world.txt" -w "/path/to/wallet" ``` Like other write commands, `pin-file` supports `--dry-run`, `--boost`, `--turbo`/`--turbo-url`, and `--gateway`. See `ardrive pin-file --help` for the full flag list. # Progress Logging of Transaction Uploads (/sdks/(clis)/ardrive-cli/(working-with-files)/progress-logging-of-transaction-uploads) Progress logging of transaction uploads to stderr can be enabled by setting the `ARDRIVE_PROGRESS_LOG` environment variable to `1`: ```shell Uploading file transaction 1 of total 2 transactions... Transaction _GKQasQX194a364Hph8Oe-oku1AdfHwxWOw9_JC1yjc Upload Progress: 0% Transaction _GKQasQX194a364Hph8Oe-oku1AdfHwxWOw9_JC1yjc Upload Progress: 35% Transaction _GKQasQX194a364Hph8Oe-oku1AdfHwxWOw9_JC1yjc Upload Progress: 66% Transaction _GKQasQX194a364Hph8Oe-oku1AdfHwxWOw9_JC1yjc Upload Progress: 100% Uploading file transaction 2 of total 2 transactions... Transaction nA1stCdTkuf290k0qsqvmJ78isEC0bwgrAi3D8Cl1LU Upload Progress: 0% Transaction nA1stCdTkuf290k0qsqvmJ78isEC0bwgrAi3D8Cl1LU Upload Progress: 13% Transaction nA1stCdTkuf290k0qsqvmJ78isEC0bwgrAi3D8Cl1LU Upload Progress: 28% Transaction nA1stCdTkuf290k0qsqvmJ78isEC0bwgrAi3D8Cl1LU Upload Progress: 42% Transaction nA1stCdTkuf290k0qsqvmJ78isEC0bwgrAi3D8Cl1LU Upload Progress: 60% Transaction nA1stCdTkuf290k0qsqvmJ78isEC0bwgrAi3D8Cl1LU Upload Progress: 76% Transaction nA1stCdTkuf290k0qsqvmJ78isEC0bwgrAi3D8Cl1LU Upload Progress: 91% Transaction nA1stCdTkuf290k0qsqvmJ78isEC0bwgrAi3D8Cl1LU Upload Progress: 100% ``` # Rename a Single File (/sdks/(clis)/ardrive-cli/(working-with-files)/rename-a-single-file) To rename an on-chain file you can make use of the `rename-file` command. The required parameters are the file ID and the new name, as well as the owner wallet or seed phrase. ```shell ardrive rename-file --file-id "290a3f9a-37b2-4f0f-a899-6fac983833b3" --file-name "My custom file name.txt" --wallet-file "wallet.json" ``` # Retrying a Failed File Data Transaction (Public Unbundled Files Only) (/sdks/(clis)/ardrive-cli/(working-with-files)/retrying-a-failed-file-data-transaction-public-unbundled-files-only) Arweave data upload transactions are split into two phases: transaction posting and chunks uploading. Once the transaction post phase has been completed, you've effectively "paid" the network for storage of the data chunks that you'll send in the next stage. If your system encounters an error while posting the transaction, you can retry posting the transaction for as long as your tx_anchor is valid ([learn more about tx_anchors here][tx_anchors]). You may retry and/or resume posting chunks at any time after your transaction has posted. The ArDrive CLI allows you to take advantage of this Arweave protocol capability. Using the CLI, when the transaction post has succeeded but the chunk upload step fails, the data transaction's ID could be lost. There are a few options to recover this ID. If the failed transaction is the most recent one sent from a wallet, the transaction ID can be recovered with the `ardrive last-tx -w /path/to/wallet` command AFTER the transaction's headers have been mined (It can take 5-10 minutes for the tx-id to become available with the last-tx approach). Other options for finding the partially uploaded transaction's ID include: - Using an Arweave gateway GQL http endpoint to search for transactions that belong to the wallet. See this [Arweave GQL Guide][gql-guide] for more info. - Browse the recent transactions associated with the wallet via a block explorer tool like [ViewBlock][viewblock]. In order to re-seed the chunks for an unbundled ArFS data transaction, a user must have the data transaction ID, the original file data, and either a destination folder ID or a valid file ID for the file. Supply that information to the `retry-tx` command like so: ```shell ardrive retry-tx --tx-id { Data Transaction ID } --parent-folder-id { Destination Folder ID } --local-path /path/to/file --wallet-file /path/to/wallet ``` **Note: Retry feature is currently only available for PUBLIC unbundled file transactions. It is also perfectly safe to mistakenly re-seed the chunks of a healthy transaction, the transaction will remain stable and the wallet balance will not be affected.** # Understanding Bundled Transactions (/sdks/(clis)/ardrive-cli/(working-with-files)/understanding-bundled-transactions) The ArDrive CLI currently uses two different methods for uploading transactions to the Arweave network: standard transactions and Direct to Network (D2N) bundled transactions. By default, the CLI will send a D2N bundled transaction for any action that would result in multiple transactions. This bundling functionality is currently used on the `upload-file` and `create-drive` commands. D2N bundled transactions come with several benefits and implications: - Bundling saves AR and enhances ArFS reliability by sending associated ArFS transactions up as one atomic bundle. - Bundled transactions are treated as a single data transaction by the Arweave network, but can be presented as separate transactions by the Arweave Gateway once they have been "unbundled". - Un-bundling can take anywhere from a few minutes up to an hour. During that time, the files in the bundle will neither appear in list- commands nor be downloadable. Similarly, they will not appear in the web app after syncs until un-bundling is complete. **This can negatively affect the accuracy of upsert operations**, so it's best to wait before retrying bulk uploads. - Bundling reliability on the gateway side degrades once bundles reach either 500 data items (or ~250 files) or 500 MiB, so the CLI will create and upload multiple bundles as necessary, or will send files that are simply too large for reliable bundling as unbundled txs. # Uploading a Custom Manifest (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-a-custom-manifest) Using the custom content type feature, it is possible for users to upload their own custom manifests. The Arweave gateways use this special content type in order to identify an uploaded file as a manifest: ```shell application/x.arweave-manifest+json ``` In addition to this content type, the manifest must also adhere to the [correct JSON structure](#manifest-json) of an Arweave manifest. A user can create their own manifest from scratch, or start by piping a generated manifest to a JSON file and editing it to their specifications: ```shell ardrive create-manifest -w /path/to/wallet -f "6c312b3e-4778-4a18-8243-f2b346f5e7cb" --dry-run | jq '{manifest}.manifest' > my-custom-manifest.json ``` After editing the generated manifest, simply perform an `upload-file` command with the custom Arweave manifest content type to any PUBLIC folder: ```shell ardrive upload-file --content-type "application/x.arweave-manifest+json" --local-path my-custom-manifest.json --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json ``` The returned `dataTxId` field on the created `file` entity will be the endpoint that the manifest can be found on Arweave, just as explained in the [manifest sections](#uploading-manifests) above: ```shell https://arweave.net/{dataTxId} https://arweave.net/{dataTxId}/custom-file-1 https://arweave.net/{dataTxId}/custom-file-2 ``` # Uploading a Folder with Files (Bulk Upload) (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-a-folder-with-files-bulk-upload) Users can perform a bulk upload by using the upload-file command on a target folder. The command will reconstruct the folder hierarchy on local disk as ArFS folders on the permaweb and upload each file into their corresponding folders: ```shell ardrive upload-file --local-path /path/to/folder --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json ``` # Uploading a Non-Bundled Transaction (NOT RECOMMENDED) (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-a-non-bundled-transaction-not-recommended) While not recommended, the CLI does provide the option to forcibly send all transactions as standard transactions rather than attempting to bundle them together. To do this, simply add the `--no-bundle` flag to the `upload-file` or `create-drive` command: ```shell ardrive upload-file --no-bundle --local-path /path/to/file --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json ``` # Uploading a Single File (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-a-single-file) To upload a file, you'll need a parent folder id, the file to upload's file path, and the path to your wallet: ```shell ardrive upload-file --local-path /path/to/file.txt --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json ``` Example output: ```shell { "created": [ { "type": "file", "entityName": "file.txt" "entityId": "6613395a-cf19-4420-846a-f88b7b765c05" "dataTxId": "l4iNWyBapfAIj7OU-nB8z9XrBhawyqzs5O9qhk-3EnI", "metadataTxId": "YfdDXUyerPCpBbGTm_gv_x5hR3tu5fnz8bM-jPL__JE", "bundledIn": "1zwdfZAIV8E26YjBs2ZQ4xjjP_1ewalvRgD_GyYw7f8", "sourceUri": "file:///path/to/file.txt" }, { "type": "bundle", "bundleTxId": "1zwdfZAIV8E26YjBs2ZQ4xjjP_1ewalvRgD_GyYw7f8" } ], "tips": [ { "txId": "1zwdfZAIV8E26YjBs2ZQ4xjjP_1ewalvRgD_GyYw7f8", "recipient": { "address": "3mxGJ4xLcQQNv6_TiKx0F0d5XVE0mNvONQI5GZXJXkt" }, "winston": "10000000" } ], "fees": { "1zwdfZAIV8E26YjBs2ZQ4xjjP_1ewalvRgD_GyYw7f8": 42819829 } } ``` NOTE: To upload to the root of a drive, specify its root folder ID as the parent folder ID for the upload destination. You can retrieve it like so: ```shell ardrive drive-info -d "c7f87712-b54e-4491-bc96-1c5fa7b1da50" | jq -r '.rootFolderId' ``` # Uploading Files with Custom MetaData (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-files-with-custom-metadata) ArDrive CLI has the capability of attaching custom metadata to ArFS File and Folder MetaData Transactions during the `upload-file` command. This metadata can be applied to either the GQL tags on the MetaData Transaction and/or into the MetaData Transaction's Data JSON. All custom metadata applied must ultimately adhere to the following JSON shapes: ```ts // GQL Tags type CustomMetaDataGqlTags = Record; // Data JSON Fields type CustomMetaDataJsonFields = Record; | string | number | boolean | null | { [member: string]: JsonSerializable } | JsonSerializable[]; ``` e.g: ```shell { IPFS-Add: 'MY_HASH' } { 'Custom Name': ['Val 1', 'Val 2'] } ``` When the custom metadata is attached to the MetaData Transaction's GQL tags, they will become visible on any Arweave GQL gateway and also third party tools that read GQL data. When these tags are added to the MetaData Transaction's Data JSON they can be read by downloading the JSON data directly from `https://arweave.net/METADATA_TX_ID`. To add this custom metadata to your file metadata transactions, CLI users can pass custom metadata these parameters: - `--metadata-file path/to/json/schema` - `--metadata-json '{"key": "val", "key-2": true, "key-3": 420, "key-4": ["more", 1337]}'` - `--metadata-gql-tags "Tag-Name" "Tag Val"` The `--metadata-file` will accept a file path to JSON file containing custom metadata: ```shell ardrive upload-file --metadata-file path/to/metadata/json # ... ``` This JSON schema object must contain instructions on where to put this metadata with the `metaDataJson` and `metaDataGqlTags` keys. e.g: ```json { "metaDataJson": { "Tag-Name": ["Value-1", "Value-2"] }, "metaDataGqlTags": { "GQL Tag Name": "Tag Value" } } ``` The `--metadata-gql-tags` parameter accepts an array of string values to be applied to the MetaData Tx GQL Tags. This method of CLI input does not support multiple tag values for a given tag name and the input must be an EVEN number of string values. (Known bug: String values starting with the `"-"` character are currently not supported. Use --metadata-file parameter instead.) e.g: ```shell upload-file --metadata-gql-tags "Custom Tag Name" "Custom Value" # ... ``` And the `--metadata-json` parameter will accept a stringified JSON input. It will apply all declared JSON fields directly to the MetaData Tx's Data JSON. e.g: ```shell upload-file --metadata-json ' { "json field": "value", "another fields": false } ' # ... ``` Custom metadata applied to files and/or folders during the `upload-file` command will be read back through all existing read commands. e.g: ```shell ardrive file-info -f 067c4008-9cbe-422e-b697-05442f73da2b { "appName": "ArDrive-CLI", "appVersion": "1.17.0", "arFS": "0.11", "contentType": "application/json", "driveId": "967215ca-a489-494b-97ec-0dd428d7be34", "entityType": "file", "name": "unique-name-9718", "txId": "sxg8bNu6_bbaHkJTxAINVVoz_F-LiFe6s7OnxzoJJk4", "unixTime": 1657655070, "size": 262148, "lastModifiedDate": 1655409872705, "dataTxId": "ublZcIff77ejl3m0uEA8lXEfnTWmSBOFoz-HibqKeyk", "dataContentType": "text/plain", "parentFolderId": "97bc4fb5-aca4-4ffe-938f-1285153d98ca", "entityId": "067c4008-9cbe-422e-b697-05442f73da2b", "fileId": "067c4008-9cbe-422e-b697-05442f73da2b", "IPFS-Add": "MY_HASH", "Tag-1": "Val", "Tag-2": "Val", "Tag-3": "Val", "Boost": "1.05" } ``` #### Applying Unique Custom MetaData During Bulk Workflows With some custom scripting and the `--metadata-file` parameter, the ArDrive CLI can be used to apply custom metadata to each file individually in a bulk workflow. For example, if you choose a numbered file naming pattern you can make use of a `for` loop: ```shell for i in {1..5} do ardrive upload-file -F f0c58c11-430c-4383-8e54-4d864cc7e927 --local-path "../uploads/test-file-$i.txt" -w "/path/to/wallet.json" --metadata-file "../custom/metadata-$i.json" --dry-run > "file-result-$i.json" done ``` # Uploading From a Remote URL (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-from-a-remote-url) You can upload a file from an existing url using the `--remote-path` flag. This must be used in conjunction with `--dest-file-name`. You can use a custom content type using the `--content-type` flag, but if this isn't used the app will use the content type from the response header of the request for the remote data. ```shell ardrive upload-file --remote-path "https://url/to/file" --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -d "example.jpg" -w /path/to/wallet.json ``` # Uploading Manifests (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-manifests) [Arweave Path Manifests][arweave-manifests] are are special `.json` files that instruct Arweave Gateways to map file data associated with specific, unique transaction IDs to customized, hosted paths relative to that of the manifest file itself. So if, for example, your manifest file had an arweave.net URL like: ```shell https://arweave.net/{manifest tx id} ``` Then, all the mapped transactions and paths in the manifest file would be addressable at URLs like: ```shell https://arweave.net/{manifest tx id}/foo.txt https://arweave.net/{manifest tx id}/bar/baz.png ``` ArDrive supports the creation of these Arweave manifests using any of your PUBLIC folders. The generated manifest paths will be links to each of the file entities within the specified folder. The manifest file entity will be created at the root of the folder. To create a manifest of an entire public drive, specify the root folder of that drive: ```shell ardrive create-manifest -f "bc9af866-6421-40f1-ac89-202bddb5c487" -w "/path/to/wallet" ``` You can also create a manifest of a folder's file entities at a custom depth by using the `--max-depth` option: ```shell ardrive create-manifest --max-depth 0 -f "867228d8-4413-4c0e-a499-e1decbf2ea38" -w "/path/to/wallet" ``` Creating a `.json` file of your manifest links output can be accomplished here with some `jq` parsing and piping to a file: ```shell ardrive create-manifest -w /path/to/wallet -f "6c312b3e-4778-4a18-8243-f2b346f5e7cb" | jq '{links}' > links.json ``` If you'd like to preview the contents of your manifest before uploading, you can perform a dry run and do some lightweight post processing to isolate the data: ```shell ardrive create-manifest -w /path/to/wallet -f "6c312b3e-4778-4a18-8243-f2b346f5e7cb" --dry-run | jq '{manifest}.manifest' ``` ```json { "manifest": "arweave/paths", "version": "0.1.0", "index": { "path": "index.html" }, "paths": { "hello_world.txt": { "id": "Y7GFF8r9y0MEU_oi1aZeD87vrmai97JdRQ2L0cbGJ68" }, "index.html": { "id": "pELonjVebHyBsdxVymvxbGTmHD96v9PuuUXj8GUHGoY" } } } ``` The manifest data transaction is tagged with a unique content-type, `application/x.arweave-manifest+json`, which tells the gateway to treat this file as a manifest. The manifest file itself is a `.json` file that holds the paths (the data transaction ids) to each file within the specified folder. When your folder is later changed by adding files or updating them with new revisions, the original manifest will NOT be updated on its own. A manifest is a permanent record of your files in their current state. However, creating a subsequent manifest with the same manifest name will create a new revision of that manifest in its new current state. Manifests follow the same name conflict resolution as outlined for files above (upsert by default). #### Hosting a Webpage with Manifest When creating a manifest, it is possible to host a webpage or web app. You can do this by creating a manifest on a folder that has an `index.html` file in its root. Using generated build folders from popular frameworks works as well. One requirement here to note is that the `href=` paths from your generated `index.html` file must not have leading a `/`. This means that the manifest will not resolve a path of `/dist/index.js` but it will resolve `dist/index.js` or `./dist/index.js`. As an example, here is a flow of creating a React app and hosting it with an ArDrive Manifest. First, generate a React app: ```shell yarn create react-app my-app ``` Next, add this field to the generated `package.json` so that the paths will resolve correctly: ```json "homepage": ".", ``` Then, create an optimized production build from within the app's directory: ```shell yarn build ``` Now, we can create and upload that produced build folder on ArDrive to any of your existing ArFS folder entities: ```shell ardrive upload-file -l "/build" -w "/path/to/wallet" --parent-folder-id "bc9af866-6421-40f1-ac89-202bddb5c487" ``` And finally, create the manifest using the generated Folder ID from the build folder creation: ```shell ardrive create-manifest -f "41759f05-614d-45ad-846b-63f3767504a4" -w "/path/to/wallet" ``` In the return output, the top link will be a link to the deployed web app: ```shell "links": [ "https://arweave.net/0MK68J8TqGhaaOpPe713Zn0jdpczMt2NGS2CtRYiuAg", "https://arweave.net/0MK68J8TqGhaaOpPe713Zn0jdpczMt2NGS2CtRYiuAg/asset-manifest.json", "https://arweave.net/0MK68J8TqGhaaOpPe713Zn0jdpczMt2NGS2CtRYiuAg/favicon.ico", "https://arweave.net/0MK68J8TqGhaaOpPe713Zn0jdpczMt2NGS2CtRYiuAg/index.html", # ... ``` This is effectively hosting a web app with ArDrive. Check out the ArDrive Price Calculator React App hosted as an [ArDrive Manifest][example-manifest-webpage]. # Uploading Multiple Files (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-multiple-files) To upload an arbitrary number of files or folders, pass a space-separated list of paths to `--local-paths`: ```shell ardrive upload-file -w wallet.json -F "6939b9e0-cc98-42cb-bae0-5888eca78885" --local-paths ./image.png ~/backups/ ../another_file.txt ardrive upload-file -w wallet.json -F "6939b9e0-cc98-42cb-bae0-5888eca78885" --local-paths ./*.json ``` # Uploading With a Custom Content Type (/sdks/(clis)/ardrive-cli/(working-with-files)/uploading-with-a-custom-content-type) Each file uploaded to the Arweave network receives a `"Content-Type"` GraphQL tag that contains the MIME type for the file. The gateway will use this content type to determine how to serve that file's data transaction at the `arweave.net/{data tx id}` endpoint. By default, the CLI will attempt to derive this content type from the file extension of the provided file. In most cases, the content type that is derived will be correct and the gateway will properly serve the file. The CLI also provides the option for users to upload files with a custom content type using the `--content-type` flag: ```shell ardrive upload-file --content-type "application/json" --local-path /path/to/file --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" -w /path/to/wallet.json ``` It is currently possible to set this value to any given string, but the gateway will still only serve valid content types. Check out this list of commonly used MIME types to ensure you're providing a valid content type: [Common MIME types][mozilla-mime-types]. Note: In the case of multi-file uploads or recursive folder uploads, setting this `--content-type` flag will set the provided custom content type on EVERY file entity within a given upload. # Creating Folders (/sdks/(clis)/ardrive-cli/(working-with-folders)/creating-folders) Creating folders manually is straightforward: ```shell ardrive create-folder --parent-folder-id "63153bb3-2ca9-4d42-9106-0ce82e793321" --folder-name "My Awesome Folder" -w /path/to/wallet.json ``` Example output: ```shell { "created": [ { "type": "folder", "metadataTxId": "AYFMBVmwqhbg9y5Fbj3Iasy5oxUqhauOW7PcS1sl4Dk", "entityId": "d1b7c514-fb12-4603-aad8-002cf63015d3", "key": "yHdCjpCKD2cuhQcKNx2d/XF5ReEjoKfZVqKunlCnPEk", "entityName": "My Awesome Folder" } ], "tips": [], "fees": { "AYFMBVmwqhbg9y5Fbj3Iasy5oxUqhauOW7PcS1sl4Dk": 1378052 } } ``` Note: Folders can also be created by supplying a folder as the --local-path of an upload-file command. In this case, the folder hierarchy on the local disk will be reconstructed on chain during the course of the recursive bulk upload. # Listing Contents of a Folder (/sdks/(clis)/ardrive-cli/(working-with-folders)/listing-contents-of-a-folder) Similar to drives, the `list-folder` command can be used to fetch the metadata of each entity within a folder. But by default, the command will fetch only the immediate children of that folder (`--max-depth 0`): ```shell ardrive list-folder --parent-folder-id "29850ab7-56d4-4e1f-a5be-cb86d5513940" ``` Example output: ```shell [ { "appName": "ArDrive-CLI", "appVersion": "2.0", "arFS": "0.11", "contentType": "application/json", "driveId": "01ea6ba3-9e58-42e7-899d-622fd110211a", "entityType": "folder", "name": "mytestfolder", "txId": "HYiKyfLwY7PT9NleTQoTiM_-qPVUwf4ClDhx1sjUAEU", "unixTime": 1635102772, "parentFolderId": "29850ab7-56d4-4e1f-a5be-cb86d5513940", "entityId": "03df2929-1440-4ab4-bbf0-9dc776e1ed96", "path": "/My Public Folder/mytestfolder", "txIdPath": "/09_x0X2eZ3flXXLS72WdTDq6uaa5g2LjsT-QH1m0zhU/HYiKyfLwY7PT9NleTQoTiM_-qPVUwf4ClDhx1sjUAEU", "entityIdPath": "/29850ab7-56d4-4e1f-a5be-cb86d5513940/03df2929-1440-4ab4-bbf0-9dc776e1ed96" }, { "appName": "ArDrive-CLI", "appVersion": "2.0", "arFS": "0.11", "contentType": "application/json", "driveId": "01ea6ba3-9e58-42e7-899d-622fd110211a", "entityType": "folder", "name": "Super sonic public folder", "txId": "VUk1B_vo1va2-EHLtqjsotzy0Rdn6lU4hQo3RD2xoTI", "unixTime": 1631283259, "parentFolderId": "29850ab7-56d4-4e1f-a5be-cb86d5513940", "entityId": "452c6aec-43dc-4015-9abd-20083068d432", "path": "/My Public Folder/Super sonic sub folder", "txIdPath": "/09_x0X2eZ3flXXLS72WdTDq6uaa5g2LjsT-QH1m0zhU/VUk1B_vo1va2-EHLtqjsotzy0Rdn6lU4hQo3RD2xoTI", "entityIdPath": "/29850ab7-56d4-4e1f-a5be-cb86d5513940/452c6aec-43dc-4015-9abd-20083068d432" }, { "appName": "ArDrive-CLI", "appVersion": "2.0", "arFS": "0.11", "contentType": "application/json", "driveId": "01ea6ba3-9e58-42e7-899d-622fd110211a", "entityType": "file", "name": "test-number-twelve.txt", "txId": "429zBqnd7ZBNzgukaix26RYz3g5SeXCCo_oIY6CPZLg", "unixTime": 1631722234, "size": 47, "lastModifiedDate": 1631722217028, "dataTxId": "vA-BxAS7I6n90cH4Fzsk4cWS3EOPb1KOhj8yeI88dj0", "dataContentType": "text/plain", "parentFolderId": "29850ab7-56d4-4e1f-a5be-cb86d5513940", "entityId": "e5948327-d6de-4acf-a6fe-e091ecf78d71", "path": "/My Public Folder/test-number-twelve.txt", "txIdPath": "/09_x0X2eZ3flXXLS72WdTDq6uaa5g2LjsT-QH1m0zhU/429zBqnd7ZBNzgukaix26RYz3g5SeXCCo_oIY6CPZLg", "entityIdPath": "/29850ab7-56d4-4e1f-a5be-cb86d5513940/e5948327-d6de-4acf-a6fe-e091ecf78d71" }, { "appName": "ArDrive-CLI", "appVersion": "2.0", "arFS": "0.11", "contentType": "application/json", "driveId": "01ea6ba3-9e58-42e7-899d-622fd110211a", "entityType": "file", "name": "wonderful-test-file.txt", "txId": "6CokwlzB81Fx7dq-lB654VM0XQykdU6eYohDmEJ2gk4", "unixTime": 1631671275, "size": 23, "lastModifiedDate": 1631283389232, "dataTxId": "UP8THwA_1gvyRqNRqYmTpWvU4-UzNWBN7SiX_AIihg4", "dataContentType": "text/plain", "parentFolderId": "29850ab7-56d4-4e1f-a5be-cb86d5513940", "entityId": "3274dae9-3487-41eb-94d5-8d5d3d8bc343", "path": "/My Public Folder/wonderful-test-file.txt", "txIdPath": "/09_x0X2eZ3flXXLS72WdTDq6uaa5g2LjsT-QH1m0zhU/6CokwlzB81Fx7dq-lB654VM0XQykdU6eYohDmEJ2gk4", "entityIdPath": "/29850ab7-56d4-4e1f-a5be-cb86d5513940/3274dae9-3487-41eb-94d5-8d5d3d8bc343" } ] ``` To list further than the immediate children, you can make use of the flags: `--all` and `--max-depth`. ```shell ardrive list-folder --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" --all ardrive list-folder --parent-folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" --max-depth 2 ``` In the case of private entitites, the `--with-keys` flag will make the command to include the keys in the output. ```shell ardrive list-folder --parent-folder-id "1b027047-4cfc-4eee-88a8-9af694f660c0" -w /my/wallet.json --with-keys ``` # Moving Folders (/sdks/(clis)/ardrive-cli/(working-with-folders)/moving-folders) Moving a folder is as simple as supplying a new parent folder ID. Note that naming collisions among entities within a folder are not allowed. ```shell ardrive move-folder --folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" --parent-folder-id "29850ab7-56d4-4e1f-a5be-cb86d5513921" -w /path/to/wallet.json ``` # Renaming Folders (/sdks/(clis)/ardrive-cli/(working-with-folders)/renaming-folders) In order to rename a folder you must provide a name different from its current one, and it must not create naming conflicts with its sibling entities. ```shell ardrive rename-folder --folder-id "568d5eba-dbf3-4a49-8129-1c58f7fd35bc" --folder-name "Folder with cool stuff" -w "./wallet.json" ``` # Viewing Folder Metadata (/sdks/(clis)/ardrive-cli/(working-with-folders)/viewing-folder-metadata) To view the metadata of a folder, users can use the `folder-info` command: ```shell ardrive folder-info --folder-id "9af694f6-4cfc-4eee-88a8-1b02704760c0" ``` # ArFS (/sdks/(clis)/ardrive-cli/arfs) [ArFS] is a data modeling, storage, and retrieval protocol designed to emulate common file system operations and to provide aspects of mutability to your data hierarchy on [Arweave]'s otherwise permanent, immutable data storage blockweave. # CLI Help (/sdks/(clis)/ardrive-cli/cli-help) Learn to use any command: ```shell ardrive --help ``` # CLI Version (/sdks/(clis)/ardrive-cli/cli-version) You can print out the version by running any of: ```shell ardrive --version ardrive -V ``` # Data Portability (/sdks/(clis)/ardrive-cli/data-portability) Data uploaded via the ArDrive CLI, once indexed by Arweave's Gateways and sufficiently seeded across enough nodes on the network, can be accessed via all other ArDrive applications including the [ArDrive Web application][ardrive-web-app] at https://app.ardrive.io. All transactions successfully executed by ArDrive can always be inspected in the [Viewblock blockchain explorer]. # ArDrive CLI (/sdks/(clis)/ardrive-cli) **For AI and LLM users**: Access the complete ArDrive CLI documentation in plain text format at llm.txt for easy consumption by AI agents and language models. # ArDrive CLI Please refer to the [source code](https://github.com/ardriveapp/ardrive-cli) for SDK details. # Intended Audience (/sdks/(clis)/ardrive-cli/intended-audience) This tool is intended for use by: - ArDrive power users with advanced workflows and resource efficiency in mind: bulk uploaders, those with larger storage demand, game developers, nft creators, storage/db admins, etc. - Automation tools - Services - Terminal aficionados - Extant and aspiring cypherpunks For deeper integrations with the [ArDrive] platform, consider using the [ArDrive Core][ardrive-core] (Node) library's configurable and intuitive class interfaces directly within your application. To simply install the latest version of the CLI to your local system and get started, follow the [Quick Start](#quick-start) instructions. To build and/or develop the CLI from source, follow the [Build and Run from Source](#build-and-run-from-source) instructions. In either case, be sure to satisfy the requirements in the [Prerequisites](#prerequisites) section. # Limitations (/sdks/(clis)/ardrive-cli/limitations) **Number of files in a bulk upload:** Theoretically unlimited **Max individual file size**: 2GB (Node.js limitation) **Max file name length**: 255 bytes **Max ANS-104 bundled transaction size:** 500 MiB per bundle. App will handle creating multiple bundles. **Max ANS-104 data item counts per bundled transaction:** 250 Files per bundle (500 Data Items). # Using the CLI # Wallet Operations (/sdks/(clis)/ardrive-cli/wallet-operations) Browsing of ArDrive public data is possible without the need for an [Arweave wallet][kb-wallets]. However, for all write operations, or read operations without encryption/decryption keys, you'll need a wallet. As you utilize the CLI, you can use either your wallet file or your seed phrase interchangeably. Consider the security implications of each approach for your particular use case carefully. If at any time you'd like to generate a new wallet altogether, start by generating a new seed phase. And if you'd like to use that seed phrase in the form of a wallet file, or if you'd like to recover an existing wallet via its seed phrase, use either or both of the following commands: ```shell ardrive generate-seedphrase "this is an example twelve word seed phrase that you could use" ardrive generate-wallet -s "this is an example twelve word seed phrase that you could use" > /path/to/wallet/file.json ``` Public attributes of Arweave wallets can be retrieved via their 43-character Arweave wallet address. You can retrieve the wallet address associated with [your wallet file or 12-word seed phrase][kb-wallets] (e.g. wallets generated by [ArConnect][arconnect]) like so: ```shell ardrive get-address -w /path/to/wallet/file.json ardrive get-address -s "this is an example twelve word seed phrase that you could use" HTTn8F92tR32N8wuo-NIDkjmqPknrbl10JWo5MZ9x2k ``` You'll need AR in your wallet for any write operations you perform in ArDrive. You can always check your wallet balance (in both AR and Winston units) by performing: ```shell ardrive get-balance -w /path/to/wallet/file.json ardrive get-balance -a "HTTn8F92tR32N8wuo-NIDkjmqPknrbl10JWo5MZ9x2k" 1500000000000 Winston 1.5 AR ``` If, at any time, you need to send AR out of your wallet to another wallet address, you may perform: ```shell ardrive send-ar -w /path/to/wallet/file.json --dest-address "HTTn8F92tR32N8wuo-NIDkjmqPknrbl10JWo5MZ9x2k" --ar-amount 2.12345 ``` # Add the Skill to Your Project (/sdks/(clis)/ario-deploy/(claude-code-integration)/add-the-skill-to-your-project) ```bash mkdir -p .claude/skills curl -o .claude/skills/deploy.md https://raw.githubusercontent.com/ar-io/ar-io-deploy/main/examples/claude-skill/deploy.md ``` Then in Claude Code, say: - "deploy to ar.io" - "deploy my app to arweave" - "set up CI/CD for ar.io deployment" Claude will build your project, detect the output folder, and run the deploy with the right flags. # What the Skill Does (/sdks/(clis)/ario-deploy/(claude-code-integration)/what-the-skill-does) 1. **Detects your build folder** (`./dist`, `./build`, `./out`) 2. **Checks for credentials** (`DEPLOY_KEY` env var or wallet file) 3. **Installs `@ar.io/deploy`** if not already available 4. **Runs the deployment** with appropriate flags 5. **Reports results** — transaction ID, Arweave URL, ArNS URL See [`examples/claude-skill/`](https://github.com/ar-io/ar-io-deploy/tree/main/examples/claude-skill) for the full skill file and customization options. --- # Advanced Usage (/sdks/(clis)/ario-deploy/(commands)/advanced-usage) Deploy to an undername (subdomain) — the ArNS authority key is a Solana wallet: ```bash ario-deploy deploy --use-arns --arns-name my-app --wallet ./wallet.json --arns-wallet ./arns-id.json --undername staging ``` Deploy with a custom TTL: ```bash ario-deploy deploy --use-arns --arns-name my-app --wallet ./wallet.json --arns-wallet ./arns-id.json --ttl-seconds 7200 ``` Update ArNS on devnet (or against a custom RPC): ```bash ario-deploy deploy --use-arns --arns-name my-app --wallet ./wallet.json --arns-wallet ./arns-id.json --cluster devnet ario-deploy deploy --use-arns --arns-name my-app --wallet ./wallet.json --arns-wallet ./arns-id.json --rpc-url https://my-rpc.example.com ``` Upload using an Ethereum wallet (file): ```bash ario-deploy deploy --sig-type ethereum --wallet ./private-key.txt ``` Upload using a Solana wallet (base58 private key): ```bash ario-deploy deploy --sig-type solana --private-key "\" ``` # Direct Commands (/sdks/(clis)/ario-deploy/(commands)/direct-commands) Use flags for faster, scriptable deployments: ```bash ario-deploy deploy --wallet ./wallet.json ario-deploy deploy --use-arns --arns-name my-app --wallet ./wallet.json --arns-wallet ./arns-id.json ``` Deploy using private key directly: ```bash ario-deploy deploy --private-key "$(cat wallet.json)" ``` Deploy using environment variable: ```bash DEPLOY_KEY=$(base64 -i wallet.json) ario-deploy deploy --deploy-folder ./dist ``` Deploy a specific folder: ```bash ario-deploy deploy --wallet ./wallet.json --deploy-folder ./build ``` Deploy a single file: ```bash ario-deploy deploy --wallet ./wallet.json --deploy-file ./path/to/file.txt ``` `--deploy-file` overrides `--deploy-folder`, and the file is uploaded as one transaction with **no manifest** — an ArNS name pointed at it resolves straight to that file, served with its own content type. Useful for a PDF, a dataset, or a single page. Manifest-only options such as `--fallback-file` do not apply. # Interactive Mode (Easiest) (/sdks/(clis)/ario-deploy/(commands)/interactive-mode-easiest) Run the deploy command without arguments to be guided through all deployment options: ```bash ario-deploy deploy ``` When ArNS details aren't supplied via flags, `deploy` asks whether you want to update an ArNS name (defaulting to yes) and, if so, prompts for the details. It will guide you through: - Whether to update an ArNS name (and which one) - Wallet method (file, string, or environment variable) - What to deploy (folder or file) - Advanced options (optional: undername, TTL, Solana cluster) Pass `--arns-name` (or `--use-arns`) to skip the ArNS confirmation, or use the `upload` command for an upload-only run. In a non-interactive environment (CI, or no TTY) `deploy` does not prompt — supply everything via flags or `DEPLOY_KEY`. # Single-page apps (/sdks/(clis)/ario-deploy/(commands)/single-page-apps) An Arweave path manifest maps each path to a transaction, and a gateway returns 404 for any path the manifest does not list. That is correct for static files but wrong for a single-page app, whose routes are not files — `/settings` is invented by the router and exists nowhere on disk. Without a fallback the root loads and every deep link 404s. Manifests have a `fallback` for exactly this, and `ario-deploy` sets it automatically when the build emits a `404.html`: ```bash ario-deploy deploy --deploy-folder ./dist ``` Most SPA builds do not emit one. Either copy your entry point before deploying: ```bash cp dist/index.html dist/404.html ``` …or name the fallback directly: ```bash ario-deploy deploy --deploy-folder ./dist --fallback-file index.html ``` The file must exist in the deploy folder; a path that is not there fails before anything is uploaded, so a typo costs nothing. > Deep links can appear broken for up to a minute after a redeploy while > gateways serve cached 404s from the previous manifest. Confirm with a > cache-busting query string (`/settings?x=1`) before assuming the deploy failed. # Upload/deploy without ArNS (/sdks/(clis)/ario-deploy/(commands)/upload-deploy-without-arns) `deploy` uploads without updating ArNS by default. You can also use the `upload` command explicitly for the same Turbo upload, dedupe cache, and payment options as deploy, minus ArNS flags: ```bash ario-deploy deploy --wallet ./wallet.json --deploy-folder ./dist ario-deploy upload --wallet ./wallet.json --deploy-folder ./dist ario-deploy upload --wallet ./wallet.json --deploy-file ./dist/index.html DEPLOY_KEY=$(base64 -i wallet.json) ario-deploy upload --deploy-folder ./dist ``` # Basic Usage (/sdks/(clis)/ario-deploy/(github-action)/basic-usage) ```yaml - uses: ar-io/ar-io-deploy@v2.0.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} # upload key (pays for the upload) arns-key: ${{ secrets.ARNS_KEY }} # Solana ArNS authority key arns-name: myapp deploy-folder: ./dist ``` # Disabling Deduplication (/sdks/(clis)/ario-deploy/(github-action)/disabling-deduplication) By default, the action caches transaction IDs to avoid re-uploading unchanged files. To disable this: ```yaml - name: Deploy without dedupe uses: ar-io/ar-io-deploy@v2.0.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} deploy-folder: ./dist no-dedupe: 'true' ``` You can also limit the cache size: ```yaml - name: Deploy with limited cache uses: ar-io/ar-io-deploy@v2.0.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} deploy-folder: ./dist dedupe-cache-max-entries: '1000' ``` # Incremental Uploads (/sdks/(clis)/ario-deploy/(github-action)/github-action-incremental-uploads) A CI job runs from a fresh checkout, so the restored transaction cache is often missing or stale — and then every redeploy pays for the whole bundle again. `incremental: 'true'` recovers those transaction ids from your wallet's own past uploads on chain, so only the files that actually changed are paid for. See [Incremental uploads](#incremental-uploads). Requires v1.2.0 or later; pin the version, since the floating v1` tag may lag. ```yaml - name: Deploy only what changed uses: ar-io/ar-io-deploy@v1.2.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} deploy-folder: ./dist incremental: 'true' ``` --- # PR Preview Deployments (/sdks/(clis)/ario-deploy/(github-action)/pr-preview-deployments) Automatically deploy preview builds for each pull request. The `preview` mode auto-generates an undername from the PR number and posts a comment with the preview URL: ```yaml name: Deploy PR Preview on: pull_request: types: [opened, synchronize] jobs: deploy-preview: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '20' - name: Install dependencies run: npm ci - name: Build run: npm run build - name: Deploy Preview uses: ar-io/ar-io-deploy@v2.0.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} arns-key: ${{ secrets.ARNS_KEY }} arns-name: myapp preview: 'true' github-token: ${{ secrets.GITHUB_TOKEN }} deploy-folder: ./dist ``` When `preview` is enabled, the action will: - Auto-generate an undername like `myapp-repo-pr-123` from the repository name and PR number - Post a comment on the PR with the preview URL (the token needs `pull-requests: write`) - Update the comment on subsequent pushes instead of creating new ones Preview undernames are not removed when the PR closes; each costs one of the ArNS name's undername slots until you remove it. The action skips every step on a `closed` event, so subscribing to it costs nothing. # Production Deployment (/sdks/(clis)/ario-deploy/(github-action)/production-deployment) Deploy to your base ArNS name when pushing to main: ```yaml name: Deploy to Production on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '20' - name: Install dependencies run: npm ci - name: Build run: npm run build - name: Deploy to Permaweb uses: ar-io/ar-io-deploy@v2.0.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} arns-key: ${{ secrets.ARNS_KEY }} arns-name: myapp deploy-folder: ./dist ``` # Updating ArNS (Solana) (/sdks/(clis)/ario-deploy/(github-action)/updating-arns-solana) ArNS updates run against the Solana ARIO programs. Provide the Solana ArNS authority key via `arns-key` (a base58 Solana secret key); the upload is still paid for by `deploy-key`. Use `cluster` to target `mainnet` (default) or `devnet`, and `rpc-url` for a custom RPC endpoint. ```yaml - name: Deploy and update ArNS uses: ar-io/ar-io-deploy@v2.0.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} # upload key arns-key: ${{ secrets.ARNS_KEY }} # Solana ArNS authority key arns-name: myapp deploy-folder: ./dist cluster: mainnet ``` # With On-Demand Payment (/sdks/(clis)/ario-deploy/(github-action)/with-on-demand-payment) ```yaml - name: Deploy with ARIO on-demand uses: ar-io/ar-io-deploy@v2.0.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} arns-key: ${{ secrets.ARNS_KEY }} arns-name: myapp deploy-folder: ./dist sig-type: solana # ARIO is a Solana token, so the upload key must be Solana on-demand: ario max-token-amount: '2.0' ``` # ArNS authority key (ARNS_KEY) (/sdks/(clis)/ario-deploy/(prerequisites)/arns-authority-key-arns-key) Set a base58-encoded **Solana** secret key as `ARNS_KEY`, or pass a `solana-keygen` `id.json` file via `--arns-wallet` (or a base58 string via `--arns-private-key`). This key must control the ArNS name being updated. ⚠️ **Important:** Use dedicated wallets for deployments to minimize security risks. Ensure your upload wallet has sufficient Turbo Credits for uploads. # Upload key (DEPLOY_KEY) (/sdks/(clis)/ario-deploy/(prerequisites)/upload-key-deploy-key) 1. **Arweave signer (default):** Encode your Arweave wallet key in base64 and set it as `DEPLOY_KEY`: ```bash base64 -i wallet.json | pbcopy ``` 2. **Ethereum/Polygon signers:** Use your raw private key (no encoding needed) as `DEPLOY_KEY`. 3. **Solana signer:** Use a base58-encoded secret key as `DEPLOY_KEY`, or a `solana-keygen` `id.json` byte-array wallet file via `--wallet`. To make a new one, run `ario-deploy keygen`. #### Create a wallet with keygen` ```bash ario-deploy keygen # writes ~/.ar.io/wallets/\.json ario-deploy keygen --out ~/wallets/my-wallet.json ``` `keygen` writes a new Solana key in `solana-keygen` `id.json` format. By default the file goes in `~/.ar.io/wallets/`, a folder in your home directory outside any project, named after the wallet's address. It prints the file path, the public address, the wallet's free upload allowance and the exact `deploy` command to run next. It never prints the secret key, and it never overwrites an existing file. Add `--dev` to look up the allowance on the Turbo sandbox. Who can read the file: - **Linux and macOS:** the file has mode `0600` and the wallets folder `0700`, so only your account can read them. - **Windows:** `keygen` removes inherited permissions with `icacls` and grants access to your account only. When that fails it prints a warning, and other accounts on the computer might be able to read the file. **Never put the wallet inside the folder you deploy.** An upload is permanent and public, and anyone who reads the file controls the wallet. `deploy` and `upload` refuse to publish any key they were given, including `DEPLOY_KEY` and `ARNS_KEY`, and any file that can be proved to be a private key (see [Files that are never uploaded](#files-that-are-never-uploaded)). --out` accepts any path, but `keygen` warns when the path is inside the current folder. When the file is inside a git repository, `keygen` adds it to the repository's `.gitignore` and then asks git to confirm that it is ignored and not tracked. If git does not confirm both, it prints a warning instead. Back up the wallet file. It is the only copy, anyone who has it controls the wallet, and nobody can recover it for you. Never paste its contents anywhere. # Bundler service (/sdks/(clis)/ario-deploy/bundler-service) Uploads go through Turbo: an upload service that accepts signed data items, and a payment service that answers balance, price and top-up questions. The two belong to the same network, and ario-deploy configures them together. | When to use | Flags | | ---------------------------------- | ------------------------------------------------------------------ | | **Default** (production) | none: `https://upload.ardrive.io` and `https://payment.ardrive.io` | | **Development sandbox** | `--dev` | | **Custom or self-hosted services** | `--uploader \` and `--payment-url \` | `--dev` selects both sandbox services (`https://upload.services.ar-io.dev` and `https://payment.services.ar-io.dev`) and testnet RPCs for `--on-demand`. Passing the sandbox URL to `--uploader` alone does the same. A custom `--uploader` without `--payment-url` keeps the production payment service and prints a warning, since balance checks and top-ups go there. ```bash ario-deploy upload --wallet ./wallet.json --deploy-folder ./dist --dev ``` The free upload limit is read from the upload service, so it follows the network: 105 KiB per item in production, 5 MiB in the sandbox. **A `--dev` upload is not permanent.** It goes to the Turbo sandbox for testing, production gateways do not serve it, and the result output says so. Never switch to `--dev` to get past an error on production: the URL it prints does not work as a permanent site. # CLI in GitHub Actions (/sdks/(clis)/ario-deploy/cli-in-github-actions) You can also use the CLI directly in your workflows: **Basic Workflow:** ```yaml name: Deploy to Permaweb on: push: branches: - main jobs: publish: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: pnpm/action-setup@v3 with: version: 9 - uses: actions/setup-node@v4 with: node-version: 20 cache: 'pnpm' - run: pnpm install - run: pnpm run deploy env: DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} ``` **With On-Demand Payment:** ```yaml name: Deploy to Permaweb with On-Demand Payment on: push: branches: - main jobs: publish: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: pnpm/action-setup@v3 with: version: 9 - uses: actions/setup-node@v4 with: node-version: 20 cache: 'pnpm' - run: pnpm install - run: pnpm build - name: Deploy with ARIO on-demand run: ario-deploy deploy --arns-name my-app --sig-type solana --on-demand ario --max-token-amount 2.0 env: DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} # upload key (pays for the upload) ARNS_KEY: ${{ secrets.ARNS_KEY }} # Solana ArNS authority key # Or upload with Ethereum and Base-ETH on-demand payment (upload only; ArNS requires Solana): # - name: Upload with Base-ETH on-demand # run: | # ario-deploy upload \ # --sig-type ethereum \ # --on-demand base-eth \ # --max-token-amount 0.2 # env: # DEPLOY_KEY: ${{ secrets.ETH_PRIVATE_KEY }} ``` # Command Options (/sdks/(clis)/ario-deploy/command-options) **`deploy`** (upload by default, optional ArNS update): - `--use-arns`: Update an ArNS/ANT record after upload. When ArNS details aren't supplied and you're in a TTY, `deploy` asks by default. - `--arns-name, -n`: The ArNS name to update. Required when using `--use-arns`; also implies ArNS mode. - `--cluster, -p`: Solana cluster for ArNS updates. Choices: `mainnet`, `devnet`. Default: `mainnet` - `--rpc-url`: Optional Solana RPC URL override for ArNS updates - `--deploy-folder, -d`: Folder to deploy. Default: `./dist` - `--deploy-file, -f`: Deploy a single file instead of a folder (no manifest is created) - `--fallback-file`: Path, relative to the deploy folder, served for routes the manifest does not list. Defaults to `404.html` when the build emits one. See [Single-page apps](#single-page-apps). - `--undername, -u`: ANT undername to update. Default: `@` - `--ttl-seconds, -t`: TTL in seconds for the ANT record (60-86400). Default: `60` - `--skip-arns-check`: Update the record even if the ArNS key does not appear to own or control the name. Without it, a deploy whose key cannot update the name is refused before anything is uploaded. Use it only right after the name changed hands, when the ANT's recorded owner can lag. Upload key (pays for the upload): - `--sig-type, -s`: Signer type for the upload key. Choices: `arweave`, `ethereum`, `polygon`, `solana`. Default: `arweave` - `--wallet, -w`: Path to the upload wallet file (JWK for Arweave, private key for Ethereum/Polygon, `solana-keygen` `id.json` for Solana). Falls back to `DEPLOY_KEY`. - `--private-key, -k`: Upload private-key string (alternative to `--wallet`). JWK JSON for Arweave, hex for EVM chains, base58 secret key for Solana. ArNS authority key (controls the name, signs the update — always Solana): - `--arns-wallet`: Path to the Solana `solana-keygen` `id.json` wallet that controls the ArNS name. Falls back to `ARNS_KEY`. - `--arns-private-key`: Base58 Solana secret key for the ArNS authority (alternative to `--arns-wallet`). Falls back to `ARNS_KEY`. Payment: - `--on-demand`: Top up with this token if the credits cannot cover the upload. Choices: `ario`, `solana`, `solana-usdc` (Solana keys), `base-eth`, `base-usdc` (EVM keys). Requires `--max-token-amount`. See [On-Demand Payment](#on-demand-payment). - `--max-token-amount`: Most the top-up may spend, in whole tokens (e.g. `0.5`). Caps the whole deploy. - `--paid-by`, `--ignore-approvals`, `--use-signer-balance-first`: who pays. See [Shared credits](#shared-credits). - `--dev`: Use Turbo's development sandbox for both upload and payment. - `--uploader` (alias `--upload-url`), `--payment-url`: Custom Turbo services. See [Bundler service](#bundler-service). Upload behaviour: - `--no-dedupe`: Disable deduplication (do not cache or reuse previous uploads) - `--dedupe-cache-max-entries`: Maximum number of entries to keep in the dedupe cache (LRU). Default: `10000` - `--incremental`: Reuse files already on Arweave, including on a machine with no local cache. Off by default. Cannot be combined with `--no-dedupe` or `--dedupe-cache-max-entries 0`. See [Incremental uploads](#incremental-uploads). - `--incremental-gateway`: Gateway whose GraphQL endpoint is queried for past uploads when `--incremental` is set. Default: `https://turbo-gateway.com` - `--compress`: Compress files before upload and tag them with `Content-Encoding`. Choices: `gzip`, `br`, `none` (default). See [Compression](#compression). - `--compress-exclude`: Comma-separated globs of files to upload uncompressed, e.g. `"llms*.txt,*.md"` **`upload`** (explicit upload without ArNS): accepts `--deploy-folder`, `--deploy-file`, `--fallback-file`, wallet/signer flags, the payment flags, the dedupe and incremental flags, and `--compress` / `--compress-exclude` only. # Compression (/sdks/(clis)/ario-deploy/compression) Arweave storage is priced per byte, and HTML, JavaScript, CSS and JSON typically shrink 5-8x when compressed (a 169 MB static docs site uploads as 22 MiB). `--compress` compresses each file before upload and tags it with `Content-Encoding`; gateways return that header, and browsers decompress transparently. ```bash ario-deploy deploy --wallet ./wallet.json --deploy-folder ./out --compress gzip ``` In the GitHub Action (`compress` needs v1.1.0 or later; pin the version, since the floating `v1` tag is moved by hand and may lag): ```yaml - uses: ar-io/ar-io-deploy@v1.1.0 with: deploy-key: ${{ secrets.DEPLOY_KEY }} deploy-folder: ./dist compress: gzip compress-exclude: 'llms*.txt,*.md' ``` - **Prefer `gzip`.** Gateways send the encoded bytes to every client, whether or not it asked for compression. Every browser and HTTP library understands gzip; `br` was ~17% smaller than gzip on a static docs site, but some non-browser clients cannot decode it. - **Formats that are already compressed** are uploaded as-is: already-compressed formats (JPEG, PNG, GIF, WebP, AVIF, HEIC, WOFF/WOFF2, MP3, M4A, Ogg/Opus, MP4, WebM, and zip/gz/br/bz2/xz/zst/7z/rar archives). Other images and fonts (`.svg`, `.ico`, `.ttf`, `.otf`) are compressed. Every other file is compressed, even a tiny one gzip makes a few bytes larger, so its tags always match how it was planned. - **Exclude files meant for non-browser clients** with `--compress-exclude`, e.g. text files that tools fetch with `curl`: `--compress-exclude "llms*.txt,*.md"`. A pattern without `/` matches the file name in any directory. - **Gateways must label items they have not indexed yet.** Right after a deploy, a gateway may serve a data item before it has indexed the item's tags. An ar-io-node without the fix for that (ar-io-node #964/#966) sends the gzip bytes with no Content-Encoding` header, and browsers render garbage until the item is indexed -- or indefinitely, on a gateway that never indexes the bundle. The ar.io and Turbo gateways (`turbo-gateway.com`, `ardrive.net`, and those serving `*.ar.io`) have the fix; other operators get it by upgrading. Deploy to a test undername first and load it through each gateway that matters, including through Wayfinder, which may pick any gateway. - **Deduplication still works**, including `--incremental`. Compressed uploads are cached (and found on chain) under their own key, so turning compression on re-uploads each file once, and later deploys skip unchanged files as usual. # Deduplication (/sdks/(clis)/ario-deploy/deduplication) By default, ario-deploy caches your deployment log to prevent uploading duplicate (unchanged) files. This saves both time and upload costs by reusing existing data on Arweave. **How it works:** 1. When you deploy, ario-deploy hashes each file in your build 2. It checks the local cache for matching hashes from previous uploads 3. Files that haven't changed are skipped - the existing transaction ID is reused 4. Files identical to another file in the same deploy are uploaded once and share its transaction (static exports often write the same payload under several names) 5. Only new or modified files are uploaded to Arweave, and each id is written to the cache the moment it lands, so a deploy that fails or is interrupted part-way does not pay for those files again 6. The cache is stored locally in `.ario-deploy/transaction-cache.json`, with a separate file per Turbo network (`--dev` uploads never stand in for production ones) Entries are keyed on the file's content and content type (plus encoding when compressed), so byte-identical files served as different types are never confused. Caches written by 1.x, keyed on the hash alone, are still honoured, except for empty files, whose hash says nothing about their type. Symlinks inside the deploy folder are followed only while they point inside it; a link to a file outside the folder stops the deploy, since uploading it would publish that file permanently. #### Files that are never uploaded The check is built to stop a key from being published by accident. It cannot promise to find a key that someone disguises on purpose (reversed, split across files, or in an encoding of their own), so keep keys outside the project folder. Before any request is made, `deploy` and `upload` list the files once, read every one of them and refuse to publish a private key. Only the files that were checked are uploaded. The error names the file and never prints the key. There is no flag to override this. **Keys the run holds.** Every key the command was given (`--wallet`, `--arns-wallet`, `--private-key`, `--arns-private-key`, `DEPLOY_KEY` and `ARNS_KEY`, whichever are set) is searched for in every file and every file name. The search covers: - the key's raw bytes, hex in any case (also as `0x0c, 0x22`, `\x0c\x22` or `0c:22`), a decimal byte list such as `[12, 34, ...]`, base58, and base64 or base64url at any alignment - an Arweave key's private JWK fields, the base64 JWK that `DEPLOY_KEY` holds, and each key file base64-encoded, as in a `data:` URI - text with spaces, line breaks, string concatenation, `\u`, `\x` and percent escapes removed, UTF-16 text, every string of a JSON file, and printable text inside binary files Gzip, brotli (`.br`), zip and tar files are opened and searched, including archives inside archives up to three levels deep. A compressed file is refused as one that could not be checked when it expands to more than 1 GiB, is nested deeper, is damaged or encrypted, or is a 7z, RAR, xz, bzip2, Zstandard or cabinet archive. The run also stops when a wallet file, or a hard link or symlink to one, is inside the deploy folder or is the `--deploy-file`. **Keys the run does not hold.** The run also stops on what can be proved to be a private key, in files and inside the archives above: - an environment file: `.env`, `.env.local`, `.env.example`, `prod.env`, in any case (scripts and pages such as `env.js` are not refused for their name) - a PEM private key block (`-----BEGIN PRIVATE KEY-----` and the RSA, EC, DSA, OPENSSH and ENCRYPTED forms); public keys and certificates are not refused - a Solana keypair written as a byte array, base58 or hex, checked by deriving its public half from its seed - an object with a private exponent `d` and an RSA-sized modulus `n` (2048 bits or more) together, as JSON, inside a string or base64-encoded What is not detected for a key the run does not hold: a 32-byte seed or an Ethereum key on its own in any form (it cannot be told from a hash), brotli data without a `.br` name, and keys inside compressed parts of other formats, such as PNG text chunks or PDF streams. `.git` folders are left out of folder uploads, with a one-line note. The Turbo credit check runs after this planning step, so it prices only what will actually be uploaded, not the whole folder. **Disable deduplication:** If you need to force a fresh upload of all files (e.g., for debugging or to ensure a completely new deployment). Files that are identical within the same deploy are still uploaded once and share a transaction, since that reuses nothing from earlier deploys: ```bash ario-deploy deploy --wallet ./wallet.json --no-dedupe ``` **Limit cache size:** The dedupe cache uses an LRU (Least Recently Used) eviction strategy. By default, it keeps up to 10,000 entries. You can adjust this limit: ```bash ario-deploy deploy --wallet ./wallet.json --dedupe-cache-max-entries 1000 ``` **Cache location:** The cache files are stored in `.ario-deploy/` in your project root. You can: - Add it to `.gitignore` if you don't want to share cache across team members - Commit it to share cached transaction IDs with your team (reduces duplicate uploads) - Delete it to start fresh: `rm -rf .ario-deploy/` # Dependencies (/sdks/(clis)/ario-deploy/dependencies) - **@ar.io/sdk** - For ANT operations and ArNS management on Solana - **@ardrive/turbo-sdk** - For fast file uploads to Arweave (and signer types) - **@solana/kit** - Solana RPC clients and transaction signers for ArNS updates - **bs58** - Base58 encoding/decoding for Solana keys - **@oclif/core** - CLI framework - **mime-types** - MIME type detection # Features (/sdks/(clis)/ario-deploy/features) - **Turbo SDK Integration:** Uses Turbo SDK for fast, reliable file uploads to Arweave - **On-Demand Payment:** Top up Turbo credits with ARIO, SOL, USDC or Base ETH when a deploy needs them - **Shared credits:** Spend credits other wallets have shared with your upload key - **Arweave Manifest v0.2.0:** Creates manifests with fallback support for SPAs - **Optional ArNS Updates:** Updates ArNS records via ANT with new transaction IDs - **Automated Workflow:** Integrates with GitHub Actions for continuous deployment - **Git Hash Tagging:** In CI (GitHub Actions), tags uploaded data items with the deploying commit SHA. Under `--incremental` the tag moves to the manifest only — see [Incremental uploads](#incremental-uploads) for why a per-deploy tag on a file cannot be allowed. - **Incremental Uploads (opt-in):** `--incremental` pays only for the files that actually changed, recovering the rest from your own past uploads even on a machine with no local cache. See [Incremental uploads](#incremental-uploads). - **404 Fallback Detection:** Automatically sets `404.html` as the manifest fallback when present, so deep links into a single-page app resolve instead of 404ing. Override with `--fallback-file \` — an SPA that only builds `index.html` can point at that instead. - **Network Support:** ArNS updates run against the Solana ARIO programs on `mainnet` or `devnet`, with an optional custom RPC URL - **Flexible Deployment:** Supports deploying a folder or a single file - **Modern CLI:** Built with oclif for a robust command-line experience - **TypeScript:** Fully typed for better developer experience # Free tier (/sdks/(clis)/ario-deploy/free-tier) Turbo uploads small files for free. The limits are: - **105 KiB per file** (per data item). A larger file is billed. - **10 MiB over the lifetime of a wallet**, and **10 MiB over the lifetime of an IP range**. Turbo meters both, and an upload is free only while both have allowance left. `ario-deploy` can check the wallet's allowance before it uploads. It cannot check the IP range, so a deploy can pass the credit check ("within this wallet's free tier") and still be refused at upload time with HTTP 402 when other people on the same network have used the range's allowance. See [402 Payment Required](#troubleshooting). To go past the free tier, add [Turbo credits](https://turbo.ardrive.io), use [`--on-demand`](#on-demand-payment), or have credits [shared](#shared-credits) to your wallet. The sandbox (`--dev`) has its own, larger limit and is for testing only: see [Bundler service](#bundler-service). # Incremental uploads (/sdks/(clis)/ario-deploy/incremental-uploads) `--incremental` makes a redeploy pay only for the files that actually changed. Arweave storage is permanent, so re-uploading byte-identical files buys nothing. Build tools content-hash their output, so between two deploys of a real site only a couple of entry chunks change — everything else is already on chain and can be referenced by its existing transaction id in the path manifest. ```bash ario-deploy deploy --wallet ./wallet.json --incremental ``` Measured on a 1,229-file static docs site, redeployed from a fresh CI runner with no local cache and `--compress gzip`: 1,228 files were found on chain and one was uploaded (2.5 MiB), where a cold deploy uploaded 33 MiB. **How it works:** 1. Every file in the folder is hashed (SHA-256). 2. Each file is looked up in the local dedupe cache, and then — for anything the cache cannot answer — among your own past uploads on chain. 3. Only the remainder is uploaded, and each transaction id reaches the cache as it lands — on the leading edge, then coalesced onto a 500 ms trailing timer, and flushed on `SIGINT`/`SIGTERM` so Ctrl-C does not lose files you have already paid for. `SIGHUP` and `SIGBREAK` are not handled, so a closed terminal or a dropped SSH session can still lose the current batch; CI is covered, since GitHub Actions cancels with `SIGINT` then `SIGTERM`. 4. The manifest is assembled from the remembered ids plus the new ones. **Why the on-chain lookup matters:** every uploaded file carries a `File-SHA256` tag, which makes it findable again from nothing but the bytes on disk. That is what a CI job needs. CI runs from a fresh checkout, so `.ario-deploy/transaction-cache.json` is often missing or stale even with `actions/cache` restoring it — and without the on-chain lookup every redeploy pays for the whole bundle again. **The tag invariant:** a data item's id covers its tags, so a tag whose value changes between deploys — a commit SHA above all — moves every file's id on every deploy and defeats deduplication. The failure is silent: the upload succeeds, the manifest is correct, and the bill doubles. In incremental mode files therefore carry only deploy-invariant tags (`App-Name`, `Content-Type`, `File-SHA256`, plus `Content-Encoding` when compressed), and the `GIT-HASH` provenance tag rides on the manifest instead, which is rewritten every deploy anyway. The tag set is asserted in code, so a future addition fails loudly rather than quietly costing money. **Reuse is keyed on content type as well as content.** Two files with identical bytes served under different types — `a.json` and `b.txt` — stay two uploads, because a gateway serves whatever `Content-Type` the data item carries and collapsing them would serve one of them as the other. Cache entries are therefore keyed `\|\` (plus `|\` when compressed) in every mode. Incremental mode never falls back to a 1.x hash-only entry, so switching a project to `--incremental` may re-upload once and is cheap from then on. **What it trusts:** only your own wallet's past transactions, matched on the 43-character address a gateway indexes an owner as — derived locally as `base64url(sha256(publicKey))`, which is correct for all four signer types. Every result is then re-checked against the owner and content type in the gateway's own response, which catches a buggy or misconfigured gateway. It cannot catch a malicious one, since the owner, tags and id all come from that same response: point `--incremental-gateway` only at a gateway you trust, because a wrong id would land in both the permanent manifest and the local cache. **Limits and caveats:** - **Lookups are batched.** Hashes are sent 100 per GraphQL request, because gateways cap the size of a query (an ar.io gateway refuses ~1,100 hashes with "Max query size exceeded"). A site of any size is covered; each batch is paged until its files are accounted for, up to 20 pages. - **The credits pre-flight prices only what will be sent**: the files still to upload plus an estimate of the manifest, which is uploaded on every deploy. A fully reused redeploy is priced at the manifest alone. - **Gateway GraphQL indexing lags an upload by a few minutes.** Two machines deploying the same _new_ file at the same moment can each pay for it. It costs a fraction of a cent and never produces a wrong manifest. - **A gateway that is slow, unreachable or erroring costs reuse, not correctness.** Requests that fail transiently (HTTP 429 or 5xx, a timeout, a network error) are retried twice with a short backoff. A batch that still fails costs only its own files, which are uploaded again, and the run says how many batches it could not look up. If no batch can be looked up at all, the run warns and uploads everything the local cache does not already hold. - **A doomed deploy takes longer to say so.** Every queued upload settles before a failure is reported, so a systemic failure (bad credentials, exhausted credits) on a very large folder surfaces at the end rather than immediately. The same uploads were always attempted, so the bill is unchanged; the alternative stranded ids that had been paid for and never written down. - **Ignored for `--deploy-file`.** Reuse works through the manifest, and a single file has no manifest. The run warns rather than silently doing nothing. - **Cache entries are keyed differently in each mode**, so a project that toggles `--incremental` on and off stores up to two entries per file against the shared `--dedupe-cache-max-entries` cap: `\` (or `gzip:\` when compressed) without it, and `\|\` (or `\|\|gzip`) with it. **Notes:** - Off by default. Nothing changes for an existing pipeline until you pass the flag. - Refused alongside `--no-dedupe` or `--dedupe-cache-max-entries 0`, which ask for the opposite. - Works with `--compress`: each file's `File-SHA256` is the hash of the file on disk, and a compressed upload also carries `Content-Encoding`, so a lookup only ever reuses an upload made with the same encoding. Turning compression on or off uploads each file once more, then reuse resumes. - The lookup uses `https://turbo-gateway.com/graphql` by default, where uploads made through Turbo are indexed within minutes (about 5-7 in our measurements), before they are bundled into a block. Override it with `--incremental-gateway` — for example when uploading through another bundler with `--uploader`. # On-Demand Payment (/sdks/(clis)/ario-deploy/on-demand-payment) With `--on-demand`, a deploy whose credits cannot cover the upload buys what it is short, once, before the first file uploads. `--max-token-amount` is required and caps that purchase for the whole deploy. The token has to be one your upload key can pay with: | Upload key (`--sig-type`) | `--on-demand` tokens | | ------------------------- | ------------------------------------- | | `solana` | `ario`, `solana`, `solana-usdc` | | `ethereum`, `polygon` | `base-eth`, `base-usdc` | | `arweave` | none: top up Turbo credits in advance | ```bash ario-deploy deploy --sig-type solana --wallet ./id.json --deploy-folder ./dist --on-demand ario --max-token-amount 1.5 ario-deploy deploy --sig-type ethereum --private-key "0x..." --on-demand base-eth --max-token-amount 0.1 ``` **How it works:** 1. Each file the deploy will actually upload is priced through Turbo. A file within the upload service's free size limit is free only while your wallet's free-tier allowance lasts, so once that is spent small files are priced too. 2. If the credits you can spend (see [Shared credits](#shared-credits)) cover it, nothing is bought. 3. Otherwise the shortfall plus a 10% buffer is converted at Turbo's quoted rate. If that exceeds `--max-token-amount`, the deploy stops before paying anything. 4. The top-up is paid once, and the deploy waits up to two minutes for Turbo to credit it before uploading anything. If Turbo has not credited the top-up by then, the deploy stops without uploading and records the transfer in `.ario-deploy/`. Re-run once it confirms: the next run waits for that transfer instead of buying another. A transfer the payment service rejects is reported as such, and nothing is uploaded. # Package.json Scripts (/sdks/(clis)/ario-deploy/package-json-scripts) Add deployment scripts to your `package.json`: ```json { "scripts": { "build": "vite build", "deploy": "pnpm build && ario-deploy deploy --arns-name \", "deploy:staging": "pnpm build && ario-deploy deploy --arns-name \ --undername staging", "deploy:devnet": "pnpm build && ario-deploy deploy --arns-name \ --cluster devnet", "deploy:on-demand": "pnpm build && ario-deploy deploy --arns-name \ --sig-type solana --on-demand ario --max-token-amount 1.5" } } ``` These read the upload key from `DEPLOY_KEY` and the Solana ArNS authority key from `ARNS_KEY`. Deploy with: ```bash DEPLOY_KEY=$(base64 -i wallet.json) ARNS_KEY=\ pnpm run deploy ``` Or with on-demand payment in ARIO, which needs a Solana upload key (here the same key does both jobs): ```bash DEPLOY_KEY=\ ARNS_KEY=\ pnpm deploy:on-demand ``` # Security & Best Practices (/sdks/(clis)/ario-deploy/security-best-practices) - **Dedicated Wallet:** Always use a dedicated wallet for deployments to minimize security risks - **Wallet Encoding:** Arweave wallets must be base64 encoded to be used in the deployment script - **ArNS Name:** Required only when updating an ANT/ArNS target undername or root record - **Turbo Credits:** Ensure your wallet has sufficient Turbo Credits, or use on-demand payment for automatic funding - **On-Demand Limits:** Set reasonable `--max-token-amount` limits to prevent unexpected costs - **Secret Management:** Keep your `DEPLOY_KEY` secret secure and never commit it to your repository - **Wallet Location:** Never keep a wallet file inside the folder you deploy. `ario-deploy` refuses to upload one it recognizes (see [Files that are never uploaded](#files-that-are-never-uploaded)), but other tools that publish the folder do not - **Build Security:** Always check your build for exposed environmental secrets before deployment, as data on Arweave is permanent # Shared credits (/sdks/(clis)/ario-deploy/shared-credits) Credits another wallet has shared with your upload key ([Turbo credit sharing](https://docs.ardrive.io/docs/turbo/)) are used automatically: the credit check counts them, and every data item names the sharing wallets as payers, which is what the bundler needs to charge them. Your own balance covers whatever they do not. - `--paid-by \`: use only these wallets' shared credits (comma-separated), then your own balance. - `--ignore-approvals`: ignore shared credits; pay only from the upload key's own balance. - `--use-signer-balance-first`: spend the upload key's own balance before shared credits. # Troubleshooting (/sdks/(clis)/ario-deploy/troubleshooting) - **Error: "DEPLOY_KEY environment variable not set":** Verify your base64 encoded wallet is set as the `DEPLOY_KEY` environment variable - **Error: "deploy-folder does not exist":** Check that your build folder exists and the path is correct - **Error: "deploy-file does not exist":** Check that your build file exists and the path is correct - **Error: "ArNS name does not exist":** Verify the ArNS name is correct and exists in the specified network - **Upload timeouts:** Files have a timeout for upload. Large files may fail and require optimization - **"402 Payment Required" (or "Turbo refused the upload as unpaid"):** The upload service will not take the files for free and no credits cover them. Free uploads are up to 105 KiB per file and 10 MiB over the lifetime of a wallet and of an IP range, so a wallet with allowance left can still be refused when its IP range has used up its own. Add Turbo credits at https://turbo.ardrive.io, re-run with `--on-demand` and `--max-token-amount`, or have credits shared to the wallet. Do not use `--dev` to get around it: a sandbox upload is not permanent. Files that uploaded before the failure are cached, so a re-run does not pay for them again - **Insufficient Turbo Credits:** Use `--on-demand` with `--max-token-amount` to automatically fund uploads when balance is low - **On-demand payment fails:** Ensure the upload wallet holds the token, and that the token matches the key: `ario`, `solana` or `solana-usdc` with `--sig-type solana`; `base-eth` or `base-usdc` with an Ethereum or Polygon key - **"Insufficient Turbo credits" on the sandbox with valid sandbox credits:** Use `--dev`, or pass `--payment-url https://payment.services.ar-io.dev` with a custom `--uploader`, so the balance is read from the sandbox - **Credits shared with you are not used:** They are used automatically unless `--ignore-approvals` is set; with `--paid-by`, only the listed wallets count - **Deep links 404 but the homepage loads:** The manifest has no `fallback`. Emit a `404.html` or pass `--fallback-file index.html` — see [Single-page apps](#single-page-apps) - **Deep links still 404 right after a redeploy:** Gateways cache the previous manifest's 404s for around a minute. Retry with a cache-busting query string before assuming the deploy failed - **Error: "Fallback file not found in folder":** `--fallback-file` takes a path relative to the deploy folder, e.g. `index.html`, not `./dist/index.html` # Networks (/sdks/ar-io-sdk/(advanced)/advanced-networks) | Network | RPC | Programs | |---|---|---| | Mainnet | `https://api.mainnet-beta.solana.com` (mainnet-beta, default) | Not yet deployed — placeholder IDs in `src/solana/constants.ts` | | Devnet | `https://api.devnet.solana.com` | See `src/solana/constants.ts` for current devnet program IDs | | Localnet | Surfpool — `https://github.com/solana-foundation/surfpool` | Localnet harness in `solana-ar-io` monorepo | The migration tooling (snapshot exporter, batch importer, claim app) lives in the [`solana-ar-io`](https://github.com/ar-io/solana-ar-io) monorepo until cutover. # Automatic Retries (/sdks/ar-io-sdk/(advanced)/automatic-retries) All RPC **read** calls (account fetches, `getProgramAccounts`, etc.) automatically retry on transient transport errors with exponential back-off. Writes are **not** retried (to avoid double-sends). Retried errors: HTTP 429/5xx, `fetch failed`, `ECONNRESET`, `ETIMEDOUT`, `AbortError` / timeouts. Non-retryable errors (account not found, invalid params, deserialization) throw immediately. Defaults: **6 attempts**, 500 ms base delay, 5 s max delay. Override per-call with the exported `withRetry` helper: ```ts const result = await withRetry(() => rpc.getAccountInfo(addr).send(), { maxAttempts: 3, baseDelayMs: 1000, }); ``` # Circuit Breaker (/sdks/ar-io-sdk/(advanced)/circuit-breaker) The SDK ships an [opossum]-backed circuit breaker that wraps the RPC transport. When the primary endpoint starts failing (429 rate-limits, 5xx errors, network timeouts) the circuit opens and subsequent calls route transparently to a fallback RPC until the primary recovers. ```ts const rpc = createCircuitBreakerRpc({ primaryUrl: 'https://my-premium-rpc.example.com', fallbackUrl: 'https://api.mainnet-beta.solana.com', }); const ario = ARIO.init({ rpc }); ``` Use `defaultFallbackUrl()` to auto-pick mainnet or devnet based on the primary URL: ```ts import { createCircuitBreakerRpc, defaultFallbackUrl, } from '@ar.io/sdk'; const primaryUrl = 'https://my-premium-rpc.example.com'; const rpc = createCircuitBreakerRpc({ primaryUrl, fallbackUrl: defaultFallbackUrl(primaryUrl), // → mainnet public RPC }); ``` Tuning knobs (all optional): | Option | Default | Description | |---|---|---| | `timeout` | `10000` | ms before a single request is timed out (`false` to disable) | | `errorThresholdPercentage` | `50` | error % at which to open the circuit | | `resetTimeout` | `30000` | ms to wait before probing the primary again (half-open) | | `volumeThreshold` | `5` | minimum requests in the rolling window before the circuit can trip | # Generated instruction builders (/sdks/ar-io-sdk/(advanced)/generated-instruction-builders) For custom transaction building, import Codama-generated typed clients from [`@ar.io/solana-contracts`](https://www.npmjs.com/package/@ar.io/solana-contracts): ```ts import { getBuyNameInstructionAsync, ARIO_ARNS_PROGRAM_ADDRESS, } from '@ar.io/solana-contracts/arns'; ``` # RPC Configuration (/sdks/ar-io-sdk/(advanced)/rpc-configuration) The SDK accepts any `@solana/kit` RPC client. For read-only usage, only `rpc` is required. Write operations additionally need `rpcSubscriptions` (WebSocket) for transaction confirmation and a `signer`. #### Basic (read-only) ```ts const rpc = createSolanaRpc('https://api.mainnet-beta.solana.com'); const ario = ARIO.init({ rpc }); ``` #### With writes (signer + WebSocket subscriptions) ```ts import { createSolanaRpc, createSolanaRpcSubscriptions, createKeyPairSignerFromBytes, } from '@solana/kit'; const rpc = createSolanaRpc('https://api.mainnet-beta.solana.com'); const rpcSubscriptions = createSolanaRpcSubscriptions( 'wss://api.mainnet-beta.solana.com', ); const signer = await createKeyPairSignerFromBytes(/* ... */); const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); ``` > **Note:** `rpcSubscriptions` opens a WebSocket connection and is only > needed for writes. If your RPC provider doesn't expose a WebSocket > endpoint, omit it and use the SDK in read-only mode. # ARIO Integrations (/sdks/ar-io-sdk/(ant-contracts)/ario-integrations) `releaseName`, `reassignName`, `approvePrimaryNameRequest`, and `removePrimaryNames` were AO-only orchestration helpers and have been removed. Their on-chain equivalents on Solana live on the `ario-arns` program and are exposed through the `ARIO` write client (`upgradeRecord`, `setPrimaryName`, etc.) or — for permissionless maintenance — through `SolanaARIOWriteable`'s prune helpers. # Balances (/sdks/ar-io-sdk/(ant-contracts)/balances) #### getBalances() Returns all token balances for the ANT. ```typescript const balances = await ant.getBalances(); ``` **Output:** ```json { "ccp3blG__gKUvG3hsGC2u06aDmqv4CuhuDJGOIg0jw4": 1, "aGzM_yjralacHIUo8_nQXMbh9l1cy0aksiL_x9M359f": 0 } ``` #### getBalance() Returns the balance of a specific address. ```typescript const balance = await ant.getBalance({ address: "ccp3blG__gKUvG3hsGC2u06aDmqv4CuhuDJGOIg0jw4", }); ``` **Output:** ```json 1 ``` # Controllers (/sdks/ar-io-sdk/(ant-contracts)/controllers) #### addController() Adds a new controller to the list of approved controllers on the ANT. Controllers can set records and change the ticker and name of the ANT process. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.addController( { controller: "aGzM_yjralacHIUo8_nQXMbh9l1cy0aksiL_x9M359f" }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### removeController() Removes a controller from the list of approved controllers on the ANT. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.removeController( { controller: "aGzM_yjralacHIUo8_nQXMbh9l1cy0aksiL_x9M359f" }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` # Initialize (/sdks/ar-io-sdk/(ant-contracts)/initialize) #### init() Factory that creates a read-only or writeable ANT client. Providing `signer` and `rpcSubscriptions` enables write methods (`setRecord`, `transfer`, `addController`, etc.). ```typescript import { createSolanaRpc, createSolanaRpcSubscriptions, } from '@solana/kit'; const rpc = createSolanaRpc('https://api.mainnet-beta.solana.com'); // Read-only const ant = await ANT.init({ processId: '\', rpc, }); // Read + write const antWrite = await ANT.init({ processId: '\', rpc, rpcSubscriptions: createSolanaRpcSubscriptions( 'wss://api.mainnet-beta.solana.com', ), signer, }); ``` # Metadata (/sdks/ar-io-sdk/(ant-contracts)/metadata) #### setName() Sets the name of the ANT process. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.setName( { name: "My ANT" }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### setTicker() Sets the ticker of the ANT process. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.setTicker( { ticker: "ANT-NEW-TICKER" }, // optional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### setDescription() Sets the description of the ANT process. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.setDescription( { description: "A friendly description of this ANT" }, // optional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### setKeywords() Sets the keywords of the ANT process. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.setKeywords( { keywords: ["Game", "FPS", "AO"] }, // optional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### getLogo() Returns the TX ID of the logo set for the ANT. ```typescript const logoTxId = await ant.getLogo(); ``` #### setLogo() Sets the Logo of the ANT - logo should be an Arweave transaction ID. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.setLogo( { txId: "U7RXcpaVShG4u9nIcPVmm2FJSM5Gru9gQCIiRaIPV7f" }, // optional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` # Records (/sdks/ar-io-sdk/(ant-contracts)/records) #### setBaseNameRecord() Adds or updates the base name record for the ANT. This is the top level name of the ANT (e.g. ardrive.ar.io). Supports undername ownership delegation and metadata. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript // get the ant for the base name const arnsRecord = await ario.getArNSRecord({ name: "ardrive" }); const ant = await ANT.init({ processId: arnsName.processId }); // Basic usage const { id: txId } = await ant.setBaseNameRecord({ transactionId: "432l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", ttlSeconds: 3600, }); // With ownership delegation and metadata const { id: txId } = await ant.setBaseNameRecord({ transactionId: "432l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", ttlSeconds: 3600, owner: "user-wallet-address-123...", // delegate ownership to another address displayName: "ArDrive", // display name logo: "logo-tx-id-123...", // logo transaction ID description: "Decentralized storage application", keywords: ["storage", "decentralized", "web3"], }); // ardrive.ar.io will now resolve to the provided transaction id and include metadata ``` #### setUndernameRecord() Adds or updates an undername record for the ANT. An undername is appended to the base name of the ANT (e.g. dapp_ardrive.ar.io). Supports undername ownership delegation and metadata. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ > Records, or `undernames` are configured with the `transactionId` - the arweave transaction id the record resolves - and `ttlSeconds`, the Time To Live in the cache of client applications. ```typescript const arnsRecord = await ario.getArNSRecord({ name: "ardrive" }); const ant = await ANT.init({ processId: arnsName.processId }); // Basic usage const { id: txId } = await ant.setUndernameRecord( { undername: "dapp", transactionId: "432l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", ttlSeconds: 900, }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); // With ownership delegation and metadata const { id: txId } = await ant.setUndernameRecord( { undername: "alice", transactionId: "432l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", ttlSeconds: 900, owner: "alice-wallet-address-123...", // delegate ownership to Alice displayName: "Alice's Site", // display name logo: "avatar-tx-id-123...", // avatar/logo transaction ID description: "Personal portfolio and blog", keywords: ["portfolio", "personal", "blog"], }, { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); // dapp_ardrive.ar.io will now resolve to the provided transaction id // alice_ardrive.ar.io will be owned by Alice and include metadata ``` #### removeUndernameRecord() Removes an undername record from the ANT process. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.removeUndernameRecord( { undername: "dapp" }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); // dapp_ardrive.ar.io will no longer resolve to the provided transaction id ``` #### setRecord() Deprecated: Use `setBaseNameRecord` or `setUndernameRecord` instead. Adds or updates a record for the ANT process. The `undername` parameter is used to specify the record name. Use `@` for the base name record. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ > Records, or `undernames` are configured with the `transactionId` - the arweave transaction id the record resolves - and `ttlSeconds`, the Time To Live in the cache of client applications. ```typescript const { id: txId } = await ant.setRecord( { undername: '@', transactionId: '432l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM' ttlSeconds: 3600 }, // optional additional tags { tags: [{ name: 'App-Name', value: 'My-Awesome-App' }] }, ); ``` #### removeRecord() Deprecated: Use `removeUndernameRecord` instead. Removes a record from the ANT process. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const arnsRecord = await ario.getArNSRecord({ name: "ardrive" }); const ant = await ANT.init({ processId: arnsName.processId }); const { id: txId } = await ant.removeRecord( { undername: "dapp" }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); // dapp_ardrive.ar.io will no longer resolve to the provided transaction id ``` # Spawn (/sdks/ar-io-sdk/(ant-contracts)/spawn) #### ANT.spawn() Static factory that mints a new MPL Core asset and initializes the `ario-ant` PDAs in a single transaction. Returns `{ processId, mint, signature }`. ```typescript const { processId, signature } = await ANT.spawn({ rpc, rpcSubscriptions, signer, state: { name: 'My ANT', ticker: 'MYANT', description: 'My ANT token', uri: 'ar://\', }, }); ``` **CLI Usage:** ```bash ar.io spawn-ant \ --wallet-file wallet.json \ --name "My ANT" \ --ticker "MYANT" \ --metadata-uri "ar://\" ``` **Parameters:** - `state.name: string` — display name of the ANT - `state.ticker?: string` — ticker symbol - `state.description?: string` — short description - `state.uri: string` — `ar://` URI of the Metaplex Core asset's JSON metadata. Build via `buildAntMetadata` from `@ar.io/sdk` and upload to Arweave (e.g. via `@ardrive/turbo-sdk`). - `state.keywords?: string[]` - `state.logo?: string` — Arweave TX ID of the logo - `state.transactionId?: string` — initial `@` record target **Returns:** ```ts { processId: string; // the MPL Core asset mint pubkey mint: Address; signature: string; // the Solana tx signature } ``` # State (/sdks/ar-io-sdk/(ant-contracts)/state) #### getInfo() Retrieves the information of the ANT process. ```typescript const info = await ant.getInfo(); ``` **Output:** ```json { "Name": "ArDrive", "Owner": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", "Ticker": "ANT-ARDRIVE", "Total-Supply": "1", "Description": "This is the ANT for the ArDrive decentralized web app.", "Keywords": ["File-sharing", "Publishing", "dApp"], "Logo": "Sie_26dvgyok0PZD_-iQAFOhOd5YxDTkczOLoqTTL_A", "Denomination": "0", "Handlers": [ "balance", "balances", "totalSupply", "info", "controllers", "record", "records", "state", "transfer", "addController", "removeController", "setRecord", "removeRecord", "setName", "setTicker", "setDescription", "setKeywords", "setLogo", "initializeState", "releaseName", "reassignName", "approvePrimaryName", "removePrimaryNames", "transferRecordOwnership", "_eval", "_default" ] } ``` #### getHandlers() Retrieves the handlers supported on the ANT ```typescript const handlers = await ant.getHandlers(); ``` **Output:** ```json [ "balance", "balances", "totalSupply", "info", "controllers", "record", "records", "state", "transfer", "addController", "removeController", "setRecord", "removeRecord", "setName", "setTicker", "setDescription", "setKeywords", "setLogo", "initializeState", "releaseName", "reassignName", "approvePrimaryName", "removePrimaryNames", "transferRecordOwnership", "_eval", "_default" ] ``` #### getState() Retrieves the state of the ANT process. ```typescript const state = await ant.getState(); ``` **Output:** ```json { "Name": "ar.io Foundation", "Ticker": "ANT-AR-IO", "Description": "A friendly description for this ANT.", "Keywords": ["keyword1", "keyword2", "keyword3"], "Denomination": 0, "Owner": "98O1_xqDLrBKRfQPWjF5p7xZ4Jx6GM8P5PeJn26xwUY", "Controllers": [], "Records": { "v1-0-0_whitepaper": { "transactionId": "lNjWn3LpyhKC95Kqe-x8X2qgju0j98MhucdDKK85vc4", "ttlSeconds": 900, "targetProtocol": 0 }, "@": { "transactionId": "2rMLb2uHAyEt7jSu6bXtKx8e-jOfIf7E-DOgQnm8EtU", "ttlSeconds": 3600, "targetProtocol": 0 }, "alice": { "transactionId": "kMk95k_3R8x_7d3wB9tEOiL5v6n8QhR_VnFCh3aeE3f", "ttlSeconds": 900, "targetProtocol": 0, "owner": "alice-wallet-address-123...", "displayName": "Alice's Portfolio", "logo": "avatar-tx-id-456...", "description": "Personal portfolio and blog", "keywords": ["portfolio", "personal", "blog"] }, "whitepaper": { "transactionId": "lNjWn3LpyhKC95Kqe-x8X2qgju0j98MhucdDKK85vc4", "ttlSeconds": 900, "targetProtocol": 0 } }, "Balances": { "98O1_xqDLrBKRfQPWjF5p7xZ4Jx6GM8P5PeJn26xwUY": 1 }, "Logo": "Sie_26dvgyok0PZD_-iQAFOhOd5YxDTkczOLoqTTL_A", "TotalSupply": 1, "Initialized": true } ``` #### getOwner() Returns the owner of the configured ANT process. ```typescript const owner = await ant.getOwner(); ``` **Output:** ```json "ccp3blG__gKUvG3hsGC2u06aDmqv4CuhuDJGOIg0jw4" ``` #### getName() Returns the name of the ANT (not the same as ArNS name). ```typescript const name = await ant.getName(); ``` **Output:** ```json "ArDrive" ``` #### getTicker() Returns the ticker symbol of the ANT. ```typescript const ticker = await ant.getTicker(); ``` **Output:** ```json "ANT-ARDRIVE" ``` #### getControllers() Returns the controllers of the configured ANT process. ```typescript const controllers = await ant.getControllers(); ``` **Output:** ```json ["ccp3blG__gKUvG3hsGC2u06aDmqv4CuhuDJGOIg0jw4"] ``` #### getRecords() Returns all records on the configured ANT process, including the required `@` record that resolve connected ArNS names. ```typescript const records = await ant.getRecords(); ``` **Output:** ```json { "@": { "transactionId": "UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk", "ttlSeconds": 3600, "targetProtocol": 0, "index": 0 }, "alice": { "transactionId": "kMk95k_3R8x_7d3wB9tEOiL5v6n8QhR_VnFCh3aeE3f", "ttlSeconds": 900, "targetProtocol": 0, "owner": "alice-wallet-address-123...", "displayName": "Alice's Portfolio", "logo": "avatar-tx-id-456...", "description": "Personal portfolio and blog", "keywords": ["portfolio", "personal", "blog"], "index": 1 }, "zed": { "transactionId": "-k7t8xMoB8hW482609Z9F4bTFMC3MnuW8bTvTyT8pFI", "ttlSeconds": 900, "targetProtocol": 0, "index": 2 }, "ardrive": { "transactionId": "-cucucachoodwedwedoiwepodiwpodiwpoidpwoiedp", "ttlSeconds": 900, "targetProtocol": 0, "index": 3 } } ``` #### getRecord() Returns a specific record by its undername. ```typescript const record = await ant.getRecord({ undername: "dapp" }); ``` **Output:** ```json { "transactionId": "432l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "ttlSeconds": 900, "targetProtocol": 0, "owner": "alice-wallet-address-123...", "displayName": "Alice's Site", "logo": "avatar-tx-id-456...", "description": "Personal portfolio and blog", "keywords": ["portfolio", "personal", "blog"] } ``` # Static Methods (/sdks/ar-io-sdk/(ant-contracts)/static-methods) `ANT.fork()` and the static `ANT.upgrade()` were AO-only (process forking + name reassignment). On Solana, schema migration is a per-asset CPI exposed as the instance method `ant.upgrade()` documented above; new ANTs are created with `ANT.spawn()`. # Transfer (/sdks/ar-io-sdk/(ant-contracts)/transfer) #### transfer() Transfers ownership of the ANT to a new target address. Target must be a Solana address. ANT transfers are standard Metaplex Core NFT transfers. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.transfer( { target: "aGzM_yjralacHIUo8_nQXMbh9l1cy0aksiL_x9M359f" }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` # Undername Ownership (/sdks/ar-io-sdk/(ant-contracts)/undername-ownership) NTs support ownership of undernames: 1. **ANT Owner** - Has full control over the ANT and all records 2. **Controllers** - Can manage records but cannot transfer ANT ownership 3. **Record Owners** - Can only update their specific delegated records When a record owner updates their own record, they **MUST** include their own address in the `owner` field. If the `owner` field is omitted or set to a different address, the record ownership will be transferred or renounced. #### transferRecord() Transfers ownership of a specific record (undername) to another address. This enables delegation of control for individual records within an ANT while maintaining the ANT owner's ultimate authority. The current record owner or ANT owner/controllers can transfer ownership. _Note: Requires `signer` to be provided on `ANT.init` to sign the transaction._ ```typescript const { id: txId } = await ant.transferRecord({ undername: "alice", // the subdomain/record to transfer recipient: "new-owner-address-123...", // address of the new owner }); // alice_ardrive.ar.io is now owned by the new owner address // The new owner can update the record but not other records in the ANT ``` **CLI Usage:** ```bash ar.io transfer-record \ --process-id "ANT_PROCESS_ID" \ --undername "alice" \ --recipient "new-owner-address-123..." \ --wallet-file "path/to/wallet.json" ``` #### Record Owner Workflow Examples **Checking Record Ownership:** ```typescript const record = await ant.getRecord({ undername: "alice" }); console.log(`Record owner: ${record.owner}`); console.log(`Transaction ID: ${record.transactionId}`); ``` **Record Owner Updating Their Own Record:** ```typescript // Alice (record owner) updating her own record const aliceAnt = await ANT.init({ processId: 'ANT_MINT_PUBKEY', rpc, rpcSubscriptions, signer: aliceSigner, // Alice's @solana/kit signer }); // ✅ CORRECT: Alice includes her own address as owner const { id: txId } = await aliceAnt.setUndernameRecord({ undername: "alice", transactionId: "new-content-tx-id-456...", ttlSeconds: 1800, owner: "alice-wallet-address-123...", // MUST be Alice's own address displayName: "Alice Updated Portfolio", description: "Updated personal portfolio and blog", }); // ❌ WRONG: Omitting owner field will renounce ownership const badUpdate = await aliceAnt.setUndernameRecord({ undername: "alice", transactionId: "new-content-tx-id-456...", ttlSeconds: 1800, // Missing owner field - this will renounce ownership! }); // ❌ WRONG: Setting different owner will transfer ownership const badTransfer = await aliceAnt.setUndernameRecord({ undername: "alice", transactionId: "new-content-tx-id-456...", ttlSeconds: 1800, owner: "someone-else-address-789...", // This transfers ownership to someone else! }); ``` **What Happens When Record Ownership is Renounced:** If a record owner updates their record without including the `owner` field, the record becomes owned by the ANT owner/controllers again: ```typescript // Before: alice record is owned by alice-wallet-address-123... const recordBefore = await ant.getRecord({ undername: "alice" }); console.log(recordBefore.owner); // "alice-wallet-address-123..." // Alice updates without owner field await aliceAnt.setUndernameRecord({ undername: "alice", transactionId: "new-tx-id...", ttlSeconds: 900, // No owner field = renounces ownership }); // After: record ownership reverts to ANT owner const recordAfter = await ant.getRecord({ undername: "alice" }); console.log(recordAfter.owner); // undefined (controlled by ANT owner again) ``` # Upgrade (/sdks/ar-io-sdk/(ant-contracts)/upgrade) #### upgrade() Migrates this ANT's on-chain state to the latest schema version (per- ANT data migration on Solana — no process forking, no name reassignment). Returns `{ id, needsMigration }`. ```typescript const result = await ant.upgrade(); if (result.needsMigration) { console.log(`Migrated: ${result.id}`); } ``` # Versions (/sdks/ar-io-sdk/(ant-contracts)/versions) #### getModuleId() Gets the module ID of the current ANT process by querying its spawn transaction tags. Results are cached after the first successful fetch. ```typescript const moduleId = await ant.getModuleId(); console.log(`ANT was spawned with module: ${moduleId}`); // With custom GraphQL URL and retries const moduleId = await ant.getModuleId({ graphqlUrl: "https://turbo-gateway.com/graphql", retries: 5, }); ``` **Output:** ```json "FKtQtOOtlcWCW2pXrwWFiCSlnuewMZOHCzhulVkyqBE" ``` #### getVersion() Gets the version string of the current ANT by matching its module ID with versions from the ANT registry. ```typescript const version = await ant.getVersion(); console.log(`ANT is running version: ${version}`); // With custom ANT registry const version = await ant.getVersion({ antRegistryId: "custom-ant-registry-id", }); ``` **Output:** ```json "23" ``` #### isLatestVersion() Checks if the current ANT version is the latest according to the ANT registry. ```typescript const isLatest = await ant.isLatestVersion(); if (!isLatest) { console.log("ANT can be upgraded to the latest version"); } ``` **Output:** ```json true ``` # ar.io Name System (ArNS) (/sdks/ar-io-sdk/(ario-contract)/ar-io-name-system-arns) #### resolveArNSName() Resolves an ArNS name to the underlying data id stored on the names corresponding ANT id. ##### Resolving a base name ```typescript const ario = ARIO.init({ rpc }); const record = await ario.resolveArNSName({ name: "ardrive" }); ``` **Output:** ```json { "name": "ardrive", "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "txId": "kvhEUsIY5bXe0Wu2-YUFz20O078uYFzmQIO-7brv8qw", "type": "lease", "ttlSeconds": 3600, "undernameLimit": 100 } ``` ##### Resolving an undername ```typescript const ario = ARIO.init({ rpc }); const record = await ario.resolveArNSName({ name: "logo_ardrive" }); ``` **Output:** ```json { "name": "ardrive", "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "txId": "kvhEUsIY5bXe0Wu2-YUFz20O078uYFzmQIO-7brv8qw", "type": "lease", "ttlSeconds": 3600, "undernameLimit": 100 } ``` #### buyRecord() Purchases a new ArNS record with the specified name, type, processId, and duration. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ **Arguments:** - `name` - _required_: the name of the ArNS record to purchase - `type` - _required_: the type of ArNS record to purchase - `processId` - _optional_: the process id of an existing ANT process. If not provided, a new ANT process using the provided `signer` will be spawned, and the ArNS record will be assigned to that process. - `years` - _optional_: the duration of the ArNS record in years. If not provided and `type` is `lease`, the record will be leased for 1 year. If not provided and `type` is `permabuy`, the record will be permanently registered. - `referrer` - _optional_: track purchase referrals for analytics (e.g. `my-app.com`) - `antState` - _optional_ (**Solana atomic buy only**): initial ANT metadata + base `@` target to bake into the newly minted ANT in the same transaction, so the name resolves to your content immediately instead of the default AR.IO logo. Fields: `transactionId` (the `@` target — Arweave TX id, or IPFS CID when `targetProtocol` is `1`), `targetProtocol` (`0` = Arweave (default), `1` = IPFS), `ticker`, `logo` (43-char Arweave TX id), `description` (≤ 512 chars), `keywords` (≤ 16). **Ignored (with a warning) when `processId` is provided** — set metadata on an existing ANT via the ANT writeable instead. Note: TTL cannot be set at mint (`initialize` has no TTL field), so a non-default TTL still needs a post-buy `setBaseNameRecord`; and a very long `description`/`keywords` set may exceed the transaction size limit — keep those on the post-buy path. ```typescript // Atomic buy that also points the name at your content in one transaction: const record = await ario.buyRecord({ name: "ardrive", type: "lease", years: 1, antState: { transactionId: "432l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", // base @ target ticker: "ARDRIVE", description: "Permanent, decentralized data storage.", keywords: ["storage", "permaweb"], }, }); ``` ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const record = await ario.buyRecord( { name: "ardrive", type: "lease", years: 1, processId: "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", // optional: assign to existing ANT process referrer: "my-app.com", // optional: track purchase referrals for analytics }, { // optional tags tags: [{ name: "App-Name", value: "ArNS-App" }], onSigningProgress: (step, event) => { console.log(`Signing progress: ${step}`); if (step === "spawning-ant") { console.log("Spawning ant:", event); } if (step === "registering-ant") { console.log("Registering ant:", event); } if (step === "verifying-state") { console.log("Verifying state:", event); } if (step === "buying-name") { console.log("Buying name:", event); } }, }, ); ``` #### upgradeRecord() Upgrades an existing leased ArNS record to a permanent ownership. The record must be currently owned by the caller and be of type "lease". _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const record = await ario.upgradeRecord( { name: "ardrive", referrer: "my-app.com", // optional: track purchase referrals for analytics }, { // optional tags tags: [{ name: "App-Name", value: "ArNS-App" }], }, ); ``` #### getArNSRecord() Retrieves the record info of the specified ArNS name. ```typescript const ario = ARIO.init({ rpc }); const record = await ario.getArNSRecord({ name: "ardrive" }); ``` **Output:** ```json { "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "startTimestamp": 1720720819969, "endTimestamp": 1752256702026, "type": "lease", "undernameLimit": 100, "purchasePrice": 75541282285 } ``` #### getArNSRecords() Retrieves all registered ArNS records of the ARIO process, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last ArNS name from the previous request. ```typescript const ario = ARIO.init({ rpc }); // get the newest 100 names const records = await ario.getArNSRecords({ limit: 100, sortBy: "startTimestamp", sortOrder: "desc", }); ``` Available `sortBy` options are any of the keys on the record object, e.g. `name`, `processId`, `endTimestamp`, `startTimestamp`, `type`, `undernames`. **Output:** ```json { "items": [ { "name": "ao", "processId": "eNey-H9RB9uCdoJUvPULb35qhZVXZcEXv8xds4aHhkQ", "purchasePrice": 75541282285, "startTimestamp": 1720720621424, "endTimestamp": 1752256702026, "type": "permabuy", "undernameLimit": 10 }, { "name": "ardrive", "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "endTimestamp": 1720720819969, "startTimestamp": 1720720620813, "purchasePrice": 75541282285, "type": "lease", "undernameLimit": 100 }, { "name": "arweave", "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "endTimestamp": 1720720819969, "startTimestamp": 1720720620800, "purchasePrice": 75541282285, "type": "lease", "undernameLimit": 100 }, { "name": "ar-io", "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "endTimestamp": 1720720819969, "startTimestamp": 1720720619000, "purchasePrice": 75541282285, "type": "lease", "undernameLimit": 100 }, { "name": "fwd", "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "endTimestamp": 1720720819969, "startTimestamp": 1720720220811, "purchasePrice": 75541282285, "type": "lease", "undernameLimit": 100 } // ...95 other records ], "hasMore": true, "nextCursor": "fwdresearch", "totalItems": 21740, "sortBy": "startTimestamp", "sortOrder": "desc" } ``` #### getArNSRecordsForAddress() Retrieves all registered ArNS records of the specified address according to the `ANTRegistry` access control list, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last ArNS name from the previous request. ```typescript const ario = ARIO.init({ rpc }); const records = await ario.getArNSRecordsForAddress({ address: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", limit: 100, sortBy: "startTimestamp", sortOrder: "desc", }); ``` Available `sortBy` options are any of the keys on the record object, e.g. `name`, `processId`, `endTimestamp`, `startTimestamp`, `type`, `undernames`. **Output:** ```json { "limit": 1, "totalItems": 31, "hasMore": true, "nextCursor": "ardrive", "items": [ { "startTimestamp": 1740009600000, "name": "ardrive", "endTimestamp": 1777328018367, "type": "permabuy", "purchasePrice": 0, "undernameLimit": 100, "processId": "hpF0HdijWlBLFePjWX6u_-Lg3Z2E_PrP_AoaXDVs0bA" } ], "sortOrder": "desc", "sortBy": "startTimestamp" } ``` #### increaseUndernameLimit() Increases the undername support of a domain up to a maximum of 10k. Domains, by default, support up to 10 undernames. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.increaseUndernameLimit( { name: "ar-io", qty: 420, referrer: "my-app.com", // optional: track purchase referrals for analytics }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### extendLease() Extends the lease of a registered ArNS domain, with an extension of 1-5 years depending on grace period status. Permanently registered domains cannot be extended. ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.extendLease( { name: "ar-io", years: 1, referrer: "my-app.com", // optional: track purchase referrals for analytics }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### getTokenCost() Calculates the price in mARIO to perform the interaction in question, eg a 'Buy-Name' interaction, where args are the specific params for that interaction. ```typescript const price = await ario .getTokenCost({ intent: "Buy-Name", name: "ar-io", type: "permabuy", }) .then((p) => new mARIOToken(p).toARIO()); // convert to ARIO for readability ``` **Output:** ```json 1642.34 ``` #### getCostDetails() Calculates the expanded cost details for the interaction in question, e.g a 'Buy-Name' interaction, where args are the specific params for that interaction. The fromAddress is the address that would be charged for the interaction, and fundFrom is where the funds would be taken from, either `balance`, `stakes`, or `any`. On Solana, the result also includes a `gasEstimate` — the total SOL (in lamports) the wallet needs to execute the intent: transaction fees (quoted from recent on-chain prioritization fees) plus rent-exempt deposits for the accounts the flow creates. For `Buy-Name` that covers both transactions (ANT spawn + buy) and the rent for the spawned asset/PDAs and the ArNS record; first-time buyers with no ACL accounts yet are quoted the ACL bootstrap rent as well (pass `fromAddress` so that check can be made). The fee side is a conservative upper bound: the write path tightens the compute-unit limit from a pre-send simulation, so the landed fee is usually lower. **Gateway operator discount (Solana).** ArNS purchases (`Buy-Name`, `Extend-Lease`, `Increase-Undername-Limit`, `Upgrade-Name`) are 20% cheaper when the payer is a gateway's operator — or, once the gateway is migrated, its operations address — and that gateway is joined, has run for at least 180 days, and passes at least 90% of epochs. Primary-name fees are never discounted. The quote and the purchase methods (`buyRecord`, `buyReturnedName`, `extendLease`, `increaseUndernameLimit`, `upgradeRecord`) apply the same rule: by default they use the payer's own gateway and add the discount only when it qualifies (the program rejects a purchase whose discount gateway does not qualify). An operations address names the gateway with `discountGatewayAddress` (its operator address); naming a gateway that does not qualify is an error. CLI: `--discount-gateway-address`. ```typescript const costDetails = await ario.getCostDetails({ intent: "Buy-Name", fromAddress: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", fundFrom: "stakes", name: "ar-io", type: "permabuy", }); ``` **Output:** ```json { "tokenCost": 1907401818, "discounts": [ { "name": "Gateway Operator", "discountTotal": 476850455, "multiplier": 0.8 } ], "gasEstimate": { "totalLamports": 14679680, "feeLamports": 23000, "baseFeeLamports": 15000, "priorityFeeLamports": 8000, "rentLamports": 14656680, "rentReclaimedLamports": 0, "priorityFeeMicroLamports": 10000, "computeUnitLimit": 400000, "signatureCount": 3, "transactionCount": 2 } } ``` #### getDemandFactor() Retrieves the current demand factor of the network. The demand factor is a multiplier applied to the cost of ArNS interactions based on the current network demand. ```typescript const ario = ARIO.init({ rpc }); const demandFactor = await ario.getDemandFactor(); ``` **Output:** ```json 1.05256 ``` #### getArNSReturnedNames() Retrieves all active returned names of the ARIO process, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last returned name from the previous request. ```typescript const ario = ARIO.init({ rpc }); const returnedNames = await ario.getArNSReturnedNames({ limit: 100, sortBy: "endTimestamp", sortOrder: "asc", // return the returned names ending soonest first }); ``` **Output:** ```json { "items": [ { "name": "permalink", "startTimestamp": 1729775641349, "endTimestamp": 1730985241349, "initiator": "GaQrvEMKBpkjofgnBi_B3IgIDmY_XYelVLB6GcRGrHc", "premiumMultiplier": 50 } ], "hasMore": false, "totalItems": 1, "limit": 100, "sortBy": "endTimestamp", "sortOrder": "asc" } ``` #### getArNSReturnedName() Retrieves the returned name data for the specified returned name. ```typescript const ario = ARIO.init({ rpc }); const returnedName = await ario.getArNSReturnedName({ name: "permalink" }); ``` **Output:** ```json { "name": "permalink", "startTimestamp": 1729775641349, "endTimestamp": 1730985241349, "initiator": "GaQrvEMKBpkjofgnBi_B3IgIDmY_XYelVLB6GcRGrHc", "premiumMultiplier": 50 } ``` # Configuration (/sdks/ar-io-sdk/(ario-contract)/configuration) `ARIO.init` accepts a `@solana/kit` RPC client plus optional program ID overrides for non-mainnet clusters. See [Networks](#networks) above for the full shape. # Epochs (/sdks/ar-io-sdk/(ario-contract)/epochs) #### getCurrentEpoch() Returns the current epoch data. ```typescript const ario = ARIO.init({ rpc }); const epoch = await ario.getCurrentEpoch(); ``` **Output:** ```json { "epochIndex": 0, "startHeight": 0, "startTimestamp": 1720720621424, "endTimestamp": 1752256702026, "distributionTimestamp": 1752256702026, "observations": { "failureSummaries": { "-Tk2DDk8k4zkwtppp_XFKKI5oUgh6IEHygAoN7mD-w8": [ "Ie2wEEUDKoU26c7IuckHNn3vMFdNQnMvfPBrFzAb3NA" ] }, "reports": { "IPdwa3Mb_9pDD8c2IaJx6aad51Ss-_TfStVwBuhtXMs": "B6UUjKWjjEWDBvDSMXWNmymfwvgR9EN27z5FTkEVlX4" } }, "prescribedNames": ["ardrive", "ar-io", "arweave", "fwd", "ao"], "prescribedObservers": [ { "gatewayAddress": "2Fk8lCmDegPg6jjprl57-UCpKmNgYiKwyhkU4vMNDnE", "observerAddress": "2Fk8lCmDegPg6jjprl57-UCpKmNgYiKwyhkU4vMNDnE", "stake": 10000000000, "startTimestamp": 1720720621424, "stakeWeight": 1, "tenureWeight": 0.4494598765432099, "gatewayPerformanceRatio": 1, "observerPerformanceRatio": 1, "gatewayRewardRatioWeight": 1, "observerRewardRatioWeight": 1, "compositeWeight": 0.4494598765432099, "normalizedCompositeWeight": 0.002057032496835938 } ], "distributions": { "totalEligibleGateways": 1, "totalEligibleRewards": 100000000, "totalEligibleObserverReward": 100000000, "totalEligibleGatewayReward": 100000000 }, "arnsStats": { "totalReturnedNames": 0, "totalActiveNames": 0, "totalGracePeriodNames": 0, "totalReservedNames": 0 } } ``` #### getEpoch() Returns the epoch data for the specified block height. If no epoch index is provided, the current epoch is used. ```typescript const ario = ARIO.init({ rpc }); const epoch = await ario.getEpoch({ epochIndex: 0 }); ``` **Output:** ```json { "epochIndex": 0, "startHeight": 0, "startTimestamp": 1720720620813, "endTimestamp": 1752256702026, "distributionTimestamp": 1752256702026, "observations": { "failureSummaries": { "-Tk2DDk8k4zkwtppp_XFKKI5oUgh6IEHygAoN7mD-w8": [ "Ie2wEEUDKoU26c7IuckHNn3vMFdNQnMvfPBrFzAb3NA" ] }, "reports": { "IPdwa3Mb_9pDD8c2IaJx6aad51Ss-_TfStVwBuhtXMs": "B6UUjKWjjEWDBvDSMXWNmymfwvgR9EN27z5FTkEVlX4" } }, "prescribedNames": ["ardrive", "ar-io", "arweave", "fwd", "ao"], "prescribedObservers": [ { "gatewayAddress": "2Fk8lCmDegPg6jjprl57-UCpKmNgYiKwyhkU4vMNDnE", "observerAddress": "2Fk8lCmDegPg6jjprl57-UCpKmNgYiKwyhkU4vMNDnE", "stake": 10000000000, "startTimestamp": 1720720620813, "stakeWeight": 1, "tenureWeight": 0.4494598765432099, "gatewayPerformanceRatio": 1, "observerPerformanceRatio": 1, "gatewayRewardRatioWeight": 1, "observerRewardRatioWeight": 1, "compositeWeight": 0.4494598765432099, "normalizedCompositeWeight": 0.002057032496835938 } ], "distributions": { "totalEligibleGateways": 1, "totalEligibleRewards": 100000000, "totalEligibleObserverReward": 100000000, "totalEligibleGatewayReward": 100000000 }, "arnsStats": { "totalReturnedNames": 0, "totalActiveNames": 0, "totalGracePeriodNames": 0, "totalReservedNames": 0 } } ``` #### getEligibleEpochRewards() Returns the eligible epoch rewards for the specified block height. If no epoch index is provided, the current epoch is used. ```typescript const ario = ARIO.init({ rpc }); const rewards = await ario.getEligibleEpochRewards({ epochIndex: 0 }); ``` **Output:** ```json { "sortOrder": "desc", "hasMore": true, "totalItems": 37, "limit": 1, "sortBy": "cursorId", "items": [ { "cursorId": "xN_aVln30LmoCffwmk5_kRkcyQZyZWy1o_TNtM_CTm0_xN_aVln30LmoCffwmk5_kRkcyQZyZWy1o_TNtM_CTm0", "recipient": "xN_aVln30LmoCffwmk5_kRkcyQZyZWy1o_TNtM_CTm0", "gatewayAddress": "xN_aVln30LmoCffwmk5_kRkcyQZyZWy1o_TNtM_CTm0", "eligibleReward": 2627618704, "type": "operatorReward" } ], "nextCursor": "xN_aVln30LmoCffwmk5_kRkcyQZyZWy1o_TNtM_CTm0_xN_aVln30LmoCffwmk5_kRkcyQZyZWy1o_TNtM_CTm0" } ``` #### getObservations() Returns the epoch-indexed observation list. If no epoch index is provided, the current epoch is used. ```typescript const ario = ARIO.init({ rpc }); const observations = await ario.getObservations(); ``` **Output:** ```json { "failureSummaries": { "-Tk2DDk8k4zkwtppp_XFKKI5oUgh6IEHygAoN7mD-w8": [ "Ie2wEEUDKoU26c7IuckHNn3vMFdNQnMvfPBrFzAb3NA" ] }, "reports": { "IPdwa3Mb_9pDD8c2IaJx6aad51Ss-_TfStVwBuhtXMs": "B6UUjKWjjEWDBvDSMXWNmymfwvgR9EN27z5FTkEVlX4", "Ie2wEEUDKoU26c7IuckHNn3vMFdNQnMvfPBrFzAb3NA": "7tKsiQ2fxv0D8ZVN_QEv29fZ8hwFIgHoEDrpeEG0DIs", "osZP4D9cqeDvbVFBaEfjIxwc1QLIvRxUBRAxDIX9je8": "aatgznEvC_UPcxp1v0uw_RqydhIfKm4wtt1KCpONBB0", "qZ90I67XG68BYIAFVNfm9PUdM7v1XtFTn7u-EOZFAtk": "Bd8SmFK9-ktJRmwIungS8ur6JM-JtpxrvMtjt5JkB1M" } } ``` #### getDistributions() Returns the current rewards distribution information. If no epoch index is provided, the current epoch is used. ```typescript const ario = ARIO.init({ rpc }); const distributions = await ario.getDistributions({ epochIndex: 0 }); ``` **Output:** ```json { "totalEligibleGateways": 1, "totalEligibleRewards": 100000000, "totalEligibleObserverReward": 100000000, "totalEligibleGatewayReward": 100000000 } ``` #### saveObservations() Saves the observations of the current epoch. Requires `signer` to be provided on `ARIO.init` to sign the transaction. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.saveObservations( { reportTxId: "fDrr0_J4Iurt7caNST02cMotaz2FIbWQ4Kcj616RHl3", failedGateways: ["t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3"], }, { tags: [{ name: "App-Name", value: "My-Awesome-App" }], }, ); ``` #### getPrescribedObservers() Retrieves the prescribed observers of the ARIO process. To fetch prescribed observers for a previous epoch set the `epochIndex` to the desired epoch index. ```typescript const ario = ARIO.init({ rpc }); const observers = await ario.getPrescribedObservers({ epochIndex: 0 }); ``` **Output:** ```json [ { "gatewayAddress": "BpQlyhREz4lNGS-y3rSS1WxADfxPpAuing9Lgfdrj2U", "observerAddress": "2Fk8lCmDegPg6jjprl57-UCpKmNgYiKwyhkU4vMNDnE", "stake": 10000000000, "startTimestamp": 1720720620813, "stakeWeight": 1, "tenureWeight": 0.41453703703703704, "gatewayPerformanceRatio": 1, "observerPerformanceRatio": 1, "gatewayRewardRatioWeight": 1, "observerRewardRatioWeight": 1, "compositeWeight": 0.41453703703703704, "normalizedCompositeWeight": 0.0018972019546783507 } ] ``` #### crankEpochStep() High-level, permissionless epoch crank. Advances the epoch lifecycle by **one step per call** and returns the action it took — run it on a loop (this is what the standalone cranker and the observer-embedded cranker do). It owns the whole sequence so you don't orchestrate the individual instructions yourself: `create` → `tally` → `prescribe` → `distribute` → `close` — closing an epoch's observation PDAs first (`close_observation`) so `close_epoch` doesn't revert — plus an idle-tail of permissionless maintenance: `compound` delegate rewards, `update_demand_factor`, and `prune_returned_names`. The close path is non-wedging: a cleanup failure never blocks creation of the next epoch. ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); // one step const result = await ario.crankEpochStep(); // → { action, epochIndex?, txId?, progress? } // action ∈ create | tally | prescribe | distribute | close // | close_observation | compound | update_demand_factor // | prune_returned_names | idle // or drive it on an interval setInterval(async () => { const r = await ario.crankEpochStep(); if (r.action !== 'idle') console.log(r.action, r.epochIndex, r.txId); }, 60_000); ``` All options are optional: `batchSize`, `enableClose`, `epochRetention`, `enableCompound`, `compoundMinPendingRewards`, `enableDemandFactorRoll`, `enablePrune`, `pruneBatchSize`, `nameRegistryAccount`. # Gateways (/sdks/ar-io-sdk/(ario-contract)/gateways) #### getGateway() Retrieves a gateway's info by its staking wallet address. ```typescript const ario = ARIO.init({ rpc }); const gateway = await ario.getGateway({ address: "-7vXsQZQDk8TMDlpiSLy3CnLi5PDPlAaN2DaynORpck", }); ``` **Output:** ```json { "observerAddress": "IPdwa3Mb_9pDD8c2IaJx6aad51Ss-_TfStVwBuhtXMs", "operatorStake": 250000000000, "totalDelegatedStake": 0, "settings": { "allowDelegatedStaking": true, "allowedDelegates": [], "autoStake": false, "delegateRewardShareRatio": 10, "minDelegatedStake": 100000000, "fqdn": "ar-io.dev", "label": "ar.io Test", "note": "Test Gateway operated by PDS for the ar.io ecosystem.", "port": 443, "properties": "raJgvbFU-YAnku-WsupIdbTsqqGLQiYpGzoqk9SCVgY", "protocol": "https" }, "startTimestamp": 1720720620813, "endTimestamp": 0, "stats": { "passedConsecutiveEpochs": 30, "failedConsecutiveEpochs": 0, "totalEpochCount": 31, "passedEpochCount": 30, "failedEpochCount": 1, "observedEpochCount": 30, "prescribedEpochCount": 31 }, "status": "joined", "weights": { "stakeWeight": 5.02400000024, "tenureWeight": 0.19444444444444, "gatewayPerformanceRatio": 1, "observerPerformanceRatio": 1, "gatewayRewardRatioWeight": 1, "observerRewardRatioWeight": 1, "compositeWeight": 0.97688888893556, "normalizedCompositeWeight": 0.19247316211083 } } ``` #### getGateways() Retrieves registered gateways of the ARIO process, using pagination and sorting by the specified criteria. The `cursor` used for pagination is the last gateway address from the previous request. ```typescript const ario = ARIO.init({ rpc }); const gateways = await ario.getGateways({ limit: 100, sortOrder: "desc", sortBy: "operatorStake", }); ``` Available `sortBy` options are any of the keys on the gateway object, e.g. `operatorStake`, `start`, `status`, `settings.fqdn`, `settings.label`, `settings.note`, `settings.port`, `settings.protocol`, `stats.failedConsecutiveEpochs`, `stats.passedConsecutiveEpochs`, etc. **Output:** ```json { "items": [ { "gatewayAddress": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", "observerAddress": "IPdwa3Mb_9pDD8c2IaJx6aad51Ss-_TfStVwBuhtXMs", "operatorStake": 250000000000, "totalDelegatedStake": 0, "settings": { "allowDelegatedStaking": true, "allowedDelegates": [], "autoStake": false, "delegateRewardShareRatio": 10, "minDelegatedStake": 100000000, "fqdn": "ar-io.dev", "label": "ar.io Test", "note": "Test Gateway operated by PDS for the ar.io ecosystem.", "port": 443, "properties": "raJgvbFU-YAnku-WsupIdbTsqqGLQiYpGzoqk9SCVgY", "protocol": "https" }, "startTimestamp": 1720720620813, "endTimestamp": 0, "stats": { "passedConsecutiveEpochs": 30, "failedConsecutiveEpochs": 0, "totalEpochCount": 31, "passedEpochCount": 30, "failedEpochCount": 1, "observedEpochCount": 30, "prescribedEpochCount": 31 }, "status": "joined", "weights": { "stakeWeight": 5.02400000024, "tenureWeight": 0.19444444444444, "gatewayPerformanceRatio": 1, "observerPerformanceRatio": 1, "gatewayRewardRatioWeight": 1, "observerRewardRatioWeight": 1, "compositeWeight": 0.97688888893556, "normalizedCompositeWeight": 0.19247316211083 } } ], "hasMore": true, "nextCursor": "-4xgjroXENKYhTWqrBo57HQwvDL51mMdfsdsxJy6Y2Z_sA", "totalItems": 316, "limit": 100, "sortBy": "operatorStake", "sortOrder": "desc" } ``` #### getGatewayDelegates() Retrieves all delegates for a specific gateway, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last delegate address from the previous request. ```typescript const ario = ARIO.init({ rpc }); const delegates = await ario.getGatewayDelegates({ address: "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", limit: 3, sortBy: "startTimestamp", sortOrder: "desc", }); ``` **Output:** ```json { "nextCursor": "ScEtph9-vfY7lgqlUWwUwOmm99ySeZGQhOX0MFAyFEs", "limit": 3, "sortBy": "startTimestamp", "totalItems": 32, "sortOrder": "desc", "hasMore": true, "items": [ { "delegatedStake": 600000000, "address": "qD5VLaMYyIHlT6vH59TgYIs6g3EFlVjlPqljo6kqVxk", "startTimestamp": 1732716956301 }, { "delegatedStake": 508999038, "address": "KG8TlcWk-8pvroCjiLD2J5zkG9rqC6yYaBuZNqHEyY4", "startTimestamp": 1731828123742 }, { "delegatedStake": 510926479, "address": "ScEtph9-vfY7lgqlUWwUwOmm99ySeZGQhOX0MFAyFEs", "startTimestamp": 1731689356040 } ] } ``` #### joinNetwork() Joins a gateway to the ar.io network via its associated wallet. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.joinNetwork( { qty: new ARIOToken(10_000).toMARIO(), // minimum operator stake allowed allowDelegatedStaking: true, // allows delegated staking minDelegatedStake: new ARIOToken(100).toMARIO(), // minimum delegated stake allowed delegateRewardShareRatio: 10, // percentage of rewards to share with delegates (e.g. 10%) label: "john smith", // min 1, max 64 characters note: "The example gateway", // max 256 characters properties: "FH1aVetOoulPGqgYukj0VE0wIhDy90WiQoV3U2PeY44", // Arweave transaction ID containing additional properties of the Gateway observerWallet: "0VE0wIhDy90WiQoV3U2PeY44FH1aVetOoulPGqgYukj", // wallet address of the observer, must match OBSERVER_WALLET on the observer fqdn: "example.com", // fully qualified domain name - note: you must own the domain and set the OBSERVER_WALLET on your gateway to match `observerWallet` port: 443, // port number protocol: "https", // only 'https' is supported }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### leaveNetwork() Sets the gateway as `leaving` on the ar.io network. Requires `signer` to be provided on `ARIO.init` to sign the transaction. The gateways operator and delegate stakes are vaulted and will be returned after leave periods. The gateway will be removed from the network after the leave period. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.leaveNetwork( // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### updateGatewaySettings() Writes new gateway settings to the callers gateway configuration. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.updateGatewaySettings( { // any other settings you want to update minDelegatedStake: new ARIOToken(100).toMARIO(), }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### updateOperationsAddress() Authorises a second address to update the caller's gateway metadata (see `updateGatewayMetadata`) and to spend the gateway's ArNS discount. Staking, delegation settings and the operations address itself stay operator-only. Pass the operator's own address to revoke a delegation. A gateway that has not yet been migrated to schema 1.2.0 cannot hold an operations address; in that case `migrate_gateway` is added to the same transaction automatically. _Note: Solana-only, so it is on `SolanaARIOWriteable` rather than the cross-backend `ARIOWrite` type. Must be signed by the gateway operator._ ```typescript const ario = new SolanaARIOWriteable({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.updateOperationsAddress({ operationsAddress: "DeLegateAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", }); ``` #### updateGatewayMetadata() Updates a gateway's `label`, `fqdn`, `port`, `protocol`, `properties` and/or `note`. Can be signed by the gateway operator, or by its operations address — in which case pass the operator as `gatewayAddress`. An operations address is only honoured once the gateway is at schema 1.2.0. _Note: Solana-only (`SolanaARIOWriteable`)._ ```typescript // signed by the gateway's operations address const ario = new SolanaARIOWriteable({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.updateGatewayMetadata({ gatewayAddress: "GatewayAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", fqdn: "gateway.example.com", port: 443, }); ``` When reading gateways, `operationsAddress` is only present for gateways at schema 1.2.0 or later; below that the program ignores the field and only the operator can act. #### migrateGateway() / `migrateGateways({ gatewayAddresses?, batchSize? })` Migrates Gateway accounts to schema 1.2.0 (ADR-0030). Permissionless — the signer pays a small rent top-up (32 bytes per gateway). `migrateGateways` with no `gatewayAddresses` migrates every gateway still below 1.2.0 (see `getUnmigratedGatewayAddresses()`), 8 per transaction by default, and stops at the first failed batch. _Note: Solana-only (`SolanaARIOWriteable`)._ ```typescript const ario = new SolanaARIOWriteable({ rpc, rpcSubscriptions, signer }); const pending = await ario.getUnmigratedGatewayAddresses(); const { migrated, signatures } = await ario.migrateGateways(); ``` #### transferEpochSettingsAuthority() Hands `EpochSettings.authority` — the key for epoch admin instructions — to a new address, such as a multisig (ADR-0031). Must be signed by the current authority. _Note: Solana-only (`SolanaARIOWriteable`)._ ```typescript const ario = new SolanaARIOWriteable({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.transferEpochSettingsAuthority({ newAuthority: "MuLtisigAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", }); ``` #### increaseDelegateStake() Increases the callers stake on the target gateway. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.increaseDelegateStake( { target: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", qty: new ARIOToken(100).toMARIO(), }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### decreaseDelegateStake() Decreases the callers stake on the target gateway. Can instantly decrease stake by setting instant to `true`. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.decreaseDelegateStake( { target: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", qty: new ARIOToken(100).toMARIO(), }, { tags: [{ name: "App-Name", value: "My-Awesome-App" }], }, ); ``` Pay the early withdrawal fee and withdraw instantly. ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.decreaseDelegateStake({ target: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", qty: new ARIOToken(100).toMARIO(), instant: true, // Immediately withdraw this stake and pay the instant withdrawal fee }); ``` #### getDelegations() Retrieves all active and vaulted stakes across all gateways for a specific address, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last delegationId (concatenated gateway and startTimestamp of the delgation) from the previous request. ```typescript const ario = ARIO.init({ rpc }); const vaults = await ario.getDelegations({ address: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", cursor: "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ_123456789", limit: 2, sortBy: "startTimestamp", sortOrder: "asc", }); ``` **Output:** ```json { "sortOrder": "asc", "hasMore": true, "totalItems": 95, "limit": 2, "sortBy": "startTimestamp", "items": [ { "type": "stake", "startTimestamp": 1727815440632, "gatewayAddress": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", "delegationId": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ_1727815440632", "balance": 1383212512 }, { "type": "vault", "startTimestamp": 1730996691117, "gatewayAddress": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", "delegationId": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ_1730996691117", "vaultId": "_sGDS7X1hyLCVpfe40GWioH9BSOb7f0XWbhHBa1q4-g", "balance": 50000000, "endTimestamp": 1733588691117 } ], "nextCursor": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ_1730996691117" } ``` #### instantWithdrawal() Instantly withdraws an existing vault on a gateway. If no `gatewayAddress` is provided, the signer's address will be used. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); // removes a delegated vault from a gateway const { id: txId } = await ario.instantWithdrawal( { // gateway address where delegate vault exists gatewayAddress: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", // delegated vault id to cancel vaultId: "fDrr0_J4Iurt7caNST02cMotaz2FIbWQ4Kcj616RHl3", }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }], }, ); // removes an operator vault from a gateway const { id: txId } = await ario.instantWithdrawal({ vaultId: "fDrr0_J4Iurt7caNST02cMotaz2FIbWQ4Kcj616RHl3", }); ``` #### cancelWithdrawal() Cancels an existing vault on a gateway. The vaulted stake will be returned to the callers stake. If no `gatewayAddress` is provided, the signer's address will be used. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); // cancels a delegated vault from a gateway const { id: txId } = await ario.cancelWithdrawal( { // gateway address where vault exists gatewayAddress: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", // vault id to cancel vaultId: "fDrr0_J4Iurt7caNST02cMotaz2FIbWQ4Kcj616RHl3", }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); // cancels an operator vault from a gateway const { id: txId } = await ario.cancelWithdrawal({ // operator vault id to cancel vaultId: "fDrr0_J4Iurt7caNST02cMotaz2FIbWQ4Kcj616RHl3", }); ``` #### getAllowedDelegates() Retrieves all allowed delegates for a specific address. The `cursor` used for pagination is the last address from the previous request. ```typescript const ario = ARIO.init({ rpc }); const allowedDelegates = await ario.getAllowedDelegates({ address: "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", }); ``` **Output:** ```json { "sortOrder": "desc", "hasMore": false, "totalItems": 4, "limit": 100, "items": [ "PZ5vIhHf8VY969TxBPQN-rYY9CNFP9ggNsMBqlWUzWM", "N4h8M9A9hasa3tF47qQyNvcKjm4APBKuFs7vqUVm-SI", "JcC4ZLUY76vmWha5y6RwKsFqYTrMZhbockl8iM9p5lQ", "31LPFYoow2G7j-eSSsrIh8OlNaARZ84-80J-8ba68d8" ] } ``` #### getGatewayVaults() Retrieves all vaults across all gateways for a specific address, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last vaultId from the previous request. ```typescript const ario = ARIO.init({ rpc }); const vaults = await ario.getGatewayVaults({ address: '"PZ5vIhHf8VY969TxBPQN-rYY9CNFP9ggNsMBqlWUzWM', }); ``` **Output:** ```json { "sortOrder": "desc", "hasMore": false, "totalItems": 1, "limit": 100, "sortBy": "endTimestamp", "items": [ { "cursorId": "PZ5vIhHf8VY969TxBPQN-rYY9CNFP9ggNsMBqlWUzWM_1728067635857", "startTimestamp": 1728067635857, "balance": 50000000000, "vaultId": "PZ5vIhHf8VY969TxBPQN-rYY9CNFP9ggNsMBqlWUzWM", "endTimestamp": 1735843635857 } ] } ``` #### getAllGatewayVaults() Retrieves all vaults across all gateways, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last vaultId from the previous request. ```typescript const ario = ARIO.init({ rpc }); const vaults = await ario.getAllGatewayVaults({ limit: 1, sortBy: "endTimestamp", sortOrder: "desc", }); ``` **Output:** ```json { "sortOrder": "desc", "hasMore": true, "totalItems": 95, "limit": 1, "sortBy": "endTimestamp", "items": [ { "cursorId": "PZ5vIhHf8VY969TxBPQN-rYY9CNFP9ggNsMBqlWUzWM_E-QVU3dta36Wia2uQw6tQLjQk7Qw5uN0Z6fUzsoqzUc", "gatewayAddress": "PZ5vIhHf8VY969TxBPQN-rYY9CNFP9ggNsMBqlWUzWM", "startTimestamp": 1728067635857, "balance": 50000000000, "vaultId": "E-QVU3dta36Wia2uQw6tQLjQk7Qw5uN0Z6fUzsoqzUc", "endTimestamp": 1735843635857 } ], "nextCursor": "PZ5vIhHf8VY969TxBPQN-rYY9CNFP9ggNsMBqlWUzWM_E-QVU3dta36Wia2uQw6tQLjQk7Qw5uN0Z6fUzsoqzUc" } ``` #### getWithdrawals() Returns every pending stake withdrawal owned by `address` — covering both operator-stake decreases (`isDelegate: false`) and delegate-stake decreases (`isDelegate: true`). A withdrawal is claimable when `Date.now() >= endTimestamp`; call `claimWithdrawal({ withdrawalId: item.vaultId })` to release the tokens. This is the per-owner read needed to drive "you have X claimable withdrawals" UIs without fanning out across every gateway the wallet has interacted with. ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const withdrawals = await ario.getWithdrawals({ address: "9xQeWvG816bUx9EPjHmaT23yvVM2ZWbrrpZb9PusVFin", }); const claimable = withdrawals.items.filter( (w) => Date.now() >= w.endTimestamp, ); ``` **Output:** ```json { "hasMore": false, "totalItems": 2, "limit": 100, "items": [ { "cursorId": "8CSdSjf7gXqQ5p1U2qfdwHzVw9sZRYHJpDpV87dnvb4d", "vaultId": "0", "gatewayAddress": "Bxz7Q2tWfqr9Q5T6cZjUnVxRk9CnHwShfgUaW5fY1Mvr", "balance": 50000000000, "startTimestamp": 1735843635857, "endTimestamp": 1738435635857, "isDelegate": true }, { "cursorId": "FmWUz4w7vSdLcz1nN8H1n2KkjJgrQQXR1n4kV3WqJ7Hf", "vaultId": "1", "gatewayAddress": "9xQeWvG816bUx9EPjHmaT23yvVM2ZWbrrpZb9PusVFin", "balance": 10000000000, "startTimestamp": 1735843835857, "endTimestamp": 1738435835857, "isDelegate": false } ] } ``` #### increaseOperatorStake() Increases the callers operator stake. Must be executed with a wallet registered as a gateway operator. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.increaseOperatorStake( { qty: new ARIOToken(100).toMARIO(), }, { tags: [{ name: "App-Name", value: "My-Awesome-App" }], }, ); ``` #### decreaseOperatorStake() Decreases the callers operator stake. Must be executed with a wallet registered as a gateway operator. Requires `signer` to be provided on `ARIO.init` to sign the transaction. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.decreaseOperatorStake( { qty: new ARIOToken(100).toMARIO(), }, { tags: [{ name: "App-Name", value: "My-Awesome-App" }], }, ); ``` #### redelegateStake() Redelegates the stake of a specific address to a new gateway. Vault ID may be optionally included in order to redelegate from an existing withdrawal vault. The redelegation fee is calculated based on the fee rate and the stake amount. Users are allowed one free redelegation every seven epochs. Each additional redelegation beyond the free redelegation will increase the fee by 10%, capping at a 60% redelegation fee. e.g: If 1000 mARIO is redelegated and the fee rate is 10%, the fee will be 100 mARIO. Resulting in 900 mARIO being redelegated to the new gateway and 100 mARIO being deducted back to the protocol balance. ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.redelegateStake({ target: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", source: "HwFceQaMQnOBgKDpnFqCqgwKwEU5LBme1oXRuQOWSRA", stakeQty: new ARIOToken(1000).toMARIO(), vaultId: "fDrr0_J4Iurt7caNST02cMotaz2FIbWQ4Kcj616RHl3", }); ``` #### getRedelegationFee() Retrieves the fee rate as percentage required to redelegate the stake of a specific address. Fee rate ranges from 0% to 60% based on the number of redelegations since the last fee reset. ```typescript const ario = ARIO.init({ rpc }); const fee = await ario.getRedelegationFee({ address: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", }); ``` **Output:** ```json { "redelegationFeeRate": 10, "feeResetTimestamp": 1730996691117 } ``` #### getAllDelegates() Retrieves all delegates across all gateways, paginated and sorted by the specified criteria. The `cursor` used for pagination is a `cursorId` derived from delegate address and the gatewayAddress from the previous request. e.g `address_gatewayAddress`. ```typescript const ario = ARIO.init({ rpc }); const delegates = await ario.getAllDelegates({ limit: 2, sortBy: "startTimestamp", sortOrder: "desc", }); ``` **Output:** ```json { "sortOrder": "desc", "hasMore": true, "totalItems": 95, "limit": 2, "sortBy": "startTimestamp", "items": [ { "startTimestamp": 1734709397622, "cursorId": "9jfM0uzGNc9Mkhjo1ixGoqM7ygSem9wx_EokiVgi0Bs_E-QVU3dta36Wia2uQw6tQLjQk7Qw5uN0Z6fUzsoqzUc", "gatewayAddress": "E-QVU3dta36Wia2uQw6tQLjQk7Qw5uN0Z6fUzsoqzUc", "address": "9jfM0uzGNc9Mkhjo1ixGoqM7ygSem9wx_EokiVgi0Bs", "delegatedStake": 2521349108, "vaultedStake": 0 }, { "startTimestamp": 1734593229454, "cursorId": "LtV0aSqgK3YI7c5FmfvZd-wG95TJ9sezj_a4syaLMS8_M0WP8KSzCvKpzC-HPF1WcddLgGaL9J4DGi76iMnhrN4", "gatewayAddress": "M0WP8KSzCvKpzC-HPF1WcddLgGaL9J4DGi76iMnhrN4", "address": "LtV0aSqgK3YI7c5FmfvZd-wG95TJ9sezj_a4syaLMS8", "delegatedStake": 1685148110, "vaultedStake": 10000000 } ], "nextCursor": "PZ5vIhHf8VY969TxBPQN-rYY9CNFP9ggNsMBqlWUzWM_QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ" } ``` # General (/sdks/ar-io-sdk/(ario-contract)/general) #### init() Factory function that creates a read-only or writeable ARIO client. Providing `signer` plus `rpcSubscriptions` enables write methods (`joinNetwork`, `delegateStake`, `buyRecord`, etc.). Without a signer, the client is read-only. ```typescript import { createSolanaRpc, createSolanaRpcSubscriptions, createKeyPairSignerFromBytes, } from '@solana/kit'; const rpc = createSolanaRpc('https://api.mainnet-beta.solana.com'); // read-only client const ario = ARIO.init({ rpc }); // read-write client (needs rpcSubscriptions for sendAndConfirm) const rpcSubscriptions = createSolanaRpcSubscriptions( 'wss://api.mainnet-beta.solana.com', ); const signer = await createKeyPairSignerFromBytes(/* 64-byte secret key */); const arioWrite = ARIO.init({ rpc, rpcSubscriptions, signer }); ``` #### getInfo() Retrieves the information of the ARIO process. ```typescript const ario = ARIO.init({ rpc }); const info = await ario.getInfo(); ``` **Output:** ```json { "Name": "AR.IO", "Ticker": "ARIO", "Logo": "", "Denomination": 6, "Handlers": [], "LastCreatedEpochIndex": 0, "LastDistributedEpochIndex": 0, "totalSupply": 1000000000000000, "protocolBalance": 0, "epochSettings": { "durationMs": 86400000, "prescribedNameCount": 25, "maxObservers": 50 } } ``` > **Note**: `Handlers`, `LastCreatedEpochIndex`, and `LastDistributedEpochIndex` > are placeholders on Solana (returned for backwards-compatible field shape > with consumer code). `totalSupply` / `protocolBalance` are live reads from > the `ArioConfig` PDA; `epochSettings` is live from the `EpochSettings` > PDA. See `src/solana/io-readable.ts` for the exact projection. #### getTokenSupply() Retrieves the total supply of tokens, returned in mARIO. The total supply includes the following: - `total` - the total supply of all tokens - `circulating` - the total supply minus locked, withdrawn, delegated, and staked - `locked` - tokens that are locked in the protocol (a.k.a. vaulted) - `withdrawn` - tokens that have been withdrawn from the protocol by operators and delegators - `delegated` - tokens that have been delegated to gateways - `staked` - tokens that are staked in the protocol by gateway operators - `protocolBalance` - tokens that are held in the protocol's treasury. This is included in the circulating supply. ```typescript const ario = ARIO.init({ rpc }); const supply = await ario.getTokenSupply(); ``` **Output:** ```json { "total": 1000000000000000000, "circulating": 998094653842520, "locked": 0, "withdrawn": 560563387278, "delegated": 1750000000, "staked": 1343032770199, "protocolBalance": 46317263683761 } ``` #### getBalance() Retrieves the balance of the specified wallet address. ```typescript const ario = ARIO.init({ rpc }); // the balance will be returned in mARIO as a value const balance = await ario .getBalance({ address: "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", }) .then((balance: number) => new mARIOToken(balance).toARIO()); // convert it to ARIO for readability ``` **Output:** ```json 100000 ``` #### getBalances() Retrieves the balances of the ARIO process in `mARIO`, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last wallet address from the previous request. ```typescript const ario = ARIO.init({ rpc }); const balances = await ario.getBalances({ cursor: "-4xgjroXENKYhTWqrBo57HQwvDL51mMdfsdsxJy6Y2Z_sA", limit: 100, sortBy: "balance", sortOrder: "desc", }); ``` **Output:** ```json { "items": [ { "address": "-4xgjroXENKYhTWqrBo57HQwvDL51mMvSxJy6Y2Z_sA", "balance": 1000000 }, { "address": "-7vXsQZQDk8TMDlpiSLy3CnLi5PDPlAaN2DaynORpck", "balance": 1000000 } // ...98 other balances ], "hasMore": true, "nextCursor": "-7vXsQZQDk8TMDlpiSLy3CnLi5PDPlAaN2DaynORpck", "totalItems": 1789, "sortBy": "balance", "sortOrder": "desc" } ``` #### transfer() Transfers `mARIO` to the designated `target` recipient address. Requires `signer` to be provided on `ARIO.init` to sign the transaction. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.transfer({ target: 'RecipientSolanaPubkeyBase58', qty: new ARIOToken(1000).toMARIO(), }); ``` # Networks (/sdks/ar-io-sdk/(ario-contract)/networks) The SDK talks to whatever cluster your `@solana/kit` RPC client points at — mainnet-beta by default. For devnet or a local validator, override the RPC URL and (on any non-mainnet cluster) the per-program addresses: ```typescript const ario = ARIO.init({ rpc: createSolanaRpc('https://api.devnet.solana.com'), coreProgramId: address('\'), garProgramId: address('\'), arnsProgramId: address('\'), antProgramId: address('\'), }); ``` On localnet (Surfpool) source program IDs from `migration/localnet/out/localnet.env` in the `solana-ar-io` monorepo. # Primary Names (/sdks/ar-io-sdk/(ario-contract)/primary-names) #### getPrimaryNames() Retrieves all primary names paginated and sorted by the specified criteria. The `cursor` used for pagination is the last name from the previous request. ```typescript const ario = ARIO.init({ rpc }); const names = await ario.getPrimaryNames({ cursor: "ao", // this is the last name from the previous request limit: 1, sortBy: "startTimestamp", sortOrder: "desc", }); ``` **Output:** ```json { "sortOrder": "desc", "hasMore": true, "totalItems": 100, "limit": 1, "sortBy": "startTimestamp", "nextCursor": "arns", "items": [ { "name": "arns", "owner": "HwFceQaMQnOBgKDpnFqCqgwKwEU5LBme1oXRuQOWSRA", "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "startTimestamp": 1719356032297 } ] } ``` #### getPrimaryName() Retrieves the primary name for a given name or address. ```typescript const ario = ARIO.init({ rpc }); const name = await ario.getPrimaryName({ name: "arns", }); // or const name = await ario.getPrimaryName({ address: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", }); ``` **Output:** ```json { "name": "arns", "owner": "HwFceQaMQnOBgKDpnFqCqgwKwEU5LBme1oXRuQOWSRA", "processId": "bh9l1cy0aksiL_x9M359faGzM_yjralacHIUo8_nQXM", "startTimestamp": 1719356032297 } ``` #### setPrimaryName() Sets an ArNS name already owned by the `signer` as their primary name. Note: `signer` must be the owner of the `processId` that is assigned to the name. If not, the transaction will fail. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); await ario.setPrimaryName({ name: 'my-arns-name' }); ``` #### requestPrimaryName() Requests a primary name for the `signer`'s address. The request must be approved by the new owner of the requested name via the `approvePrimaryNameRequest`[#approveprimarynamerequest-name-address-] API. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.requestPrimaryName({ name: "arns", }); ``` #### getPrimaryNameRequest() Retrieves the primary name request for a a wallet address. ```typescript const ario = ARIO.init({ rpc }); const request = await ario.getPrimaryNameRequest({ initiator: "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", }); ``` **Output:** ```json { "initiator": "t4Xr0_J4Iurt7caNST02cMotaz2FIbWQ4Kbj616RHl3", "name": "arns", "startTimestamp": 1728067635857, "endTimestamp": 1735843635857 } ``` # Vaults (/sdks/ar-io-sdk/(ario-contract)/vaults) #### getVault() Retrieves the locked-balance user vault of the ARIO process by the specified wallet address and vault ID. ```typescript const ario = ARIO.init({ rpc }); const vault = await ario.getVault({ address: "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", vaultId: "vaultIdOne", }); ``` **Output:** ```json { "balance": 1000000, "startTimestamp": 123, "endTimestamp": 4567 } ``` #### getVaults() Retrieves all locked-balance user vaults of the ARIO process, paginated and sorted by the specified criteria. The `cursor` used for pagination is the last wallet address from the previous request. ```typescript const ario = ARIO.init({ rpc }); const vaults = await ario.getVaults({ cursor: "0", limit: 100, sortBy: "balance", sortOrder: "desc", }); ``` **Output:** ```json { "items": [ { "address": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", "vaultId": "vaultIdOne", "balance": 1000000, "startTimestamp": 123, "endTimestamp": 4567 }, { "address": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", "vaultId": "vaultIdTwo", "balance": 1000000, "startTimestamp": 123, "endTimestamp": 4567 } // ...98 other addresses with vaults ], "hasMore": true, "nextCursor": "QGWqtJdLLgm2ehFWiiPzMaoFLD50CnGuzZIPEdoDRGQ", "totalItems": 1789, "sortBy": "balance", "sortOrder": "desc" } ``` #### vaultedTransfer() Transfers `mARIO` to the designated `recipient` address and locks the balance for the specified `lockLengthMs` milliseconds. The `revokable` flag determines if the vaulted transfer can be revoked by the sender. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.vaultedTransfer( { recipient: "-5dV7nk7waR8v4STuwPnTck1zFVkQqJh5K9q9Zik4Y5", quantity: new ARIOToken(1000).toMARIO(), lockLengthMs: 1000 * 60 * 60 * 24 * 365, // 1 year revokable: true, }, // optional additional tags { tags: [{ name: "App-Name", value: "My-Awesome-App" }] }, ); ``` #### revokeVault() Revokes a vaulted transfer by the recipient address and vault ID. Only the sender of the vaulted transfer can revoke it. _Note: Requires `signer` to be provided on `ARIO.init` to sign the transaction._ ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.revokeVault({ recipient: "-5dV7nk7waR8v4STuwPnTck1zFVkQqJh5K9q9Zik4Y5", vaultId: "IPdwa3Mb_9pDD8c2IaJx6aad51Ss-_TfStVwBuhtXMs", }); ``` #### createVault() Creates a vault for the specified `quantity` of mARIO from the signer's balance and locks it for the specified `lockLengthMs` milliseconds. ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.createVault({ lockLengthMs: 1000 * 60 * 60 * 24 * 365, // 1 year quantity: new ARIOToken(1000).toMARIO(), }); ``` #### extendVault() Extends the lock length of a signer's vault by the specified `extendLengthMs` milliseconds. ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.extendVault({ vaultId: "vaultIdOne", extendLengthMs: 1000 * 60 * 60 * 24 * 365, // 1 year }); ``` #### increaseVault() Increases the balance of a signer's vault by the specified `quantity` of mARIO. ```typescript const ario = ARIO.init({ rpc, rpcSubscriptions, signer }); const { id: txId } = await ario.increaseVault({ vaultId: "vaultIdOne", quantity: new ARIOToken(1000).toMARIO(), }); ``` # Escrow (/sdks/ar-io-sdk/escrow) Trustless, multi-protocol escrow for handing an asset to a recipient identified by an **Arweave** or **Ethereum** address, claimable once they hold a Solana wallet. Backed by the `ario-ant-escrow` program. Two clients: - `TokenEscrow` — escrow liquid **ARIO** (SPL) or a **time-locked vault**. - `ANTEscrow` — escrow an **ANT** (Metaplex Core NFT). Each supports **deposit → claim → cancel/refund → update-recipient**. Claims work three ways: **Arweave-attested** (an off-chain attestor re-signs the canonical claim with Ed25519, verified on-chain), **Ethereum** (on-chain `secp256k1_recover` + EIP-191), and **vault** (instruction introspection that preserves the remaining lock). ```typescript const escrow = new TokenEscrow({ rpc, rpcSubscriptions, signer, programId, coreProgram, }); // deposit 50 ARIO to an Ethereum recipient await escrow.depositTokens({ assetId, // 32-byte client-supplied id amount: 50_000_000n, arioMint, depositorTokenAccount, recipient: { protocol: 'ethereum', publicKey: ethAddress20 }, }); // the recipient claims (Ethereum path) once they have a Solana wallet await escrow.claimTokensEthereum({ depositor, assetId, claimant, claimantTokenAccount, escrowTokenAccount, signature, // recipient's EIP-191 signature over canonicalMessageV2(...) }); ``` Build the exact bytes a recipient signs with `canonicalMessage` / `canonicalMessageV2` — byte-identical to the on-chain program (and the off-chain attestor). See the contracts repo's escrow design + protocol spec for the full flow and the cross-language canonical-message vectors. # ar.io SDK (/sdks/ar-io-sdk) **For AI and LLM users**: Access the complete ar.io SDK documentation in plain text format at{" "} llm.txt for easy consumption by AI agents and language models. See [AI Agents & LLMs](/build/agents) for the full agent toolkit. The ar.io SDK provides comprehensive tools for interacting with ar.io and the Arweave ecosystem. Built with TypeScript, it offers type-safe interfaces for ArNS name management, gateway operations, and Solana program interactions. The SDK defaults to the Solana backend as of v3.23. ## Quick Start **Requires `@ar.io/sdk` version 3.23+** for Solana support. Write operations also need `@solana/kit` for signing — see the [Configuration](/sdks/ar-io-sdk/configuration) page for setup details. ### Install the SDK ```npm npm install @ar.io/sdk ``` ### Use the SDK ```javascript // Connect to mainnet const ario = ARIO.mainnet(); // Get paginated gateway registry const { items: gateways } = await ario.getGateways(); // Get paginated ArNS registry const { items: records } = await ario.getArNSRecords(); console.log('Network data:', { gateways, records }); ``` ### Install the SDK ```npm npm install @ar.io/sdk ``` ### Install polyfills (required for web environments) Polyfills are required for React web environments due to the use of `crypto`, `buffer` and `process` types in the SDK's dependencies. ```npm npm install --save-dev vite-plugin-node-polyfills ``` ```js // vite.config.js plugins: [ nodePolyfills({ globals: { Buffer: true, global: true, process: true, }, }), ], }); ``` Configure your bundler (Webpack, Vite, Rollup, etc.) to provide polyfills for `crypto`, `process`, and `buffer`. Refer to your bundler's documentation for polyfill configuration. ### Use the SDK ```javascript // Connect to mainnet const ario = ARIO.mainnet(); // Get paginated gateway registry const { items: gateways } = await ario.getGateways(); // Get paginated ArNS registry const { items: records } = await ario.getArNSRecords(); console.log('Network data:', { gateways, records }); ``` ```html ar.io SDK Example // Polyfills are included in the minimized web bundle, so not necessary to import directly // Connect to mainnet const ario = ARIO.mainnet(); // Function to load ar.io data async function loadArioData() { try { // Get gateway information const { items: gateways } = await ario.getGateways(); // Get ArNS records const { items: records } = await ario.getArNSRecords(); // Display results document.getElementById('results').innerHTML = ` Gateways: ${Object.keys(gateways).length} ArNS Records: ${Object.keys(records).length} ${JSON.stringify({ gateways, records }, null, 2)} `; } catch (error) { document.getElementById('results').innerHTML = ` Error: ${error.message} `; } } // Load data when page loads window.addEventListener('load', loadArioData); ar.io SDK Example Loading ar.io data... ``` ## API Reference & Documentation } title="API Reference" description="Complete API documentation for all SDK methods and classes" href="/apis/ar-io-node" /> } title="SDK Details" description="Detailed guides for ARIO contracts, ANT operations, and utilities" href="/sdks/ar-io-sdk/ar-io-name-system-arns" /> ## Core Features } title="ARIO Protocol Operations" description="ArNS management, gateway discovery, and network configuration" href="/sdks/ar-io-sdk/ar-io-name-system-arns" /> } title="ANT Contract Integration" description="Initialize, manage records, and transfer ANT ownership" href="/sdks/ar-io-sdk/initialize" /> } title="Pagination & Utilities" description="Handle large datasets and token conversions efficiently" href="/sdks/ar-io-sdk/pagination" /> } title="Token Operations" description="ARIO token conversions and management utilities" href="/sdks/ar-io-sdk/token-conversion" /> # Logging (/sdks/ar-io-sdk/logging) The library uses a lightweight console logger by default for both Node.js and web environments. The logger outputs structured JSON logs with timestamps. You can configure the log level via `setLogLevel()` API or provide a custom logger that satisfies the `ILogger` interface. #### Default Logger ```typescript // set the log level Logger.default.setLogLevel("debug"); // Create a new logger instance with a specific level const logger = new Logger({ level: "debug" }); ``` #### Custom Logger Implementation You can provide any custom logger that implements the `ILogger` interface: ```typescript // Custom logger example const customLogger: ILogger = { info: (message, ...args) => console.log(`[INFO] ${message}`, ...args), warn: (message, ...args) => console.warn(`[WARN] ${message}`, ...args), error: (message, ...args) => console.error(`[ERROR] ${message}`, ...args), debug: (message, ...args) => console.debug(`[DEBUG] ${message}`, ...args), setLogLevel: (level) => { /* implement level filtering */ }, }; // Set it as the default logger across the entire SDK — every class // (ARIO, ANT, ANTRegistry, etc.) will route logs through it. `ARIO.init` // does not accept a per-instance logger. Logger.default = customLogger; ``` # Pagination (/sdks/ar-io-sdk/pagination) #### Overview Certain APIs that could return a large amount of data are paginated using cursors. The SDK uses the `cursor` pattern (as opposed to pages) to better protect against changing data while paginating through a list of items. For more information on pagination strategies refer to [this article](https://www.getknit.dev/blog/api-pagination-best-practices#api-pagination-techniques-). Paginated results include the following properties: - `items`: the list of items on the current request, defaulted to 100 items. - `nextCursor`: the cursor to use for the next batch of items. This is `undefined` if there are no more items to fetch. - `hasMore`: a boolean indicating if there are more items to fetch. This is `false` if there are no more items to fetch. - `totalItems`: the total number of items available. This may change as new items are added to the list, only use this for informational purposes. - `sortBy`: the field used to sort the items, by default this is `startTimestamp`. - `sortOrder`: the order used to sort the items, by default this is `desc`. To request all the items in a list, you can iterate through the list using the `nextCursor` until `hasMore` is `false`. ```typescript let hasMore = true; let cursor: string | undefined; const gateaways = []; while (hasMore) { const page = await ario.getGateways({ limit: 100, cursor }); gateaways.push(...items); cursor = page.nextCursor; hasMore = page.hasMore; } ``` #### Filtering Paginated APIs also support filtering by providing a `filters` parameter. Filters can be applied to any field in the response. When multiple keys are provided, they are treated as AND conditions (all conditions must match). When multiple values are provided for a single key (as an array), they are treated as OR conditions (any value can match). Example: ```typescript const records = await ario.getArNSRecords({ filters: { type: "lease", processId: [ "ZkgLfyHALs5koxzojpcsEFAKA8fbpzP7l-tbM7wmQNM", "r61rbOjyXx3u644nGl9bkwLWlWmArMEzQgxBo2R-Vu0", ], }, }); ``` In the example above, the query will return ArNS records where: - The type is "lease" AND - The processId is EITHER "ZkgLfyHALs5koxzojpcsEFAKA8fbpzP7l-tbM7wmQNM" OR "r61rbOjyXx3u644nGl9bkwLWlWmArMEzQgxBo2R-Vu0" # Token Conversion (/sdks/ar-io-sdk/token-conversion) The ARIO process stores all values as mARIO (micro-ARIO) to avoid floating-point arithmetic issues. The SDK provides an `ARIOToken` and `mARIOToken` classes to handle the conversion between ARIO and mARIO, along with rounding logic for precision. **All process interactions expect values in mARIO. If numbers are provided as inputs, they are assumed to be in raw mARIO values.** #### Converting ARIO to mARIO ```typescript const arioValue = 1; const mARIOValue = new ARIOToken(arioValue).toMARIO(); const mARIOValue = 1_000_000; const arioValue = new mARIOToken(mARIOValue).toARIO(); ``` # Anonymous Operations (/sdks/ardrive-core-js/(advanced-features)/anonymous-operations) Use ArDrive without a wallet for read-only operations: ```typescript const anonymousArDrive = arDriveAnonymousFactory({}); // Read public data const publicFile = await anonymousArDrive.getPublicFile({ fileId }); const folderContents = await anonymousArDrive.listPublicFolder({ folderId }); ``` # Bundle Support (/sdks/ardrive-core-js/(advanced-features)/bundle-support) Large uploads are automatically bundled for efficiency: ```typescript // Bundling happens automatically for multiple files const bulkResult = await arDrive.uploadAllEntities({ entitiesToUpload: manyFiles // Bundling is handled internally }); ``` # Caching (/sdks/ardrive-core-js/(advanced-features)/caching) ArDrive Core maintains a metadata cache for improved performance: ```shell Windows: /ardrive-caches/metadata Non-Windows: /.ardrive/caches/metadata ``` Enable cache logging: ```bash ``` # Community Features (/sdks/ardrive-core-js/(advanced-features)/community-features) Send tips to the ArDrive community: ```typescript // Send community tip await arDrive.sendCommunityTip({ tokenAmount: new Winston(1000000000000), // 1 AR walletAddress, communityWalletAddress }); ``` # Incremental Drive Synchronization (/sdks/ardrive-core-js/(advanced-features)/incremental-drive-synchronization) ArDrive Core provides efficient incremental synchronization capabilities for tracking changes in drives over time. This feature enables applications to sync only new or modified content rather than fetching entire drive structures repeatedly. **Note:** The standard `arDriveFactory` creates an ArDrive instance with in-memory sync state caching (5-minute TTL). For persistent storage across sessions, see the [Persistent Storage](#persistent-storage-for-sync-state) section below. #### Basic Sync Operations ```typescript // Important: Requires ArFSDAOIncrementalSync for full functionality // The standard arDriveFactory may not support all sync features // For basic sync with in-memory caching, first create the DAO: const arfsDao = new ArFSDAOIncrementalSync( wallet, arweave, false // dryRun ); const arDrive = arDriveFactory({ wallet: myWallet, arfsDao }); // Now sync operations will work: const syncResult = await arDrive.syncPublicDrive(driveId); console.log(`Found ${syncResult.entities.length} total entities`); console.log(`Added: ${syncResult.changes.added.length}`); console.log(`Modified: ${syncResult.changes.modified.length}`); console.log(`Unreachable: ${syncResult.changes.unreachable.length}`); // Sync a private drive with decryption const privateSyncResult = await arDrive.syncPrivateDrive( driveId, driveKey ); ``` #### Incremental Sync with Previous State ```typescript // First sync - gets all entities const initialSync = await arDrive.syncPublicDrive(driveId); // Save the sync state for later const syncState = initialSync.newSyncState; // Later, sync only changes since last sync const incrementalSync = await arDrive.syncPublicDrive(driveId, undefined, { syncState: syncState // Pass previous state }); // Only new/modified entities since last sync console.log(`New entities: ${incrementalSync.changes.added.length}`); ``` #### Progress Tracking ```typescript // Track sync progress for large drives const result = await arDrive.syncPublicDrive(driveId, undefined, { onProgress: (processed, total) => { console.log(`Progress: ${processed}/${total} entities`); } }); ``` #### Advanced Sync Options ```typescript const syncOptions = { // Include all file revisions (not just latest) includeRevisions: true, // Batch size for GraphQL queries (default: 100, max: 100) batchSize: 50, // Stop early after finding N consecutive known entities (optimization) stopAfterKnownCount: 10, // Progress callback onProgress: (processed, total) => { console.log(`Syncing: ${processed}/${total}`); } }; const result = await arDrive.syncPublicDrive(driveId, owner, syncOptions); ``` #### Working with Sync Results ```typescript // Sync result structure const result = await arDrive.syncPublicDrive(driveId); // All entities in the drive (files and folders) result.entities.forEach(entity => { console.log(`${entity.entityType}: ${entity.name} (${entity.entityId})`); }); // Change detection result.changes.added.forEach(entity => { console.log(`New: ${entity.name}`); }); result.changes.modified.forEach(entity => { console.log(`Modified: ${entity.name} at block ${entity.blockHeight}`); }); result.changes.unreachable.forEach(entity => { console.log(`No longer accessible: ${entity.name}`); }); // Sync statistics console.log(`Processed from cache: ${result.stats.fromCache}`); console.log(`Fetched from network: ${result.stats.fromNetwork}`); console.log(`Block range: ${result.stats.lowestBlockHeight} - ${result.stats.highestBlockHeight}`); ``` #### Error Handling ```typescript try { const result = await arDrive.syncPublicDrive(driveId); } catch (error) { if (error instanceof IncrementalSyncError) { // Partial results are available even if sync failed console.log(`Sync failed but got ${error.partialResult.entities.length} entities`); console.log(`Error: ${error.message}`); // Can continue from partial state const partialState = error.partialResult.newSyncState; } } ``` # Manifest Creation (/sdks/ardrive-core-js/(advanced-features)/manifest-creation) Create Arweave manifests for web hosting: ```typescript // Create a manifest for a folder const manifest = await arDrive.uploadPublicManifest({ folderId, destManifestName: 'index.html', conflictResolution: 'upsert' }); // Access: https://arweave.net/{manifestId} ``` # Persistent Storage for Sync State (/sdks/ardrive-core-js/(advanced-features)/persistent-storage-for-sync-state) By default, sync state is only cached in memory for 5 minutes. To maintain sync state across application restarts, ArDrive Core provides storage adapters that automatically persist and restore sync state. #### Available Storage Adapters - **MemorySyncStateStore** - In-memory storage (default behavior) - **FileSystemSyncStateStore** - Persists to disk (Node.js) - **LocalStorageSyncStateStore** - Browser localStorage - **IndexedDBSyncStateStore** - Browser IndexedDB for larger datasets - **SQLiteSyncStateStore** - SQLite database (optional, see SQLite section below) #### Quick Start: Persistent Sync (Node.js) ```typescript import { arDriveFactory, ArFSDAOIncrementalSync, FileSystemSyncStateStore } from 'ardrive-core-js'; // 1. Create persistent storage adapter const syncStateStore = new FileSystemSyncStateStore('./.ardrive-cache'); // 2. Create DAO with storage adapter const arfsDao = new ArFSDAOIncrementalSync( wallet, arweave, false, // dryRun 'MyApp', '1.0.0', undefined, // use default settings for these undefined, undefined, syncStateStore // ← Pass storage adapter here ); // 3. Create ArDrive with the DAO const arDrive = arDriveFactory({ wallet, arfsDao }); // 4. Sync operations now persist state automatically const result = await arDrive.syncPublicDrive(driveId); // State is saved to disk and will be reused on next run ``` **Important:** The storage adapter must be passed to `ArFSDAOIncrementalSync`, not to `arDriveFactory`. #### Browser Storage Options ```typescript // Option 1: localStorage (simple, ~5-10MB limit) const syncStateStore = new LocalStorageSyncStateStore('ardrive-sync-'); // Option 2: IndexedDB (for larger datasets) const syncStateStore = new IndexedDBSyncStateStore('ardrive-sync-db'); // Use with ArFSDAOIncrementalSync same as Node.js example const arfsDao = new ArFSDAOIncrementalSync( wallet, arweave, false, 'MyApp', '1.0.0', undefined, undefined, undefined, syncStateStore ); ``` #### Working Example See `examples/persistent-sync-example.js` for a complete working example that demonstrates: - Setting up persistent storage - Performing initial full sync - Simulating app restart - Performing incremental sync from saved state - Managing stored sync states #### SQLite Storage (Optional) The SQLite adapter provides advanced features like statistics, cleanup, and backups. To use it: 1. Install the peer dependency: `yarn add better-sqlite3` 2. Copy `src/utils/sync_state_store_sqlite.ts.optional` to your project 3. Import and use like other storage adapters **Note:** SQLite adapter is not included in the default build to avoid forcing the peer dependency. #### Storage Management Methods All storage adapters implement these methods: ```typescript // List all drives with cached state const driveIds = await syncStateStore.list(); // Load specific drive state const state = await syncStateStore.load(driveId); // Clear specific drive await syncStateStore.clear(driveId); // Clear all cached states await syncStateStore.clearAll(); ``` #### Custom Storage Implementation Create your own storage adapter by implementing the `SyncStateStore` interface: ```typescript class CustomSyncStateStore implements SyncStateStore { async save(driveId: DriveID, state: DriveSyncState): Promise\ { // Your storage logic } async load(driveId: DriveID): Promise\ { // Your retrieval logic } async clear(driveId: DriveID): Promise\ { // Your deletion logic } async list(): Promise { // Return all stored drive IDs } async clearAll(): Promise\ { // Clear all stored states } } ``` #### Serialization for External Storage If you need to store sync state in an external system: ```typescript // Serialize state for storage const result = await arDrive.syncPublicDrive(driveId); const serialized = serializeSyncState(result.newSyncState); const jsonString = JSON.stringify(serialized); // Store in your backend await myAPI.saveSyncState(driveId, jsonString); // Later, retrieve and deserialize const stored = await myAPI.getSyncState(driveId); const parsed = JSON.parse(stored); const syncState = deserializeSyncState(parsed); // Use restored state for incremental sync const nextSync = await arDrive.syncPublicDrive(driveId, undefined, { syncState }); ``` # Progress Tracking (/sdks/ardrive-core-js/(advanced-features)/progress-tracking) Enable upload progress logging: ```bash ``` Progress will be logged to stderr: ``` Uploading file transaction 1 of total 2 transactions... Transaction _GKQasQX194a364Hph8Oe-oku1AdfHwxWOw9_JC1yjc Upload Progress: 0% Transaction _GKQasQX194a364Hph8Oe-oku1AdfHwxWOw9_JC1yjc Upload Progress: 35% Transaction _GKQasQX194a364Hph8Oe-oku1AdfHwxWOw9_JC1yjc Upload Progress: 66% Transaction _GKQasQX194a364Hph8Oe-oku1AdfHwxWOw9_JC1yjc Upload Progress: 100% ``` # Turbo Integration (/sdks/ardrive-core-js/(advanced-features)/turbo-integration) Enable Turbo for optimized uploads: ```typescript // Node.js const arDriveWithTurbo = arDriveFactory({ wallet: myWallet, turboSettings: { turboUploadUrl: new URL('https://upload.ardrive.io') } }); // Browser const arDrive = arDriveFactory({ signer: myBrowserSigner, turboSettings: { turboUploadUrl: new URL('https://upload.ardrive.io') } }); // Uploads will automatically use Turbo const result = await arDriveWithTurbo.uploadAllEntities({ entitiesToUpload: [{ wrappedEntity, destFolderId }] }); ``` # Bulk Operations (/sdks/ardrive-core-js/(api-reference)/bulk-operations) #### Upload Multiple Files and Folders ```typescript // Prepare entities for upload const folder1 = wrapFileOrFolder('/path/to/folder1'); const folder2 = wrapFileOrFolder('/path/to/folder2'); const file1 = wrapFileOrFolder('/path/to/file1.txt'); // Upload everything in one operation const bulkUpload = await arDrive.uploadAllEntities({ entitiesToUpload: [ // Public folder { wrappedEntity: folder1, destFolderId: rootFolderId }, // Private folder { wrappedEntity: folder2, destFolderId: rootFolderId, driveKey: privateDriveKey }, // Public file { wrappedEntity: file1, destFolderId: someFolderId } ], conflictResolution: 'upsert' }); // Results include all created entities console.log('Created folders:', bulkUpload.created.length); console.log('Total cost:', bulkUpload.totalCost.toString()); ``` #### Create Folder and Upload Contents ```typescript // Create folder and upload all children const folderWithContents = await arDrive.createPublicFolderAndUploadChildren({ parentFolderId, wrappedFolder: wrapFileOrFolder('/path/to/folder'), conflictResolution: 'skip' }); ``` # Conflict Resolution (/sdks/ardrive-core-js/(api-reference)/conflict-resolution) Available strategies when uploading files/folders that already exist: ```typescript // Skip existing files await arDrive.uploadAllEntities({ entitiesToUpload: [...], conflictResolution: 'skip' }); // Replace all existing files await arDrive.uploadAllEntities({ entitiesToUpload: [...], conflictResolution: 'replace' }); // Update only if content differs (default) await arDrive.uploadAllEntities({ entitiesToUpload: [...], conflictResolution: 'upsert' }); // Rename conflicting files await arDrive.uploadAllEntities({ entitiesToUpload: [...], conflictResolution: 'rename' }); // Throw error on conflicts await arDrive.uploadAllEntities({ entitiesToUpload: [...], conflictResolution: 'error' }); // Interactive prompt (CLI only) await arDrive.uploadAllEntities({ entitiesToUpload: [...], conflictResolution: 'ask' }); ``` # Custom Metadata (/sdks/ardrive-core-js/(api-reference)/custom-metadata) Attach custom metadata to files: ```typescript const fileWithMetadata = wrapFileOrFolder('/path/to/file.txt', 'text/plain', { metaDataJson: { 'Custom-Field': 'Custom Value', Version: '1.0' }, metaDataGqlTags: { 'App-Name': ['MyApp'], 'App-Version': ['1.0.0'] }, dataGqlTags: { 'Content-Type': ['text/plain'] } }); // Upload with custom metadata await arDrive.uploadPublicFile({ parentFolderId, wrappedFile: fileWithMetadata }); ``` # Download Operations (/sdks/ardrive-core-js/(api-reference)/download-operations) #### Download Files ```typescript // Download public file const publicData = await arDrive.downloadPublicFile({ fileId }); // publicData is a Buffer/Uint8Array // Download private file (automatically decrypted) const privateData = await arDrive.downloadPrivateFile({ fileId, driveKey }); ``` #### Download Folders ```typescript // Download entire folder const folderData = await arDrive.downloadPublicFolder({ folderId, destFolderPath: '/local/download/path' }); // Download private folder const privateFolderData = await arDrive.downloadPrivateFolder({ folderId, driveKey, destFolderPath: '/local/download/path' }); ``` # Drive Operations (/sdks/ardrive-core-js/(api-reference)/drive-operations) #### Creating Drives ```typescript // Public drive const publicDrive = await arDrive.createPublicDrive({ driveName: 'My Public Drive' }); // Private drive with password const privateDrive = await arDrive.createPrivateDrive({ driveName: 'My Private Drive', drivePassword: 'mySecretPassword' }); ``` #### Reading Drive Information ```typescript // Get public drive const publicDriveInfo = await arDrive.getPublicDrive({ driveId }); // Get private drive (requires drive key) const privateDriveInfo = await arDrive.getPrivateDrive({ driveId, driveKey }); // Get all drives for an address const allDrives = await arDrive.getAllDrivesForAddress({ address: walletAddress, privateKeyData: wallet.getPrivateKey() }); ``` #### Renaming Drives ```typescript // Rename public drive await arDrive.renamePublicDrive({ driveId, newName: 'Updated Drive Name' }); // Rename private drive await arDrive.renamePrivateDrive({ driveId, driveKey, newName: 'Updated Private Name' }); ``` # Encryption & Security (/sdks/ardrive-core-js/(api-reference)/encryption-security) #### Key Derivation ```typescript // Derive drive key from password const driveKey = await deriveDriveKey('myPassword', driveId.toString(), JSON.stringify(wallet.getPrivateKey())); // File keys are automatically derived from drive keys const fileKey = await deriveFileKey(driveKey, fileId); ``` #### Manual Encryption/Decryption ```typescript // Encrypt data const { cipher, cipherIV } = await driveEncrypt(driveKey, data); // Decrypt data const decrypted = await driveDecrypt(cipherIV, driveKey, cipher); ``` # File Operations (/sdks/ardrive-core-js/(api-reference)/file-operations) #### Uploading Files ```typescript // Wrap file for upload const wrappedFile = wrapFileOrFolder('/path/to/file.pdf'); // Upload public file const publicUpload = await arDrive.uploadPublicFile({ parentFolderId, wrappedFile, conflictResolution: 'upsert' // skip, replace, upsert, or error }); // Upload private file const privateUpload = await arDrive.uploadPrivateFile({ parentFolderId, driveKey, wrappedFile }); ``` #### Reading File Information ```typescript // Get public file metadata const publicFile = await arDrive.getPublicFile({ fileId }); // Get private file metadata const privateFile = await arDrive.getPrivateFile({ fileId, driveKey }); ``` #### Moving and Renaming Files ```typescript // Move file await arDrive.movePublicFile({ fileId, newParentFolderId }); // Rename file await arDrive.renamePublicFile({ fileId, newName: 'renamed-file.pdf' }); ``` # Folder Operations (/sdks/ardrive-core-js/(api-reference)/folder-operations) #### Creating Folders ```typescript // Public folder const publicFolder = await arDrive.createPublicFolder({ folderName: 'Documents', driveId, parentFolderId }); // Private folder const privateFolder = await arDrive.createPrivateFolder({ folderName: 'Secret Documents', driveId, driveKey, parentFolderId }); ``` #### Listing Folder Contents ```typescript // List public folder const publicContents = await arDrive.listPublicFolder({ folderId, maxDepth: 2, // Optional: limit recursion depth includeRoot: true // Optional: include root folder in results }); // List private folder const privateContents = await arDrive.listPrivateFolder({ folderId, driveKey, maxDepth: 1 }); ``` #### Moving and Renaming Folders ```typescript // Move folder await arDrive.movePublicFolder({ folderId, newParentFolderId }); // Rename folder await arDrive.renamePublicFolder({ folderId, newName: 'New Folder Name' }); ``` # Pricing & Cost Estimation (/sdks/ardrive-core-js/(api-reference)/pricing-cost-estimation) ```typescript // Get price estimator const priceEstimator = arDrive.getArDataPriceEstimator(); // Estimate cost for data size const cost = await priceEstimator.getARPriceForByteCount( new ByteCount(1024 * 1024) // 1MB ); // Get base Winston price (without tips) const basePrice = await priceEstimator.getBaseWinstonPriceForByteCount( new ByteCount(5 * 1024 * 1024) // 5MB ); ``` # Entity IDs (/sdks/ardrive-core-js/(core-concepts)/entity-ids) Use the type-safe entity ID constructors: ```typescript // Generic entity ID const entityId = EID('10108b54a-eb5e-4134-8ae2-a3946a428ec7'); // Specific entity IDs const driveId = new DriveID('12345674a-eb5e-4134-8ae2-a3946a428ec7'); const folderId = new FolderID('47162534a-eb5e-4134-8ae2-a3946a428ec7'); const fileId = new FileID('98765432a-eb5e-4134-8ae2-a3946a428ec7'); ``` # Entity Types (/sdks/ardrive-core-js/(core-concepts)/entity-types) ArDrive uses a hierarchical structure: - **Drives**: Top-level containers (public or private) - **Folders**: Organize files within drives - **Files**: Individual files stored on Arweave Each entity has a unique ID (`DriveID`, `FolderID`, `FileID`) and can be either public (unencrypted) or private (encrypted). # Wallet Management (/sdks/ardrive-core-js/(core-concepts)/wallet-management) ```typescript // Create wallet from JWK const wallet = new JWKWallet(jwkKey); // Check wallet balance const balance = await wallet.getBalance(); ``` # ArDrive Core JS (/sdks/ardrive-core-js) **For AI and LLM users**: Access the complete ArDrive Core JS documentation in plain text format at llm.txt for easy consumption by AI agents and language models. The ArDrive Core JS SDK provides a comprehensive TypeScript library for building applications on ArDrive. It offers type-safe interfaces for drive management, file operations, encryption, and seamless integration with Arweave. ## Quick Start ### Install the SDK ```npm npm install ardrive-core-js ``` ### Initialize with a Wallet ```typescript // Load your Arweave wallet const wallet = readJWKFile('./wallet.json'); // Create an ArDrive instance const arDrive = arDriveFactory({ wallet }); ``` ### Create a Drive and Upload Files ```typescript // Create a new public drive const { driveId, rootFolderId } = await arDrive.createPublicDrive({ driveName: 'My-Drive' }); console.log('Drive created:', driveId.toString()); console.log('Root folder:', rootFolderId.toString()); // Upload a file to the drive const wrappedFile = wrapFileOrFolder('./my-file.pdf'); const uploadResult = await arDrive.uploadPublicFile({ parentFolderId: rootFolderId, wrappedFile }); console.log('File uploaded:', uploadResult.fileId.toString()); ``` ### Install the SDK ```npm npm install ardrive-core-js ``` ### Configure Polyfills Polyfills are required for web environments due to Node.js dependencies used by the SDK. ```npm npm install --save-dev vite-plugin-node-polyfills ``` ```js // vite.config.js plugins: [ nodePolyfills({ globals: { Buffer: true, global: true, process: true, }, }), ], }); ``` Configure your bundler (Webpack, Vite, Rollup, etc.) to provide polyfills for `crypto`, `process`, and `buffer`. Refer to your bundler's documentation for polyfill configuration. ### Use the SDK ```typescript // Initialize with a JWK wallet object const arDrive = arDriveFactory({ wallet: jwkWallet }); // Create a public drive const { driveId, rootFolderId } = await arDrive.createPublicDrive({ driveName: 'My-Drive' }); console.log('Drive created:', driveId.toString()); ``` ## Documentation } title="Source Code" description="View the complete source code and contribute on GitHub" href="https://github.com/ardriveapp/ardrive-core-js" /> } title="API Reference" description="Detailed documentation for all SDK methods and classes" href="/sdks/ardrive-core-js/drive-operations" /> ## Core Features } title="Drive Operations" description="Create and manage public and private drives" href="/sdks/ardrive-core-js/drive-operations" /> } title="Folder Operations" description="Create folders, list contents, and organize your data" href="/sdks/ardrive-core-js/folder-operations" /> } title="File Operations" description="Upload, download, and manage files on Arweave" href="/sdks/ardrive-core-js/file-operations" /> } title="Encryption & Security" description="End-to-end encryption for private drives and files" href="/sdks/ardrive-core-js/encryption-security" /> } title="Pricing & Cost Estimation" description="Estimate upload costs before committing transactions" href="/sdks/ardrive-core-js/pricing-cost-estimation" /> } title="Advanced Features" description="Turbo integration, bundling, manifests, and more" href="/sdks/ardrive-core-js/turbo-integration" /> # Introduction (/sdks) Build powerful applications with our comprehensive suite of SDKs designed for the ar.io ecosystem. } title="Upload data with the Turbo SDK" description="TypeScript SDK for Turbo, an ar.io gateway run by the ArDrive team" href="/sdks/turbo-sdk" /> } title="Interact with ar.io using the ar.io SDK" description="Access ar.io protocols, manage ArNS names, interact with ANTs, and integrate gateway services" href="/sdks/ar-io-sdk" /> } title="Decentralized access with Wayfinder SDK" description="Robust, censorship-resistant access to Arweave data through the distributed ar.io gateway network" href="/sdks/wayfinder" /> ## Choose Your SDK Each SDK serves a specific purpose in the ar.io ecosystem: - **Turbo SDK** - The client for Turbo, an ar.io gateway run by the ArDrive team - **turbo-upload** - The same uploads with no dependencies, for adding Turbo to a server that already exists - **ar.io SDK** - For ArNS names, ANT contracts and gateway data on the ar.io network - **Wayfinder SDK** - For decentralized data access with built-in verification and gateway routing All of them ship TypeScript types. Every one runs in both Node.js and the browser except `turbo-upload`, which is server-side only: it signs with a raw Arweave key, and that does not belong in a browser. ## Next Steps # TurboAuthenticatedClient (/sdks/turbo-sdk/(apis)/turboauthenticatedclient) #### getBalance() Issues a signed request to get the credit balance of a wallet measured in AR (measured in Winston Credits, or winc). ```typescript const { winc: balance } = await turbo.getBalance(); ``` #### getFreeStatus() Returns the wallet's remaining free-tier upload allowance in bytes as `{ bytesRemaining }`, so you can tell up front whether an upload will be free. `bytesRemaining` is `null` for a wallet with an unlimited allowance (an exempt/partner wallet), and `0` when the free tier is disabled on the target Turbo deployment. It is advisory — the authoritative free/charge decision is made at upload time — and is a wallet-side figure (a per-network cap may also apply). Deployment-wide free-tier limits are on the service's `/info` endpoint. ```typescript const { bytesRemaining } = await turbo.getFreeStatus(); ``` It is also available on the `TurboUnauthenticatedClient` for any wallet by address: ```typescript const { bytesRemaining } = await turbo.getFreeStatus('a-native-address'); ``` #### getPaymentHistory() Issues a signed request for the signing wallet's own completed top-up (payment) history — both cryptocurrency and fiat top-ups — merged newest-first and keyset-paginated. This is self-scoped: it returns **only** the signing wallet's rows (the service reads the address from the signature, never a query parameter), so it is available on the `TurboAuthenticatedClient` only. `limit` is the page size (1-100, default 50). To page, pass the previous response's `cursor` while `hasMore` is `true`. Each item is discriminated by `type`: a `'crypto'` item includes `wincCredited`, `tokenType`, `tokenQuantity`, `usdEquivalent`, `senderAddress`, `transactionId`, and `blockHeight`; a `'fiat'` item includes `wincCredited`, `paymentAmount`, `currencyType`, `paymentProvider`, `receiptId`, and `giftMessage`. Every item has an ISO-8601 UTC `date`. ```typescript const { payments, hasMore, cursor } = await turbo.getPaymentHistory({ limit: 25, }); // Fetch the next page while more results remain if (hasMore) { const next = await turbo.getPaymentHistory({ limit: 25, cursor }); } ``` #### signer.getNativeAddress() Returns the [native address][docs/native-address] of the connected signer. ```typescript const address = await turbo.signer.getNativeAddress(); ``` #### getWincForFiat() Returns the current amount of Winston Credits including all adjustments for the provided fiat currency, amount, and optional promo codes. ```typescript const { winc, paymentAmount, quotedPaymentAmount, adjustments } = await turbo.getWincForFiat({ amount: USD(100), promoCodes: ['MY_PROMO_CODE'], // promo codes require an authenticated client }); ``` #### createCheckoutSession() Creates a Stripe checkout session for a Turbo Top Up with the provided amount, currency, owner, and optional promo codes. The returned URL can be opened in the browser, all payments are processed by Stripe. Promo codes require an authenticated client. ```typescript const { url, winc, paymentAmount, quotedPaymentAmount, adjustments } = await turbo.createCheckoutSession({ amount: USD(10.0), // $10.00 USD owner: publicArweaveAddress, promoCodes: ['MY_PROMO_CODE'], // promo codes require an authenticated client }); // open checkout session in a browser window.open(url, '_blank'); ``` #### upload() The easiest way to upload data to Turbo. The `signal` is an optional [AbortSignal] that can be used to cancel the upload or timeout the request. `dataItemOpts` is an optional object that can be used to configure tags, target, and anchor for the data item upload. ```typescript const uploadResult = await turbo.upload({ data: 'The contents of my file!', signal: AbortSignal.timeout(10_000), // cancel the upload after 10 seconds dataItemOpts: { // optional }, events: { // optional }, }); ``` #### uploadFile() Signs and uploads a raw file. There are two ways to provide the file to the SDK: 1. Using a `file` parameter 2. Using a `fileStreamFactory` and `fileSizeFactory` ##### Using file` In Web with a file input: ```typescript const selectedFile = e.target.files[0]; const uploadResult = await turbo.uploadFile({ file: selectedFile, dataItemOpts: { tags: [{ name: 'Content-Type', value: 'text/plain' }], }, events: { onUploadProgress: ({ totalBytes, processedBytes }) => { console.log('Upload progress:', { totalBytes, processedBytes }); }, onUploadError: (error) => { console.log('Upload error:', { error }); }, onUploadSuccess: () => { console.log('Upload success!'); }, }, }); ``` In NodeJS with a file path: ```typescript const filePath = path.join(__dirname, './my-unsigned-file.txt'); const fileSize = fs.stateSync(filePath).size; const uploadResult = await turbo.uploadFile({ file: filePath, dataItemOpts: { tags: [{ name: 'Content-Type', value: 'text/plain' }], }, }); ``` ##### Using fileStreamFactory` and `fileSizeFactory` Note: The provided `fileStreamFactory` should produce a NEW file data stream each time it is invoked. The SDK calls it again for every retry, and a stream can be read only once. In the browser, pass `() => file.stream()` rather than a stream you created ahead of time; a factory that returns one stream instance fails on the first retry. The `fileSizeFactory` is a function that returns the size of the file. The `signal` is an optional [AbortSignal] that can be used to cancel the upload or timeout the request. `dataItemOpts` is an optional object that can be used to configure tags, target, and anchor for the data item upload. ```typescript const filePath = path.join(__dirname, './my-unsigned-file.txt'); const fileSize = fs.stateSync(filePath).size; const uploadResult = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream(filePath), fileSizeFactory: () => fileSize, }); ``` ##### Customize Multi-Part Upload Behavior By default, the Turbo upload methods will split files that are larger than 10 MiB into chunks and send them to the upload service multi-part endpoints. This behavior can be customized with the following inputs: - `chunkByteCount`: The maximum size in bytes for each chunk. Must be between 5 MiB and 500 MiB. Defaults to 5 MiB. - `maxChunkConcurrency`: The maximum number of chunks to upload concurrently. Defaults to 5. Reducing concurrency will slow down uploads, but reduce memory utilization and serialize network calls. Increasing it will upload faster, but can strain available resources. - `chunkingMode`: The chunking mode to use. Can be 'auto', 'force', or 'disabled'. Defaults to 'auto'. Auto behavior means chunking is enabled if the file would be split into at least three chunks. - `maxFinalizeMs`: The maximum time in milliseconds to wait for the finalization of all chunks after the last chunk is uploaded. Defaults to 1 minute per GiB of the total file size. ```typescript // Customize chunking behavior await turbo.upload({ ...params, chunkByteCount: 1024 * 1024 * 500, // Max chunk size maxChunkConcurrency: 1, // Minimize concurrency }); ``` ```typescript // Disable chunking behavior await turbo.upload({ ...params, chunkingMode: 'disabled', }); ``` ```typescript // Force chunking behavior await turbo.upload({ ...params, chunkingMode: 'force', }); ``` #### On Demand Uploads With the upload methods, you can choose to Top Up with selected crypto token on demand if the connected wallet does not have enough credits to complete the upload. This is done by providing the `OnDemandFunding` class to the `fundingMode` parameter on upload methods. The `maxTokenAmount` (optional) is the maximum amount of tokens in the token type's smallest unit value (e.g: Winston for arweave token type) to fund the wallet with. The `topUpBufferMultiplier` (optional) is the multiplier to apply to the estimated top-up amount to avoid underpayment during on-demand top-ups due to price fluctuations on longer uploads. Defaults to 1.1, meaning a 10% buffer. Note: On demand API currently only available for $ARIO (`ario`), $SOL (`solana`), $ETH on Base Network (`base-eth`), $USDC on Base Network (`base-usdc`) and $USDC on Solana (`solana-usdc`) token types. ```typescript const turbo = TurboFactory.authenticated({ privateKey: bs58.encode(secretKey), // a Solana key holding $ARIO token: 'ario', }); await turbo.upload({ ...params, fundingMode: new OnDemandFunding({ maxTokenAmount: ARIOToTokenAmount(500), // Max 500 $ARIO topUpBufferMultiplier: 1.1, // 10% buffer to avoid underpayment }), }); ``` #### x402 Uploads Another method of uploading files is via the x402 protocol. This method is optimized for agent workflows and allows for direct uploads to Arweave gateways that support the x402 protocol using an EVM wallet and base-usdc token type. > **Note:** x402 uploads need the optional peer dependency `x402-fetch`, which keeps its dependency tree out of installs that pay with credits: > > ```shell > npm install x402-fetch > ``` > > Without it, credit-paid uploads and the x402 price routes work as usual, and an x402 upload fails before signing with a message naming this install. Browser consumers of the prebuilt bundle resolve `x402-fetch` themselves, through their bundler or an import map, because the bundle leaves it external. ```typescript const turbo = TurboFactory.authenticated({ signer: ethereumSignerWithBaseUSDC, token: 'base-usdc', }); await turbo.uploadFile({ ...params, fundingMode: new X402Funding({ maxMUSDCAmount: 1_000_000 }), // Max 1 USDC. Opt out if too expensive }); ``` Large items are uploaded in chunks and paid for when the upload is created, so the payload is never sent just to discover its price. Smaller items go in a single request, which is buffered in memory so its length can be declared — the service prices an x402 upload from `Content-Length`, and a streamed body has none. The service URL must be HTTPS: an x402 authorization is a bearer credential, so the SDK refuses to send one over cleartext. Loopback is the exception — `localhost`, `127.0.0.1` and `::1` are allowed over plain HTTP, so local development against a bundler on your own machine still works. #### Pricing an x402 Upload Before Sending It `getX402PriceForDataItem` prices a signed data item from its byte count, so you can learn the cost without transmitting the payload. Without it the only way to get a price is to POST the data and read the 402 challenge. ```typescript const turbo = TurboFactory.unauthenticated({ token: 'base-usdc' }); const quote = await turbo.getX402PriceForDataItem({ byteCount: signedDataItemByteCount, // the SIGNED item, not the payload inside it }); console.log(quote.usdcAmount); // amount to pay, in USDC's smallest unit ``` `getX402PriceForRawData` prices raw data that Turbo will wrap into a data item itself, and reports the wrapping overhead — a data item is larger than its payload by its header, signature and tags, which a caller cannot compute. ```typescript const quote = await turbo.getX402PriceForRawData({ byteCount: myRawData.byteLength, tagCount: 3, // tags you intend to attach; they change the overhead contentType: 'image/png', }); console.log(quote.overhead, quote.estimatedDataItemSize); ``` Both take an optional `network`, defaulting to `base`. **This is the x402 network, not the SDK token type**: the route builds its token as `usdc-{network}`, so `base-usdc` is accepted on mainnet only because the network there is literally `base`. Against a testnet service, pass `network: 'base-sepolia'`. #### Raw x402 Data Uploads Using the x402 protocol, you can also upload raw data to Turbo without signing a data item. This method is ideal for quick agent workflows where the ownership of the data is not required to be tied to a specific wallet. The eventual data item on chain will be signed by Turbo's x402 EVM signer. ```typescript const turbo = TurboFactory.authenticated({ signer: ethereumSignerWithBaseUSDC, token: 'base-usdc', }); await turbo.uploadRawX402Data({ data: myRawData, maxMUSDCAmount: 1_000_000, // Max 1 USDC. Opt out if too expensive }); ``` NOTE: For free uploads under 105 KiB, this method of upload currently does not require a signature and can be used with an unauthenticated client. ```ts // Unsigned free upload of raw data under 105 KiB const turbo = TurboFactory.unauthenticated({ token: 'base-usdc' }); await turbo.uploadRawX402Data({ data: myRawData, }); ``` #### uploadFolder() Signs and uploads a folder of files. For NodeJS, the `folderPath` of the folder to upload is required. For the browser, an array of `files` is required. The `dataItemOpts` is an optional object that can be used to configure tags, target, and anchor for the data item upload. The `signal` is an optional [AbortSignal] that can be used to cancel the upload or timeout the request. The `maxConcurrentUploads` is an optional number that can be used to limit the number of concurrent uploads. The `throwOnFailure` is an optional boolean that can be used to throw an error if any upload fails. The `manifestOptions` is an optional object that can be used to configure the manifest file, including a custom index file, fallback file, or whether to disable manifests altogether. Manifests are enabled by default. The `folderIndex` is an optional [folder index](#incremental-folder-uploads) that skips files already on Arweave. The manifestDataItemOpts` is an optional object that configures the manifest data item only, and defaults to `dataItemOpts`. ##### NodeJS Upload Folder ```typescript const folderPath = path.join(__dirname, './my-folder'); const { manifest, fileResponses, manifestResponse } = await turbo.uploadFolder({ folderPath, dataItemOpts: { // optional tags: [ { // User defined content type will overwrite file content type name: 'Content-Type', value: 'text/plain', }, { name: 'My-Custom-Tag', value: 'my-custom-value', }, ], // no timeout or AbortSignal provided }, manifestOptions: { // optional indexFile: 'custom-index.html', fallbackFile: 'custom-fallback.html', disableManifests: false, }, }); ``` ##### Browser Upload Folder ```html const folderInput = document.getElementById('folder'); folderInput.addEventListener('change', async (event) => { const selectedFiles = folderInput.files; console.log('Folder selected:', selectedFiles); const { manifest, fileResponses, manifestResponse } = await turbo.uploadFolder({ files: Array.from(selectedFiles).map((file) => file), }); console.log(manifest, fileResponses, manifestResponse); }); ``` ##### Upload Folder with Progress Events The `uploadFolder` method supports folder-level and per-file events for tracking upload progress. This is useful for building progress bars or providing feedback to users during folder uploads. ```typescript const folderPath = path.join(__dirname, './my-folder'); const { manifest, fileResponses, manifestResponse } = await turbo.uploadFolder({ folderPath, events: { // Per-file events onFileStart: ({ fileName, fileSize, fileIndex, totalFiles }) => { console.log( `Starting file ${ fileIndex + 1 }/${totalFiles}: ${fileName} (${fileSize} bytes)`, ); }, onFileProgress: ({ fileName, fileIndex, totalFiles, fileProcessedBytes, fileTotalBytes, step, }) => { const percentComplete = (fileProcessedBytes / fileTotalBytes) * 100; console.log( `File ${ fileIndex + 1 }/${totalFiles} (${fileName}) ${step}: ${percentComplete.toFixed(2)}%`, ); }, onFileComplete: ({ fileName, fileIndex, totalFiles, id }) => { console.log( `Completed file ${fileIndex + 1}/${totalFiles}: ${fileName} (${id})`, ); }, onFileError: ({ fileName, fileIndex, totalFiles, error }) => { console.error( `Error uploading file ${fileIndex + 1}/${totalFiles}: ${fileName}`, error, ); }, // Folder-level aggregate events onFolderProgress: ({ processedFiles, totalFiles, processedBytes, totalBytes, currentPhase, }) => { const percentComplete = (processedBytes / totalBytes) * 100; console.log( `Folder progress (${currentPhase}): ${processedFiles}/${totalFiles} files, ${percentComplete.toFixed( 2, )}%`, ); }, onFolderError: (error) => { console.error('Folder upload error:', error); }, onFolderSuccess: () => { console.log('Folder upload complete!'); }, }, }); ``` ##### Incremental Folder Uploads A runnable version of everything below is in [`examples/folder-index`](https://github.com/ardriveapp/turbo-sdk/blob/alpha/examples/folder-index/index.mjs): it deploys the same folder three times and prints what each run uploaded and reused. An Arweave upload is permanent, so paying twice for byte identical files buys nothing. Pass a `folderIndex` and `uploadFolder` hashes every file, asks the index which of those files already have a data item on Arweave, and signs, uploads and pays for only the rest. The manifest is assembled from the ids that were already known plus the ids of whatever this run uploaded. ```typescript import { composeFolderIndex, createChainFolderIndex, createFileFolderIndex, } from '@ardrive/turbo-sdk/node'; const folderIndex = composeFolderIndex([ // Fast local cache, kept outside the folder being uploaded. createFileFolderIndex({ filePath: '.turbo/folder-index.jsonl' }), // Fallback for a machine that has never deployed before, e.g. a CI runner. // getPublicKey() is the one form every signer type can produce. createChainFolderIndex({ owner: await turbo.signer.getPublicKey() }), ]); const { manifest, manifestResponse, folderIndexSummary } = await turbo.uploadFolder({ folderPath: path.join(__dirname, './dist'), folderIndex, // Deploy varying tags belong on the manifest, which is rewritten every time. manifestDataItemOpts: { tags: [{ name: 'Git-Commit', value: process.env.GITHUB_SHA }], }, }); console.log(folderIndexSummary); // { totalFiles: 143, uploadedFiles: 2, reusedFiles: 141, ... } ``` ###### What a reused file is matched on An index key is `\.\`, and both halves matter. Keying on the bytes alone would reuse a data item whose tags are not the ones you asked for: an empty `a.css` and an empty `b.js` hash identically, and sharing one item between them would serve JavaScript as `text/css`, which a browser refuses to execute. Covering the tags means **a reused data item is always exactly the data item this call would otherwise have created** — same bytes, same `Content-Type`, same `dataItemOpts` tags. Files uploaded with an index carry one extra tag, `File-SHA256`, holding the sha-256 of their own bytes. That tag is what `createChainFolderIndex` filters on. ###### The trade-off this buys, and how you find out The corollary is a real cost cliff, so it is worth being blunt about. **A per file tag whose value changes between deploys changes every key, and re-uploads the whole folder at full price.** A commit sha, a build number or a timestamp in `dataItemOpts` means you never reuse anything, and the deploy still succeeds, so nothing about the run looks wrong except the bill. That is deliberate. The alternative — keying on bytes alone — reuses an item tagged with a _previous_ deploy's commit sha, so the tags on chain quietly stop describing what is on chain. A wrong bill is recoverable; a data item that lies about itself is permanent. So the index errs towards paying again. To keep the cliff from being silent, `uploadFolder` logs a warning when a file it is about to upload has bytes the index already holds **under a different set of tags**, which is what a deploy-varying per file tag looks like: ``` 3 of the 3 file(s) this run is about to upload are already on Arweave byte for byte, under a different set of tags. Their content has not changed but their tags have, so they are being paid for again. A folder index key covers the tags on a file as well as its bytes. That is usually a tag in dataItemOpts whose value changes between deploys -- a commit sha, a build number, a timestamp -- in which case move it to manifestDataItemOpts rather than paying for these files again. It can also be a file that kept its content but changed its Content-Type, through a rename or a new extension, which is expected and costs one upload. ``` Very little else produces that signal: a folder the index has never seen has unknown bytes, and a layer that could not be reached reports nothing known, so neither triggers it. A file that kept its content but changed its Content-Type through a rename does trigger it, and the message says so. It also fires for one drifted file among a hundred reused ones, not only when everything misses. A layer that does not implement the optional `knownContentHashes` cannot answer the question and stays quiet. The fix, whenever it is a varying tag, is always the same: move it to `manifestDataItemOpts`, since the manifest is rewritten on every deploy anyway. ###### Index layers | Layer | Where it lives | Survives a fresh checkout | | -------------------------------------------------- | ---------------- | ------------------------- | | `createMemoryFolderIndex(seed?)` | memory | no | | `createFileFolderIndex({ filePath })` (NodeJS) | a JSON lines log | only if the file is kept | | `createChainFolderIndex({ owner, appName?, ... })` | gateway GraphQL | yes | | `composeFolderIndex([...])` | layers the above | -- | Reads fall through a composed index in order and writes go to every layer that is not `readOnly`, so an id recovered from the gateway is cached locally for the next run. **A layer that throws is skipped, not propagated** — a full disk under the file layer must not stop the memory layer from holding ids the run has already paid for, and an unreachable gateway must not stop the local cache from answering. Pass a `logger` as the second argument to `composeFolderIndex` to see which layer was skipped and why. `createFileFolderIndex` writes an append-only log, one JSON record per line, compacted when it is next loaded. It appends after every single upload rather than rewriting at the end of the run, so a deploy killed part way through never loses a file it has paid for — and appending is constant work per file, where rewriting the whole file per upload is quadratic and costs minutes and gigabytes of writes on a first deploy of a few thousand files. It is also the more crash safe shape: a process killed mid write can only damage the last line, which is dropped on load, where a torn rewrite loses every id in the file. An index is a cache. A `get` or `resolve` that throws is treated as a miss and logged — an unreachable gateway costs you a re-upload, it does not fail your deploy. Anything with `get` and `set` is a valid index, so implement `TurboFolderUploadIndex` to back one with a database, an object store, or a CI cache. Treat the keys as opaque. ###### Telling a gateway whose uploads to sweep `createChainFolderIndex` needs the owner **a gateway indexes uploads under**, which is the base64url sha-256 of the signer's public key. Pass `await turbo.signer.getPublicKey()` and the SDK derives it, which works for every signer type. A bare string is deliberately rejected, because it cannot be disambiguated: a raw 32 byte ed25519 public key base64urls to exactly 43 characters, the same shape as an owner address, and guessing wrong means the sweep matches nothing and the whole folder is re-uploaded with no error at all. Say which one you have — `{ publicKey }` or `{ address }` — if you are not passing the bytes. An `0x...` Ethereum address or a base58 Solana address is not accepted, because `owners:` on a gateway does not match those. (Verified against `arweave.net`: `owners` matches the 43 character address and returns nothing for the raw public key, so the conversion has to happen client side.) ###### Trust model The sweep is scoped to `owners: [your own address]`, so it can only ever find items you signed. Within that scope, `File-SHA256` is **self asserted** — it is a tag your own past uploads wrote, not something a gateway verifies against the bytes — and the index trusts it. That is safe for uploads this SDK made, since it only ever writes a hash it computed from the file in front of it. `uploadFolder` writes whichever tag the index it is given declares, so setting `hashTagName` moves both the tag that is written and the tag the sweep filters on, and the two cannot drift apart. Every layer in a `composeFolderIndex` stack that declares one has to declare the same one, or the call throws: one tag is written per file, so a stack that disagrees would leave whichever layer lost matching nothing, for ever, without an error. It stops being safe if you point `hashTagName` at a tag you were already using for something else. Any of your own past items carrying 64 hex characters under that name would be treated as a candidate, and one whose tag set happens to match would be reused — putting a manifest path in front of unrelated bytes. Use a name nothing else of yours writes. ###### When the sweep runs out of pages A sweep can examine at most `pageSize * maxPages` items, 2,000 by default. A folder with more files than that, or a long enough deployment history, can therefore reach the page limit with files still unresolved — and those files are uploaded and paid for again while the summary reports them as ordinary new files. Pass a `logger` to `createChainFolderIndex` and it says so when this happens, naming how many files were left. Raise `maxPages` or `pageSize`, or put a `createFileFolderIndex` in front, and the sweep has less to find. ###### In the browser `createMemoryFolderIndex`, `createChainFolderIndex` and `composeFolderIndex` all work in the browser. `createFileFolderIndex` is NodeJS only, since there is no filesystem to write to; persist the map yourself and seed `createMemoryFolderIndex` with it, or rely on the chain index. Note that hashing differs by platform. NodeJS streams each file through a `node:crypto` digest, so file size is not a concern. The browser has no streaming WebCrypto digest, so each `File` is buffered whole before it is hashed — a very large `File` can exhaust the tab. ###### Known limitation A gateway indexes an upload minutes after it lands, so two machines deploying the same brand new file at the same moment can each pay for it once. Only the bill is affected, and only for genuinely new bytes -- the manifest is correct either way. #### topUpWithTokens() Tops up the connected wallet with Credits by submitting a payment transaction for the token amount to the Turbo wallet and then submitting that transaction id to Turbo Payment Service for top up processing. - The `tokenAmount` is the amount of tokens in the token type's smallest unit value (e.g: Winston for arweave token type) to fund the wallet with. - The `feeMultiplier` (optional) is the multiplier to apply to the reward for the transaction to modify its chances of being mined. Credits will be added to the wallet balance after the transaction is confirmed on the given blockchain. Defaults to 1.0, meaning no multiplier. - The `turboCreditDestinationAddress` (optional) is the native address to credit the funds to. If not provided, the connected wallet's native address will be used. ##### Arweave (AR) Crypto Top Up ```typescript const turbo = TurboFactory.authenticated({ signer, token: 'arweave' }); const { winc, status, id, ...fundResult } = await turbo.topUpWithTokens({ tokenAmount: WinstonToTokenAmount(100_000_000), // 0.0001 AR feeMultiplier: 1.1, // 10% increase in reward for improved mining chances turboCreditDestinationAddress: '0xabc...123', // Any custom EVM / SOL / AR native destination address }); ``` ##### AR.IO Network (ARIO) Crypto Top Up $ARIO is an SPL token on Solana, so pay with a Solana key. Without a `turboCreditDestinationAddress`, the credits go to the account of that key's base58 public key: the same account `getBalance()` reads and uploads signed by that key pay from. ```typescript const turbo = TurboFactory.authenticated({ privateKey: bs58.encode(secretKey), token: 'ario', }); const { winc, status, id, ...fundResult } = await turbo.topUpWithTokens({ tokenAmount: ARIOToTokenAmount(100), // 100 $ARIO }); ``` ##### USDC Crypto Top Up ```typescript // USDC on Ethereum Mainnet const { winc, status, id, ...fundResult } = await TurboFactory.authenticated({ signer, token: 'usdc', }).topUpWithTokens({ tokenAmount: USDCToTokenAmount(1), // 1 USDC }); // USDC on Base Network const { winc, status, id, ...fundResult } = await TurboFactory.authenticated({ signer, token: 'base-usdc', }).topUpWithTokens({ tokenAmount: USDCToTokenAmount(1), // 1 USDC }); // USDC on Polygon Network const { winc, status, id, ...fundResult } = await TurboFactory.authenticated({ signer, token: 'polygon-usdc', }).topUpWithTokens({ tokenAmount: USDCToTokenAmount(1), // 1 USDC }); // USDC (SPL) on Solana — signed with a SOLANA wallet, not an EVM one. The // payment is an SPL transfer into Turbo's associated token account, so the // payer just needs USDC plus a little SOL for fees. const { winc, status, id, ...fundResult } = await TurboFactory.authenticated({ privateKey: bs58.encode(secretKey), token: 'solana-usdc', }).topUpWithTokens({ tokenAmount: USDCToTokenAmount(1), // 1 USDC }); ``` The mint for `solana-usdc` is chosen from the RPC you point at: a `gatewayUrl` whose host contains `devnet` uses Circle's devnet USDC, otherwise mainnet USDC. That is a heuristic — if you use a **private or paid devnet RPC** whose hostname does not say `devnet`, name the mint explicitly, or you will sign a transfer of the wrong mint: ```typescript const gatewayUrl = 'https://my-rpc.example.com/\'; const turbo = TurboFactory.authenticated({ privateKey: bs58.encode(secretKey), token: 'solana-usdc', gatewayUrl, tokenTools: new SolanaUsdcToken({ gatewayUrl, mintAddress: '4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU', // devnet USDC }), }); ``` ##### Ethereum (ETH) Crypto Top Up ```typescript const turbo = TurboFactory.authenticated({ signer, token: 'ethereum' }); const { winc, status, id, ...fundResult } = await turbo.topUpWithTokens({ tokenAmount: ETHToTokenAmount(0.00001), // 0.00001 ETH }); ``` ##### Polygon (POL / MATIC) Crypto Top Up ```typescript const turbo = TurboFactory.authenticated({ signer, token: 'pol' }); const { winc, status, id, ...fundResult } = await turbo.topUpWithTokens({ tokenAmount: POLToTokenAmount(0.00001), // 0.00001 POL }); ``` ##### Eth on Base Network Crypto Top Up ```typescript const turbo = TurboFactory.authenticated({ signer, token: 'base-eth' }); const { winc, status, id, ...fundResult } = await turbo.topUpWithTokens({ tokenAmount: ETHToTokenAmount(0.00001), // 0.00001 ETH bridged on Base Network }); ``` ##### Solana (SOL) Crypto Top Up ```typescript const turbo = TurboFactory.authenticated({ signer, token: 'solana' }); const { winc, status, id, ...fundResult } = await turbo.topUpWithTokens({ tokenAmount: SOLToTokenAmount(0.00001), // 0.00001 SOL }); ``` #### shareCredits() Shares credits from the connected wallet to the provided native address and approved winc amount. This action will create a signed data item for the approval ```typescript const { approvalDataItemId, approvedWincAmount } = await turbo.shareCredits({ approvedAddress: '2cor...VUa', approvedWincAmount: 800_000_000_000, // 0.8 Credits expiresBySeconds: 3600, // Credits will expire back to original wallet in 1 hour }); ``` #### revokeCredits() Revokes all credits shared from the connected wallet to the provided native address. ```typescript const revokedApprovals = await turbo.revokeCredits({ revokedAddress: '2cor...VUa', }); ``` #### getCreditShareApprovals() Returns all given or received credit share approvals for the connected wallet or the provided native address. ```typescript const { givenApprovals, receivedApprovals } = await turbo.getCreditShareApprovals({ userAddress: '2cor...VUa', }); ``` # TurboFactory (/sdks/turbo-sdk/(apis)/turbofactory) #### unauthenticated() Creates an instance of a client that accesses Turbo's unauthenticated services. ```typescript const turbo = TurboFactory.unauthenticated(); ``` #### authenticated() Creates an instance of a client that accesses Turbo's authenticated and unauthenticated services. Requires either a signer, or private key to be provided. See the [Signers] section for all supported signers and authentication methods. ```typescript const signer = new ArweaveSigner(jwk); const turbo = TurboFactory.authenticated({ signer }); ``` #### Testnet Configuration For development and testing, you can configure the SDK to use blockchain testnets. This allows you to test your integration with free testnet tokens without spending real cryptocurrency. **Important**: The SDK defaults to mainnet. You must explicitly set the `gatewayUrl` parameter to use a testnet. ```typescript // Base Sepolia (recommended for testing) const turbo = TurboFactory.authenticated({ privateKey: process.env.BASE_SEPOLIA_PRIVATE_KEY, token: 'base-eth', gatewayUrl: 'https://sepolia.base.org', // Required for testnet paymentServiceConfig: { url: 'https://payment.services.ar-io.dev', // ar.io testnet sandbox }, uploadServiceConfig: { url: 'https://upload.services.ar-io.dev', // ar.io testnet sandbox } }); // Solana Devnet const turbo = TurboFactory.authenticated({ privateKey: bs58.encode(secretKey), token: 'solana', gatewayUrl: 'https://api.devnet.solana.com', paymentServiceConfig: { url: 'https://payment.services.ar-io.dev', }, uploadServiceConfig: { url: 'https://upload.services.ar-io.dev', } }); // Ethereum Sepolia const turbo = TurboFactory.authenticated({ privateKey: process.env.SEPOLIA_PRIVATE_KEY, token: 'ethereum', gatewayUrl: 'https://sepolia.gateway.tenderly.co', paymentServiceConfig: { url: 'https://payment.services.ar-io.dev', }, uploadServiceConfig: { url: 'https://upload.services.ar-io.dev', }, }); ``` These endpoints are the **ar.io Testnet Sandbox** — the full ar.io stack (upload, payment, ArNS, and gateway) running on testnet, with a faucet so nothing costs real money. Uploaded data is served from the sandbox gateway at `https://ar-io.dev` and is **ephemeral** (purged after ~3 days); it is never posted to mainnet Arweave. See [the ar.io Testnet Sandbox docs](https://docs.ar.io/build/testnet). **Supported Testnets**: - **ARIO staging** (`ario`) - Staging ARIO on Solana devnet; fee-free funding, claim from the [ar.io faucet](https://faucet.services.ar-io.dev) - **Base Sepolia** (`base-eth`) - Supports on-demand funding - **Solana Devnet** (`solana`) - Supports on-demand funding - **Ethereum Sepolia** (`ethereum`) - Manual top-up only - **Polygon Amoy** (`pol`) - Manual top-up only # TurboUnauthenticatedClient (/sdks/turbo-sdk/(apis)/turbounauthenticatedclient) #### getSupportedCurrencies() Returns the list of currencies supported by the Turbo Payment Service for topping up a user balance of AR Credits (measured in Winston Credits, or winc). ```typescript const currencies = await turbo.getSupportedCurrencies(); ``` #### getSupportedCountries() Returns the list of countries supported by the Turbo Payment Service's top up workflow. ```typescript const countries = await turbo.getSupportedCountries(); ``` #### getFiatToAR() Returns the current raw fiat to AR conversion rate for a specific currency as reported by third-party pricing oracles. ```typescript const fiatToAR = await turbo.getFiatToAR({ currency: 'USD' }); ``` #### getFiatRates() Returns the current fiat rates for 1 GiB of data for supported currencies, including all top-up adjustments and fees. ```typescript const rates = await turbo.getFiatRates(); ``` #### getWincForFiat() Returns the current amount of Winston Credits including all adjustments for the provided fiat currency. ```typescript const { winc, actualPaymentAmount, quotedPaymentAmount, adjustments } = await turbo.getWincForFiat({ amount: USD(100), }); ``` #### getWincForToken() Returns the current amount of Winston Credits including all adjustments for the provided token amount. ```typescript const { winc, actualTokenAmount, equivalentWincTokenAmount } = await turbo.getWincForToken({ tokenAmount: WinstonToTokenAmount(100_000_000), }); ``` #### getFiatEstimateForBytes() Get the current price from the Turbo Payment Service, denominated in the specified fiat currency, for uploading a specified number of bytes to Turbo. ```typescript const turbo = TurboFactory.unauthenticated(); const { amount } = await turbo.getFiatEstimateForBytes({ byteCount: 1024 * 1024 * 1024, currency: 'usd', // specify the currency for the price }); console.log(amount); // Estimated usd price for 1 GiB ``` **Output:** ```json { "byteCount": 1073741824, "amount": 20.58, "currency": "usd", "winc": "2402378997310" } ``` #### getTokenPriceForBytes() Get the current price from the Turbo Payment Service, denominated in the specified token, for uploading a specified number of bytes to Turbo. ```typescript const turbo = TurboFactory.unauthenticated({ token: 'solana' }); const { tokenPrice } = await turbo.getTokenPriceForBytes({ byteCount: 1024 * 1024 * 100, }); console.log(tokenPrice); // Estimated SOL Price for 100 MiB ``` #### getUploadCosts() Returns the estimated cost in Winston Credits for the provided file sizes, including all upload adjustments and fees. ```typescript const [uploadCostForFile] = await turbo.getUploadCosts({ bytes: [1024] }); const { winc, adjustments } = uploadCostForFile; ``` #### uploadSignedDataItem() Uploads a signed data item. The provided `dataItemStreamFactory` should produce a NEW signed data item stream each time is it invoked. The `dataItemSizeFactory` is a function that returns the size of the file. The `signal` is an optional [AbortSignal] that can be used to cancel the upload or timeout the request. The `events` parameter is an optional object that can be used to listen to upload progress, errors, and success (refer to the [Events] section for more details). ```typescript const filePath = path.join(__dirname, './my-signed-data-item'); const dataItemSize = fs.statSync(filePath).size; const uploadResponse = await turbo.uploadSignedDataItem({ dataItemStreamFactory: () => fs.createReadStream(filePath), dataItemSizeFactory: () => dataItemSize, signal: AbortSignal.timeout(10_000), // cancel the upload after 10 seconds events: { // track upload events only onUploadProgress: ({ totalBytes, processedBytes }) => { console.log('Upload progress:', { totalBytes, processedBytes }); }, onUploadError: (error) => { console.log('Upload error:', { error }); }, onUploadSuccess: () => { console.log('Upload success!'); }, }, }); ``` #### createCheckoutSession() Creates a Stripe checkout session for a Turbo Top Up with the provided amount, currency, owner. The returned URL can be opened in the browser, all payments are processed by Stripe. To leverage promo codes, see [TurboAuthenticatedClient]. ##### Arweave (AR) Fiat Top Up ```typescript const { url, winc, paymentAmount, quotedPaymentAmount, adjustments } = await turbo.createCheckoutSession({ amount: USD(10.0), // $10.00 USD owner: publicArweaveAddress, // promo codes require an authenticated client }); // Open checkout session in a browser window.open(url, '_blank'); ``` ##### Ethereum (ETH) Fiat Top Up ```typescript const turbo = TurboFactory.unauthenticated({ token: 'ethereum' }); const { url, winc, paymentAmount } = await turbo.createCheckoutSession({ amount: USD(10.0), // $10.00 USD owner: publicEthereumAddress, }); ``` ##### Solana (SOL) Fiat Top Up ```typescript const turbo = TurboFactory.unauthenticated({ token: 'solana' }); const { url, winc, paymentAmount } = await turbo.createCheckoutSession({ amount: USD(10.0), // $10.00 USD owner: publicSolanaAddress, }); ``` ##### Polygon (POL / MATIC) Fiat Top Up ```typescript const turbo = TurboFactory.unauthenticated({ token: 'pol' }); const { url, winc, paymentAmount } = await turbo.createCheckoutSession({ amount: USD(10.0), // $10.00 USD owner: publicPolygonAddress, }); ``` #### submitFundTransaction() Submits the transaction ID of a funding transaction to Turbo Payment Service for top up processing. The `txId` is the transaction ID of the transaction to be submitted. Use this API if you've already executed your token transfer to the Turbo wallet. Otherwise, consider using `topUpWithTokens` to execute a new token transfer to the Turbo wallet and submit its resulting transaction ID for top up processing all in one go ```typescript const turbo = TurboFactory.unauthenticated(); // defaults to arweave token type const { status, id, ...fundResult } = await turbo.submitFundTransaction({ txId: 'my-valid-arweave-fund-transaction-id', }); ``` # Buying a name with a credit card (fiat / Stripe) (/sdks/turbo-sdk/(arns-names)/buying-a-name-with-a-credit-card-fiat-stripe) `getArNSFiatPurchaseQuote` prices a purchase in fiat and returns a Stripe payment session, so a user can buy a name without holding credits first. ```typescript const quote = await turbo.getArNSFiatPurchaseQuote({ name: 'my-name', intent: 'Buy-Name', type: 'lease', years: 1, currency: 'usd', }); ``` Its `paymentAmount` is the real charge and **already includes** the ANT spawn surcharge. (On the `getArNSPriceForName` fiat estimate the split is the other way round: `fiatEstimate.paymentAmount` is the base and `fiatEstimate.paymentAmountWithAntSpawn` is the total.) Throws `FiatPaymentsDisabledError` when the service has Stripe switched off. # Listing a wallet's names (/sdks/turbo-sdk/(arns-names)/listing-a-wallet-s-names) ```typescript const { names } = await turbo.getArNSNames(); // defaults to the signer's address ``` Receipt history, not a live ownership check: a name transferred away still appears. Verify present control on chain using the returned `antId`. # Nonces, retries and refunds (/sdks/turbo-sdk/(arns-names)/nonces-retries-and-refunds) Credits are debited when the action is **created**, not when it is signed. So: - **Persist the nonce before prompting for a signature** — use `onNonce`. - **Never re-create an action to retry.** That debits again. Poll instead: `await turbo.getArNSActionStatus(nonce)`. - **An abandoned action refunds itself** — don't build a refund flow. - Replaying `signArNSAction` on a completed action returns `{ alreadyCompleted: true }` rather than buying twice. `InsufficientCreditsError` (HTTP 402) is thrown when the balance is short; prompt a top-up, then create a **fresh** action. **The owner has about 30 seconds to sign.** Solana accepts the transaction Turbo builds for only about 30 seconds after the action is created, so prompt the owner straight away. `expiresAt` is not that deadline: it is the ~15-minute point at which an uncompleted action is refunded. `signArNSAction` throws `ArNSActionExpiredError` (a `FailedRequestError`, with `nonce`, `status` and `creditsReleased`) when the signature arrived too late: a 409, or a 400 reading `Action \ expired...`. `creditsReleased` is `true` when the service has already returned the credits, and `false` when they are held until the ~15-minute refund. Either way, create a **new** action if the change is still wanted. A 503 `Blockhash not found` is different: the service could not prove the transaction expired. Re-post the **same** signed bytes to `signArNSAction` rather than creating a new action; a resubmission is idempotent. `/sign` needs no payer signature: the owner's signature inside the transaction authorises it, and the credits were debited at create. So `signArNSAction` sends no payer headers by default. Its optional third argument, `headers`, is for a service that requires them. # Not covered — these still cost you SOL (/sdks/turbo-sdk/(arns-names)/not-covered-these-still-cost-you-sol) Sponsorship covers the **twelve actions above and nothing else**. Everything else in the ArNS, ANT and core programs stays on the direct-signer path via [`@ar.io/sdk`](https://github.com/ar-io/ar-io-sdk) and costs the user SOL — notably **buying a returned name** (auctions, deliberately excluded: the premium is unbounded), claiming a reserved name, the **primary-name** flow (which lives in the ario _core_ program), release/reassign, and **ANT-level** metadata. Note ANT-level metadata (the ANT's own name/ticker/description/keywords/logo) is distinct from RECORD-level metadata, which `setArNSRecordMetadata` does sponsor. Don't tell users they can "manage a name forever without SOL" — scope the claim to the twelve actions above. # Pricing — quote the total (/sdks/turbo-sdk/(arns-names)/pricing-quote-the-total) ```typescript const price = await turbo.getArNSPriceForName({ intent: 'Buy-Name', name: 'my-name', type: 'lease', years: 1, }); price.wincTotal; // <- charge or display THIS price.winc; // the name only, EXCLUDING the ANT spawn surcharge ``` Buying mints a fresh ANT, and Turbo fronts that account's Solana rent. A flat cost-recovery surcharge covers it, and in a real response **the surcharge can exceed the name's own price** — so reading `winc` under-quotes every purchase. `wincTotal` is added by the SDK precisely so the correct field is the obvious one. Never hardcode the surcharge: it is config-driven and derived from live rates. # The twelve sponsored actions (/sdks/turbo-sdk/(arns-names)/the-twelve-sponsored-actions) ```typescript const turbo = TurboFactory.authenticated({ privateKey: jwk }); // Buy — the ONE signature in the whole lifecycle. Grants Turbo controller // rights in this SAME transaction, which is why everything below needs no // signature of its own until you revoke it. const { antId, messageId } = await turbo.buyArNSName({ name: 'my-name', owner, type: 'lease', // or 'permabuy' years: 1, // leases only onNonce: (nonce) => persist(nonce), // fires BEFORE the wallet prompt }); // Lifecycle — no signature at all, spends ARIO. await turbo.extendArNSLease({ name: 'my-name', years: 2 }); await turbo.upgradeArNSName({ name: 'my-name' }); await turbo.increaseArNSUndernameLimit({ name: 'my-name', increaseQty: 5 }); // Records — a small flat/derived credits margin recovers the sponsored SOL // rent. Handled whichever shape the server picks. await turbo.setArNSRecord({ antId, owner, transactionId, undername: '@', ttlSeconds: 900, }); await turbo.removeArNSRecord({ antId, owner, undername: 'docs' }); // Record metadata — display name, logo, description, keywords. Same margin, // same shape rules as setArNSRecord. `null` clears a field; omit to leave it. await turbo.setArNSRecordMetadata({ antId, owner, undername: '@', displayName: 'My Docs', recordDescription: null, // clear it }); await turbo.removeArNSRecordMetadata({ antId, owner, undername: 'docs' }); // Hand ONE record to another address — distinct from transferring the ANT. await turbo.transferArNSRecord({ antId, owner, undername: 'docs', target: newOwnerAddress, }); // Controllers and transfer — owner-signed, same flat/derived margin. // addArNSController is for RE-granting after a revoke, or granting some // OTHER address — Turbo already has it from the buy above. await turbo.addArNSController({ antId, owner }); // omit target => Turbo await turbo.removeArNSController({ antId, owner }); // the revoke await turbo.transferArNSAnt({ antId, owner, target: newOwnerAddress }); ``` | Action | Costs credits | Owner signature | | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------ | --------------------------- | | `buyArNSName` | yes — ARIO purchase + ANT spawn rent | **always**, once | | `extendArNSLease` / `upgradeArNSName` / `increaseArNSUndernameLimit` | yes — ARIO purchase | no | | `setArNSRecord` / `removeArNSRecord` / `setArNSRecordMetadata` / `removeArNSRecordMetadata` / `transferArNSRecord` | yes — small flat/derived margin | only after you revoke Turbo | | `addArNSController` / `removeArNSController` / `transferArNSAnt` | yes — small flat/derived margin | yes | #### Point the name at your content while you buy it Pass `antState` and the ANT's opening record is written by the `ario_ant::initialize` that runs inside the transaction you already sign — free and atomic. No second action, no second signature, no second debit. Without it a fresh name resolves to the AR.IO logo, which is the on-chain default. ```typescript await turbo.buyArNSName({ name: 'my-name', owner, type: 'permabuy', antState: { transactionId: '\', // the root `@` target targetProtocol: 0, // 0 = Arweave (default), 1 = IPFS CID ticker: 'MYSITE', }, }); ``` `antState` is accepted on `buyArNSName` only — the service rejects it on every other action. Note it is nested: a **top-level** `transactionId` on a buy is a 400 by design, because that spelling means the set-record target and silently accepting it would point the name at the logo while the caller believed otherwise. **Mind the size budget.** The sponsored mint is ONE Solana transaction against the 1232-byte packet limit, shared with Turbo's fee-payer transfer and an `add_controller` grant. At a worst-case 51-character name only ~71 bytes are spare: | Fields | Cost | Fits? | | ------------------------------------------- | --------- | ------ | | `transactionId` + `targetProtocol` | ~1 byte | always | | `ticker` (16) + `logo` (43) | ~65 bytes | yes | | `description` (512), or a full keyword list | — | **no** | The budget is dynamic — a shorter name buys headroom — so this SDK imposes no client-side cap. The server measures the real transaction and returns a 400 naming Solana's 1232-byte limit _before_ you are handed anything to sign, with the credit debit refunded inline. That error is deterministic: do not retry it, and surface the server's message rather than replacing it, because it names which fields to drop. Field limits, all rejected at the service edge before any debit: `description` ≤ 512 characters, `keywords` ≤ 16 entries, and `logo` (plus `transactionId` when `targetProtocol` is 0 or unset) must be 43-character Arweave ids. When `targetProtocol` is 1 the target is an IPFS CID and is not shape-checked as an Arweave id. See [`ARNS_ACTIONS_API.md#buy-name-takes-the-ants-opening-state](https://github.com/ar-io/ar-io-bundler/blob/main/docs/architecture/ARNS_ACTIONS_API.md#buy-name-takes-the-ants-opening-state) in `ar-io/ar-io-bundler` for the measured byte table. Every action costs credits — gas sponsorship was never meant to be _free_ sponsorship. The four purchase actions charge the ARIO cost (plus, for `buyArNSName`, a rent-derived surcharge for the ANT it mints); the other eight charge a small margin that recovers the Solana rent/fees Turbo fronts on your behalf, computed the same `max(rent-derived, flat floor)` way as the ANT spawn surcharge. Preview it before you pay: ```typescript const { wincQty } = await turbo.getArNSActionPrice('remove-controller'); ``` `getArNSActionPrice` covers the eight non-purchase actions, by their route name (`set-record`, `remove-record`, `set-record-metadata`, `remove-record-metadata`, `transfer-record`, `add-controller`, `remove-controller`, `transfer`) — use `getArNSPriceForName` for the four purchase actions instead, since their cost is dominated by the ARIO purchase, not this margin. `buyArNSName` grants Turbo controller rights **inside the same transaction you sign** — the `add-controller(Turbo)` instruction rides along with the mint, so there is no separate step. That's why `setArNSRecord` and the rest complete in a single call immediately after buying, with no signature of their own. `addArNSController` is for re-granting after a revoke, or adding a different controller — not something you call after a fresh buy. Revoking is always available — but, like every other action here, not free of credits. # The two shapes, if you drive it yourself (/sdks/turbo-sdk/(arns-names)/the-two-shapes-if-you-drive-it-yourself) Every action returns one of two shapes, and **the server picks which**: ```typescript let res = await turbo.createArNSAction('buy-name', { name, ownerAddress }); if (res.status === 'awaiting-signature') { res = await turbo.signArNSAction( res.nonce, await owner.signTransaction(res.transaction), ); } // res.status === 'completed'; res.messageId is the on-chain write ``` Branch on `status`, never on which action you called: `setArNSRecord` completes alone while Turbo is a controller and flips to `awaiting-signature` the moment you revoke Turbo. It degrades instead of breaking. **Sign the exact bytes returned.** Turbo has already signed as fee payer; rebuilding the transaction invalidates that signature. # Two identities, never conflated (/sdks/turbo-sdk/(arns-names)/two-identities-never-conflated) | | Who | How it travels | | ------------- | ---------------------------------------------------------------- | --------------------- | | **Payer** | the Turbo identity holding credits — Arweave, Ethereum or Solana | the client's signer | | **ANT owner** | always a **Solana** address | the `owner` parameter | They are allowed to be different wallets, and routinely are: one account pays while another owns. # You need a Solana key, not Solana funds (/sdks/turbo-sdk/(arns-names)/you-need-a-solana-key-not-solana-funds) An ANT is a Metaplex Core asset on Solana, so the owner is always a Solana address — even when you pay with Arweave or Ethereum credits. But the owner never pays: Turbo is the fee payer on every sponsored action, so **the owner's SOL balance can stay at zero for the life of the name**. Supply the owner as an `ArNSOwnerSigner`: ```typescript // From a secret key (servers, scripts, tests) const owner = solanaOwnerSigner(bs58SolanaSecretKey); ``` A browser wallet (Phantom, Solflare, or an app's embedded wallet) should implement the interface directly rather than exposing a secret key: ```typescript const owner = { getAddress: () => wallet.publicKey.toBase58(), signTransaction: async (txBase64) => { // atob/btoa rather than Buffer: browsers do not provide Buffer unless the // app polyfills it. The spread is safe here because a Solana transaction // is capped at 1232 bytes. const tx = VersionedTransaction.deserialize( Uint8Array.from(atob(txBase64), (c) => c.charCodeAt(0)), ); const signed = await wallet.signTransaction(tx); return btoa(String.fromCharCode(...signed.serialize())); }, signMessage: (message) => wallet.signMessage(message), }; ``` # File Upload Events (/sdks/turbo-sdk/(events)/file-upload-events) These events are available for `upload`, `uploadFile`, and `uploadSignedDataItem` methods: - `onProgress` - emitted when the overall progress changes (includes both upload and signing). Each event consists of the total bytes, processed bytes, and the step (upload or signing) - `onError` - emitted when the overall upload or signing fails (includes both upload and signing) - `onSuccess` - emitted when the overall upload or signing succeeds (includes both upload and signing) - this is the last event emitted for the upload or signing process - `onSigningProgress` - emitted when the signing progress changes. - `onSigningError` - emitted when the signing fails. - `onSigningSuccess` - emitted when the signing succeeds - `onUploadProgress` - emitted when the upload progress changes - `onUploadError` - emitted when the upload fails - `onUploadSuccess` - emitted when the upload succeeds ```typescript const uploadResult = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream(filePath), fileSizeFactory: () => fileSize, events: { // overall events (includes signing and upload events) onProgress: ({ totalBytes, processedBytes, step }) => { console.log('Overall progress:', { totalBytes, processedBytes, step }); }, onError: ({ error, step }) => { console.log('Overall error:', { error, step }); }, onSuccess: () => { console.log('Overall success!'); }, // signing events onSigningProgress: ({ totalBytes, processedBytes }) => { console.log('Signing progress:', { totalBytes, processedBytes }); }, onSigningError: (error) => { console.log('Signing error:', { error }); }, onSigningSuccess: () => { console.log('Signing success!'); }, // upload events onUploadProgress: ({ totalBytes, processedBytes }) => { console.log('Upload progress:', { totalBytes, processedBytes }); }, onUploadError: (error) => { console.log('Upload error:', { error }); }, onUploadSuccess: () => { console.log('Upload success!'); }, }, }); ``` # Folder Upload Events (/sdks/turbo-sdk/(events)/folder-upload-events) These events are available for the `uploadFolder` method: - `onFileStart` - emitted when a file in the folder starts uploading. Includes the file name, file size, file index, and total number of files - `onFileProgress` - emitted when a file's upload or signing progress changes. Includes the file name, file index, total files, processed bytes for the file, total bytes for the file, and the current step (signing or upload) - `onFileComplete` - emitted when a file successfully completes uploading. Includes the file name, file index, total files, and the data item ID - `onFileError` - emitted when a file upload fails. Includes the file name, file index, total files, and the error - `onFolderProgress` - emitted when the overall folder upload progress changes. Includes the number of processed files, total files, processed bytes across all files, total bytes across all files, and the current phase (files or manifest) - `onFolderError` - emitted when the overall folder upload fails - `onFolderSuccess` - emitted when the folder upload successfully completes (including manifest generation) - this is the last event emitted for the folder upload process ```typescript const uploadResult = await turbo.upload({ data: 'The contents of my file!', signal: AbortSignal.timeout(10_000), // cancel the upload after 10 seconds dataItemOpts: { // optional }, events: { // overall events (includes signing and upload events) onProgress: ({ totalBytes, processedBytes, step }) => { const percentComplete = (processedBytes / totalBytes) * 100; console.log('Overall progress:', { totalBytes, processedBytes, step, percentComplete: percentComplete.toFixed(2) + '%', // eg 50.68% }); }, onError: (error) => { console.log('Overall error:', { error }); }, onSuccess: () => { console.log('Signed and upload data item!'); }, // upload events onUploadProgress: ({ totalBytes, processedBytes }) => { console.log('Upload progress:', { totalBytes, processedBytes }); }, onUploadError: (error) => { console.log('Upload error:', { error }); }, onUploadSuccess: () => { console.log('Upload success!'); }, // signing events onSigningProgress: ({ totalBytes, processedBytes }) => { console.log('Signing progress:', { totalBytes, processedBytes }); }, onSigningError: (error) => { console.log('Signing error:', { error }); }, onSigningSuccess: () => { console.log('Signing success!'); }, }, }); ``` # Arweave (/sdks/turbo-sdk/(signers)/arweave) #### Arweave JWK ```typescript const jwk = await arweave.crypto.generateJWK(); const turbo = TurboFactory.authenticated({ privateKey: jwk }); ``` #### ArweaveSigner ```typescript const signer = new ArweaveSigner(jwk); const turbo = TurboFactory.authenticated({ signer }); ``` #### ArconnectSigner ```typescript const signer = new ArconnectSigner(window.arweaveWallet); const turbo = TurboFactory.authenticated({ signer }); ``` # Base (/sdks/turbo-sdk/(signers)/base) #### Base ETH Private Key ```typescript const turbo = TurboFactory.authenticated({ privateKey: ethHexadecimalPrivateKey, token: 'base-eth', }); ``` #### Base USDC Private Key ```typescript const turbo = TurboFactory.authenticated({ privateKey: ethHexadecimalPrivateKey, token: 'base-usdc', }); ``` # Ethereum (/sdks/turbo-sdk/(signers)/ethereum) #### EthereumSigner ```typescript const signer = new EthereumSigner(privateKey); const turbo = TurboFactory.authenticated({ signer }); ``` #### Ethereum Private Key ```typescript const turbo = TurboFactory.authenticated({ privateKey: ethHexadecimalPrivateKey, token: 'ethereum', }); ``` #### POL (MATIC) Private Key ```typescript const turbo = TurboFactory.authenticated({ privateKey: ethHexadecimalPrivateKey, token: 'pol', }); ``` # Solana (/sdks/turbo-sdk/(signers)/solana) #### HexSolanaSigner ```typescript const signer = new HexSolanaSigner(bs58.encode(secretKey)); const turbo = TurboFactory.authenticated({ signer }); ``` #### Solana Web Wallet Adapter ```typescript const turbo = TurboFactory.authenticated({ walletAdapter: window.solana, token: 'solana', }); ``` #### Solana Secret Key ```typescript const turbo = TurboFactory.authenticated({ privateKey: bs58.encode(secretKey), token: 'solana', }); ``` # Dependency advisories (/sdks/turbo-sdk/dependency-advisories) A clean install reports advisories from transitive dependencies, none from this package's own code. The list moves as those dependencies publish, so run `npm audit` for the current one. Three critical advisories come from `elliptic`, which reaches the tree through `@dha-team/arbundles` and its ethers v5 dependencies, along with a high advisory in `secp256k1`. Both clear with a package manager override, measured as three criticals to zero: ```json { "overrides": { "elliptic": "6.6.1", "secp256k1": "5.0.1" } } ``` Yarn reads the same pinning under `resolutions`. Two notes on what the remaining advisories mean here: - The one `ws` copy inside an advisory range, `7.4.6`, sits under `@ethersproject/providers`. Its WebSocket provider is never instantiated by this SDK. - `npm audit fix --force` offers to downgrade this package to 1.13.0. That is npm giving up inside the version ranges, not a fix. # Turbo SDK (/sdks/turbo-sdk) **For AI and LLM users**: Access the complete Turbo SDK documentation in plain text format at{" "} llm.txt for easy consumption by AI agents and language models. See [AI Agents & LLMs](/build/agents) for the full agent toolkit. The Turbo SDK provides a high-level interface for uploading data to Arweave through Turbo's optimized infrastructure. Built with TypeScript, it offers seamless integration with built-in error handling, automatic retries, and transparent pricing. ## Quick Start ### Install the SDK ```npm npm install @ardrive/turbo-sdk ``` ### Use the SDK ```javascript // Create an authenticated client const turbo = TurboFactory.authenticated({ privateKey: yourPrivateKey }); // Upload data with automatic payment const result = await turbo.uploadFile({ fileStreamFactory: () => fs.createReadStream('./my-file.pdf'), fileSizeFactory: () => fs.statSync('./my-file.pdf').size, }); console.log('Upload successful:', result); ``` ### Install the SDK ```npm npm install @ardrive/turbo-sdk ``` ### Install polyfills (required for web environments) Polyfills are required for React web environments due to the use of `crypto`, `buffer` and `process` types in the SDK's dependencies. ```npm npm install --save-dev vite-plugin-node-polyfills ``` ```js // vite.config.js plugins: [ nodePolyfills({ globals: { Buffer: true, global: true, process: true, }, }), ], }); ``` Configure your bundler (Webpack, Vite, Rollup, etc.) to provide polyfills for `crypto`, `process`, and `buffer`. Refer to your bundler's documentation for polyfill configuration. ### Use the SDK ```javascript // Create an authenticated client const turbo = TurboFactory.authenticated({ privateKey: yourPrivateKey }); // Upload data with automatic payment const fileInput = document.querySelector('input[type="file"]'); const file = fileInput.files[0]; const result = await turbo.uploadFile({ fileStreamFactory: () => file.stream(), fileSizeFactory: () => file.size, }); console.log('Upload successful:', result); ``` ```html Turbo SDK Upload Example // Polyfills are included in the minimized web bundle, so not necessary to import directly // Function to handle file upload async function uploadFile() { const fileInput = document.getElementById('fileInput'); const file = fileInput.files[0]; const privateKeyInput = document.getElementById('privateKey'); if (!file) { alert('Please select a file'); return; } if (!privateKeyInput.value) { alert('Please enter your private key'); return; } try { // Show loading state document.getElementById('status').textContent = 'Uploading...'; // Create authenticated client const turbo = TurboFactory.authenticated({ privateKey: privateKeyInput.value }); // Upload file const result = await turbo.uploadFile({ fileStreamFactory: () => file.stream(), fileSizeFactory: () => file.size, }); // Show success document.getElementById('status').innerHTML = ` Upload successful! Transaction ID: ${result.id} Data Item ID: ${result.dataItemId} `; } catch (error) { document.getElementById('status').innerHTML = ` Error: ${error.message} `; } } // Add click handler window.addEventListener('load', () => { document.getElementById('uploadBtn').addEventListener('click', uploadFile); }); Turbo SDK Upload Example Private Key (JWK): Select File: Upload to Arweave ``` ## API Reference & Documentation } title="API Reference" description="Complete API documentation for all Turbo client methods" href="/apis/turbo" /> } title="SDK Details" description="Advanced features, events, logging, and credit sharing" href="/sdks/turbo-sdk" /> ## Core Features } title="Upload Management" description="Authenticated and unauthenticated upload clients with retry logic" href="/sdks/turbo-sdk/turboauthenticatedclient" /> } title="Events & Monitoring" description="Monitor upload progress and handle events in real-time" href="/sdks/turbo-sdk/file-upload-events" /> } title="Credit Sharing" description="Manage shared credit pools for streamlined billing" href="/sdks/turbo-sdk/turbo-credit-sharing" /> } title="Logging & Configuration" description="Configure logging for debugging and monitoring uploads" href="/sdks/turbo-sdk/logging" /> # Logging (/sdks/turbo-sdk/logging) The SDK uses winston for logging. You can set the log level using the `setLogLevel` method. ```typescript TurboFactory.setLogLevel('debug'); ``` # Turbo Credit Sharing (/sdks/turbo-sdk/turbo-credit-sharing) Users can share their purchased Credits with other users' wallets by creating Credit Share Approvals. These approvals are created by uploading a signed data item with tags indicating the recipient's wallet address, the amount of Credits to share, and an optional amount of seconds that the approval will expire in. The recipient can then use the shared Credits to pay for their own uploads to Turbo. Shared Credits cannot be re-shared by the recipient to other recipients. Only the original owner of the Credits can share or revoke Credit Share Approvals. Credits that are shared to other wallets may not be used by the original owner of the Credits for sharing or uploading unless the Credit Share Approval is revoked or expired. Approvals can be revoked at any time by similarly uploading a signed data item with tags indicating the recipient's wallet address. This will remove all approvals and prevent the recipient from using the shared Credits. All unused Credits from expired or revoked approvals are returned to the original owner of the Credits. To use the shared Credits, recipient users must provide the wallet address of the user who shared the Credits with them in the `x-paid-by` HTTP header when uploading data. This tells Turbo services to look for and use Credit Share Approvals to pay for the upload before using the signer's balance. For user convenience, during upload the Turbo CLI will use any available Credit Share Approvals found for the connected wallet before using the signing wallet's balance. To instead ignore all Credit shares and only use the signer's balance, use the `--ignore-approvals` flag. To use the signer's balance first before using Credit shares, use the `--use-signer-balance-first` flag. In contrast, the Turbo SDK layer does not provide this functionality and will only use approvals when `paidBy` is provided. The Turbo SDK provides the following methods to manage Credit Share Approvals: - `shareCredits`: Creates a Credit Share Approval for the specified wallet address and amount of Credits. - `revokeCredits`: Revokes all Credit Share Approvals for the specified wallet address. - `getCreditShareApprovals`: Lists all Credit Share Approvals given or received by the specified wallet address or the connected wallet. - `dataItemOpts: { ...opts, paidBy: string[] }`: Upload methods now accept 'paidBy', an array of wallet addresses that have provided credit share approvals to the user from which to pay, in the order provided and as necessary, for the upload. The Turbo CLI provides the following commands to manage Credit Share Approvals: - `share-credits`: Creates a Credit Share Approval for the specified wallet address and amount of Credits. - `revoke-credits`: Revokes all Credit Share Approvals for the specified wallet address. - `list-shares`: Lists all Credit Share Approvals for the specified wallet address or connected wallet. - `paidBy: --paid-by `: Upload commands now accept '--paid-by', an array of wallet addresses that have provided credit share approvals to the user from which to pay, in the order provided and as necessary, for the upload. - `--ignore-approvals`: Ignore all Credit Share Approvals and only use the signer's balance. - `--use-signer-balance-first`: Use the signer's balance first before using Credit Share Approvals. # turbo-upload (/sdks/turbo-upload) **For AI and LLM users**: Access the complete turbo-upload documentation in plain text format at{" "} llm.txt for easy consumption by AI agents and language models. See [AI Agents & LLMs](/build/agents) for the full agent toolkit. `turbo-upload` signs ANS-104 data items with an Arweave JWK or a Solana key and uploads them to Turbo. That is all it does, and it has no dependencies. ## Which One To Use **Start with the [Turbo SDK](/sdks/turbo-sdk).** It is the full client and the right choice for most things. Reach for `turbo-upload` when the size of a new dependency decides whether your change gets merged. | | [Turbo SDK](/sdks/turbo-sdk) | turbo-upload | |---|---|---| | Keys | Arweave, Solana, Ethereum, KYVE, Polygon | Arweave JWK or Solana key | | Runs in | Node and the browser | Node only | | Does | upload, payments, credits, folders, CLI, ArNS | sign and upload | | Installed on its own | 784 lockfile entries, 895 MB | 1 entry, 460 KB | Measured on 2026-09-11 against `@ardrive/turbo-sdk@1.43.0` and `@ardrive/turbo-upload@0.3.0`, installing each into an empty project. Re-run it rather than trusting the table. The two share no state, so having both installed is fine. ## Quick Start ### Install ```npm npm install @ardrive/turbo-upload ``` ### Upload ```javascript // .testnet() is free and nothing it writes is permanent. // .production() is mainnet: permanent, public, paid from this wallet. const client = TurboUpload.testnet({ jwk: JSON.parse(process.env.ARWEAVE_JWK) }); const { id, winc } = await client.upload({ data: Buffer.from("hello permanence"), tags: [{ name: "Content-Type", value: "text/plain" }], }); ``` Read it back from any gateway at `/`, so retrieval never depends on the service you uploaded through. [Wayfinder](/sdks/wayfinder) handles gateway choice and verification for you. ## Solana Keys ```javascript const client = TurboUpload.production({ jwk: process.env.SOLANA_SECRET_KEY, token: "solana", }); ``` Takes any form a Solana user holds: a base58 secret key as Phantom exports it, the JSON array `solana-keygen` writes, raw 64 bytes, or a bare 32-byte seed. `client.address` is the base58 Solana address. This is **ANS-104 signature type 4**, the same type the Turbo SDK uses for `token: "solana"`, so both produce identical ids for identical content. A 64-byte key carries its own public key, and that half is checked rather than trusted. A key whose halves disagree is refused when the client is constructed, because signing with one produces items nothing can verify and you find out after paying. ## What Gets Written Wrong The rest of the API behaves as you would expect. These do not. - **Signing twice charges twice.** RSA-PSS uses a fresh random salt each time, so the same payload signed twice has a different id. `upload()` signs internally, so `sign()` then `upload()` charges for two items. Use `sign()` then `uploadSigned(item)`, and record `item.idB64Url`, which is the string form. - **`target` and `anchor` look alike and are not.** Adjacent 32-byte fields: `target` is base64url that decodes to 32 bytes, `anchor` is 32 raw bytes. A 43-character base64url anchor throws. - **Production is the default.** A bare `new TurboUpload({ jwk })` writes to mainnet, permanently, paid. - **An unrecognised option throws** rather than being ignored, and the error names the key you meant. Do not spread `PRODUCTION` or `TESTNET` into the constructor. Those records also carry `name` and `gatewayUrl`, which are not constructor options. Use `TurboUpload.production()` and `TurboUpload.testnet()`. ## Endpoints | | upload | payment | gateway | |---|---|---|---| | `PRODUCTION` | `https://upload.ardrive.io` | `https://payment.ardrive.io` | `https://turbo-gateway.com` | | `TESTNET` | `https://upload.services.ar-io.dev` | `https://payment.services.ar-io.dev` | `https://ar-io.dev` | Exported as constants. Import `PRODUCTION` or `TESTNET` rather than typing a hostname: a near-miss on a testnet host can resolve to something that answers `200` with HTML, which fails much less obviously than a bad hostname should. ## Requirements Node 18.17 or later, tested on 18.17, 18, 20, 22 and 24. Server-side only, because a raw Arweave key does not belong in a browser. Published with npm provenance. Every error extends `TurboError`. Catch `TurboPaymentError` separately: it means the wallet cannot pay, and retrying never helps. ## Next Steps # Wayfinder SDK's (/sdks/wayfinder) **For AI and LLM users**: Access the complete Wayfinder documentation in plain text format at{" "} llm.txt for easy consumption by AI agents and language models. See [AI Agents & LLMs](/build/agents) for the full agent toolkit. Wayfinder leverages the decentralized ar.io Network to provide robust, censorship-resistant access to data stored on Arweave, removing reliance on centralized gateways. By routing requests through a distributed set of community-operated gateways, Wayfinder ensures high availability, redundancy, and improved performance for users and applications. The `ar://` protocol enables decentralized resolution and access to Arweave data using several flexible URL formats: - `ar://TRANSACTION_ID` — Direct access to a specific Arweave transaction - `ar://NAME` — Resolution of ArNS names (with optional path support) - `ar:///info` — Direct access to gateway endpoints (e.g., `/info`) To learn more about the Wayfinder protocol and how it works, visit [/learn/wayfinder](/learn/wayfinder). ## Getting Started Choose your environment to get started with Wayfinder: } title="@ar.io/wayfinder-core" description="Core SDK for Node.js applications and server environments" href="/sdks/wayfinder/wayfinder-core" /> } title="@ar.io/wayfinder-react" description="React hooks and components for browser applications" href="/sdks/wayfinder/wayfinder-react" /> } title="Wayfinder Router" description="Standalone proxy service with built-in verification and caching" href="/build/run-wayfinder-router" /> **Don't want to embed an SDK?** [Wayfinder Router](/build/run-wayfinder-router) is a standalone proxy you can deploy to provide a single verified endpoint for your users. ## Quick Examples ### Node.js ```javascript // Create a Wayfinder client const wayfinder = createWayfinderClient({ ario: ARIO.mainnet(), }); // Fetch data using the ar:// protocol const response = await wayfinder.request('ar://ardrive'); console.log(response); ``` ### React ```jsx // Wrap your app with the provider function App() { return ( ); } // Use the hook in your components function WayfinderImage({ txId }: { txId: string }) { const { resolvedUrl, isLoading, error } = useWayfinderUrl({ txId }); if (error) { return Error resolving URL: {error.message}; } if (isLoading) { return Resolving URL...; } return ( ); } ``` ## Next Steps } title="Learn about ar.io" description="Understanding the ar.io decentralized infrastructure" href="/learn/what-is-ario" /> } title="Decentralized Access" description="Learn how to access Arweave data in a decentralized way" href="/build/access" /> } title="Run Wayfinder Router" description="Deploy a standalone verified proxy for your users" href="/build/run-wayfinder-router" /> } title="Run a Gateway" description="Join the network by operating your own ar.io gateway" href="/build/run-a-gateway/quick-start" /> # Data Retrieval Strategies (/sdks/wayfinder/wayfinder-core/data-retrieval-strategies) Wayfinder supports multiple data retrieval strategies to fetch transaction data from AR.IO gateways. These strategies determine how data is requested and assembled from the underlying storage layer. | Strategy | Use Case | Requirements | | --------------------------------- | ------------------------------------------- | -------------------------------------- | | `ContiguousDataRetrievalStrategy`| Standard data fetching via direct GET | Gateway has the data cached or able to fetch from trusted peers | | `ChunkDataRetrievalStrategy` | Chunk-based data assembly | Gateway supports `/chunk/\/data` endpoint (r58) and has requested transactions indexed | #### ContiguousDataRetrievalStrategy The default strategy that fetches data using a direct GET request to the gateway. This is the most straightforward approach and works for most use cases. ```javascript const wayfinder = new Wayfinder({ dataRetrievalStrategy: new ContiguousDataRetrievalStrategy(), }); ``` #### ChunkDataRetrievalStrategy An advanced strategy that provides the easiest way to load chunks stored on Arweave nodes via the robust chunk API provided by AR.IO gateways. This approach is particularly useful for: - **Direct chunk access**: Efficiently retrieves data directly from the underlying chunk storage layer - **Bundled data items**: Seamlessly fetches data items from within ANS-104 bundles using calculated offsets - **x402 payment compatibility**: Both strategies support custom fetch clients for payment-enabled requests - **Large file handling**: More reliable for large transactions that may time out with direct requests **Requirements:** - Gateway must support the `/chunk/\/data` endpoint (added in [r58](https://github.com/ar-io/ar-io-node/releases/tag/r58)) - Gateway must have the requested transaction indexed (offsets are needed to fetch directly from chunks) ```javascript const wayfinder = new Wayfinder({ dataRetrievalStrategy: new ChunkDataRetrievalStrategy(), }); ``` **How it works:** 1. Makes a HEAD request to get transaction metadata (root transaction ID, data offset, content length) 2. Queries `/tx/{root-tx-id}/offset` to get the root transaction's absolute offset in the weave 3. Calculates the absolute offset for the requested data item 4. Fetches data in chunks using `/chunk/\/data` and assembles the complete data stream 5. Validates that chunks belong to the expected root transaction for security **Sequence Diagram:** ```mermaid sequenceDiagram participant Client participant Wayfinder participant Gateway as AR.IO Gateway participant Arweave as Arweave Nodes Client->>Wayfinder: request('ar://data-item-id') activate Wayfinder Wayfinder->>Gateway: HEAD /tx/{data-item-id} Note over Gateway: Lookup data item metadatafrom indexed bundles Gateway-->>Wayfinder: Headers:- x-root-tx-id (bundle ID)- x-data-offset- content-length Wayfinder->>Gateway: GET /tx/{root-tx-id}/offset Gateway->>Arweave: GET /tx/{root-tx-id}/offset Note over Arweave: Lookup transaction offsetin the weave Arweave-->>Gateway: Root transaction offset Gateway-->>Wayfinder: Root transaction offset in weave Note over Wayfinder: Calculate absolute offset:absolute = root_offset + data_offset loop For each chunk needed Wayfinder->>Gateway: GET /chunk/{absolute-offset}/data Note over Gateway: Serve chunk data fromindexed storage usingroot transaction ID Gateway-->>Wayfinder: Chunk data + validation headers(x-root-tx-id for security) Note over Wayfinder: Validate chunk belongsto expected root TX Wayfinder-->>Client: Stream chunk data end Wayfinder-->>Client: Complete response deactivate Wayfinder ``` **Example with createWayfinderClient:** ```javascript const wayfinder = createWayfinderClient({ dataRetrievalStrategy: new ChunkDataRetrievalStrategy(), }); // Fetch a data item from within an ANS-104 bundle const response = await wayfinder.request('ar://data-item-id'); ``` #### x402 Support Both data retrieval strategies support custom fetch implementations, allowing you to use x402-enabled fetch clients for paid gateway requests. ```javascript const x402Fetch = createX402Fetch({ /* payment config */ }); const wayfinder = createWayfinderClient({ fetch: x402Fetch, dataRetrievalStrategy: new ChunkDataRetrievalStrategy({ fetch: x402Fetch, }), }); ``` # Dynamic Routing (/sdks/wayfinder/wayfinder-core/dynamic-routing) Wayfinder supports a `resolveUrl` method which generates dynamic redirect URLs to a target gateway based on the provided routing strategy. This function can be used to directly replace any hard-coded gateway URLs, and instead use Wayfinder's routing logic to select a gateway for the request. #### ArNS names Given an ArNS name, the redirect URL will be the same as the original URL, but with the gateway selected by Wayfinder's routing strategy. ```javascript const redirectUrl = await wayfinder.resolveUrl({ arnsName: 'ardrive', }); // results in https://ardrive.\ ``` #### Transaction Ids Given a txId, the redirect URL will be the same as the original URL, but with the gateway selected by Wayfinder's routing strategy. ```javascript const redirectUrl = await wayfinder.resolveUrl({ txId: 'example-tx-id', }); // results in https://\/example-tx-id ``` #### Legacy URLs Given a legacy arweave.net or arweave.dev URL, the redirect URL will be the same as the original URL, but with the gateway selected by Wayfinder's routing strategy. ```javascript const redirectUrl = await wayfinder.resolveUrl({ originalUrl: 'https://arweave.net/example-tx-id', }); // results in https://\/example-tx-id ``` #### ar:// URLs Given an ar:// URL, the redirect URL will be the same as the original URL, but with the gateway selected by Wayfinder's routing strategy. ```javascript const redirectUrl = await wayfinder.resolveUrl({ originalUrl: 'ar://example-name/subpath?query=value', }); // results in https://\/example-name/subpath?query=value ``` # Events and Monitoring (/sdks/wayfinder/wayfinder-core/events-and-monitoring) #### Global Events ```javascript const wayfinder = createWayfinderClient({ routingSettings: { events: { onRoutingStarted: (event) => console.log('Routing started:', event), onRoutingSucceeded: (event) => console.log('Gateway selected:', event), }, }, verificationSettings: { events: { onVerificationProgress: (event) => { const percentage = (event.processedBytes / event.totalBytes) * 100; console.log(`Verification: ${percentage.toFixed(2)}%`); }, onVerificationSucceeded: (event) => console.log('Verified:', event.txId), }, }, }); ``` #### Request-Specific Events ```javascript const response = await wayfinder.request('ar://example', { verificationSettings: { events: { onVerificationProgress: (event) => { console.log(`This request: ${event.txId}`); }, }, }, }); ``` # Gateway Providers (/sdks/wayfinder/wayfinder-core/gateway-providers) Gateway providers supply the list of gateways for routing. **By default, `createWayfinderClient` uses a cached `TrustedPeersGatewaysProvider`**. | Provider | Description | Use Case | | ------------------------------ | ---------------------------------------------- | --------------------------------------- | | `NetworkGatewaysProvider` | Returns gateways from AR.IO Network | Leverage AR.IO Network with quality filtering | | `TrustedPeersGatewaysProvider` | Fetches from trusted gateway's peers | Dynamic gateway discovery (default) | | `StaticGatewaysProvider` | Returns a static list of gateways | Testing or specific gateways | | `SimpleCacheGatewaysProvider` | In-memory caching wrapper | Reduce API calls (used by default) | | `LocalStorageGatewaysProvider` | Browser localStorage caching | Persistent caching (used by default in browsers) | | `CompositeGatewaysProvider` | Chains multiple providers with fallback | Maximum resilience with multiple sources | #### NetworkGatewaysProvider Returns a list of gateways from the ARIO Network based on on-chain [Gateway Address Registry](https://docs.ar.io/learn/gateways/gateway-registry). You can specify on-chain metrics for gateways to prioritize the highest quality gateways. Requires `@ar.io/sdk` and `@solana/kit`. ```javascript const ario = ARIO.init({ rpc: createSolanaRpc('https://api.mainnet-beta.solana.com'), }); const gatewayProvider = new NetworkGatewaysProvider({ ario, sortBy: 'operatorStake', sortOrder: 'desc', limit: 10, filter: (gateway) => gateway.status === 'joined', }); ``` #### TrustedPeersGatewaysProvider Fetches a dynamic list of trusted peer gateways from an AR.IO gateway's `/ar-io/peers` endpoint. This provider is useful for discovering available gateways from a trusted source. ```javascript const gatewayProvider = new TrustedPeersGatewaysProvider({ trustedGateway: 'https://turbo-gateway.com', }); ``` #### CompositeGatewaysProvider Chains multiple gateway providers together, trying each in sequence until one succeeds. This is useful for building resilient gateway discovery with fallbacks. **How it works:** 1. Tries each provider in the order they're provided 2. If a provider returns a non-empty list of gateways, those gateways are used 3. If a provider throws an error or returns an empty list, moves to the next provider 4. If all providers fail, throws an error ```javascript import { CompositeGatewaysProvider, NetworkGatewaysProvider, StaticGatewaysProvider, TrustedPeersGatewaysProvider, } from '@ar.io/wayfinder-core'; const ario = ARIO.init({ rpc: createSolanaRpc('https://api.mainnet-beta.solana.com'), }); // Example: Network-first with static fallback const gatewayProvider = new CompositeGatewaysProvider({ providers: [ // Try fetching from AR.IO network first new NetworkGatewaysProvider({ ario, sortBy: 'operatorStake', limit: 10, }), // Fallback to trusted peers if network fetch fails new TrustedPeersGatewaysProvider({ trustedGateway: 'https://turbo-gateway.com', }), // Final fallback to static list new StaticGatewaysProvider({ gateways: ['https://turbo-gateway.com', 'https://g8way.io'], }), ], }); ``` # Wayfinder Core (/sdks/wayfinder/wayfinder-core) **Building for the web?** Consider using [@ar.io/wayfinder-react](/sdks/wayfinder/wayfinder-react) for React applications, which provides hooks and components optimized for browser environments. ## Quick Start ### Install ```npm npm install @ar.io/wayfinder-core @ar.io/sdk ``` ### Import and configure ```javascript const wayfinder = createWayfinderClient(); ``` ### Make your first request ```javascript // Use Wayfinder to fetch and verify data using ar:// protocol const response = await wayfinder.request('ar://ardrive'); console.log(response); ``` ### Use custom strategies ```javascript const customWayfinder = createWayfinderClient({ routingStrategy: createRoutingStrategy({ strategy: 'random' }) verificationStrategy: createVerificationStrategy({ strategy: 'hash', trustedGateways: ['https://turbo-gateway.com'] }) }); ``` ### Enable Telemetry (Optional) ```javascript const customWayfinder = createWayfinderClient({ telemetrySettings: { enabled: true, sampleRate: 0.1, // 10% sampling clientName: 'my-app', clientVersion: '1.0.0', } }); ``` ## Next Steps } title="Dynamic Routing" description="Use Wayfinder for dynamic routing" href="/sdks/wayfinder/wayfinder-core/dynamic-routing" /> } title="Gateway Providers" description="Learn about different gateway provider strategies" href="/sdks/wayfinder/wayfinder-core/gateway-providers" /> } title="Request Flow" description="Understanding how Wayfinder routes and verifies requests" href="/sdks/wayfinder/wayfinder-core/request-flow" /> } title="Monitoring & Events" description="Monitor performance and handle events" href="/sdks/wayfinder/wayfinder-core/events-and-monitoring" /> # Request Flow (/sdks/wayfinder/wayfinder-core/request-flow) ```mermaid sequenceDiagram participant Client participant Wayfinder participant Gateways Provider participant Routing Strategy participant Selected Gateway participant Verification Strategy participant Trusted Gateways Client->>Wayfinder: request('ar://example') activate Wayfinder Wayfinder->>+Gateways Provider: getGateways() Gateways Provider-->>-Wayfinder: List of gateway URLs Wayfinder->>+Routing Strategy: selectGateway() Routing Strategy-->>-Wayfinder: Selected gateway Wayfinder->>+Selected Gateway: HTTP request Selected Gateway-->>-Wayfinder: Response data opt Verification enabled Wayfinder->>+Verification Strategy: verifyData() Verification Strategy->>Trusted Gateways: Get verification data Trusted Gateways-->>Verification Strategy: Verification headers Verification Strategy-->>-Wayfinder: Verification result end Wayfinder-->>Client: Response or error deactivate Wayfinder ``` # Resiliency (/sdks/wayfinder/wayfinder-core/resiliency) Wayfinder includes built-in resiliency features: - **Gateway retry**: If a gateway returns a 5xx error or a network failure occurs, Wayfinder automatically re-selects a different gateway and retries (up to 3 attempts). Client errors (4xx) are returned immediately without retry. - **Fetch timeouts**: All outbound requests include configurable timeouts — 10s for metadata (HEAD, peer list), 30s for data retrieval — to prevent indefinite hangs on slow or dead gateways. - **Smart pagination**: `NetworkGatewaysProvider` stops fetching from the on-chain registry once enough gateways pass the filter, avoiding unnecessary RPC calls. # Routing Strategies (/sdks/wayfinder/wayfinder-core/routing-strategies) Wayfinder supports multiple routing strategies to select target gateways for your requests. | Strategy | Description | Use Case | | ---------------------------- | ---------------------------------------------- | --------------------------------------- | | `RandomRoutingStrategy` | Selects a random gateway from a list | Good for load balancing and resilience | | `StaticRoutingStrategy` | Always uses a single gateway | When you need to use a specific gateway | | `RoundRobinRoutingStrategy` | Selects gateways in round-robin order | Good for load balancing and resilience | | `FastestPingRoutingStrategy` | Selects the fastest gateway based on ping time | Good for performance and latency | | `PreferredWithFallbackRoutingStrategy` | Uses a preferred gateway, with a fallback strategy if the preferred gateway is not available | Good for performance and resilience. Ideal for builders who run their own gateways. | | `CompositeRoutingStrategy` | Chains multiple routing strategies together, trying each sequentially until one succeeds | Good for complex fallback scenarios and maximum resilience | #### RandomRoutingStrategy Selects a random gateway from a list of gateways. ```javascript const strategy = new RandomRoutingStrategy({ gatewaysProvider: myGatewaysProvider, }); ``` #### FastestPingRoutingStrategy Selects the fastest gateway based on ping time. This strategy pings all available gateways and selects the one with the lowest latency. ```javascript const strategy = new FastestPingRoutingStrategy({ timeoutMs: 1000, gatewaysProvider: myGatewaysProvider, }); ``` #### PreferredWithFallbackRoutingStrategy Uses a preferred gateway, with a fallback strategy if the preferred gateway is not available. This is useful for builders who run their own gateways and want to use their own gateway as the preferred gateway, but also want to have a fallback strategy in case their gateway is not available. This strategy is built using `CompositeRoutingStrategy` internally. It first attempts to ping the preferred gateway (using `PingRoutingStrategy` with `StaticRoutingStrategy`), and if that fails, it falls back to the specified fallback strategy. ```javascript const strategy = new PreferredWithFallbackRoutingStrategy({ preferredGateway: 'https://my-gateway.com', fallbackStrategy: new FastestPingRoutingStrategy({ timeoutMs: 500 }), }); ``` #### CompositeRoutingStrategy The `CompositeRoutingStrategy` allows you to chain multiple routing strategies together, providing maximum resilience by trying each strategy in sequence until one succeeds. This is ideal for complex fallback scenarios where you want to combine different routing approaches. **How it works:** 1. Tries each strategy in the order they're provided 2. If a strategy successfully returns a gateway, that gateway is used (remaining strategies are skipped) 3. If a strategy throws an error, moves to the next strategy 4. If all strategies fail, throws an error **Common use cases:** - **Performance + Resilience**: Try fastest ping first, fallback to random if ping fails - **Preferred + Network**: Use your own gateway first, fallback to AR.IO network selection - **Multi-tier Fallback**: Try premium gateways, then standard gateways, then any available gateway ```javascript import { createWayfinderClient, CompositeRoutingStrategy, FastestPingRoutingStrategy, RandomRoutingStrategy, StaticRoutingStrategy, NetworkGatewaysProvider, } from '@ar.io/wayfinder-core'; const ario = ARIO.init({ rpc: createSolanaRpc('https://api.mainnet-beta.solana.com'), }); // Example 1: Performance-first with resilience fallback const performanceWayfinder = createWayfinderClient({ routingStrategy: new CompositeRoutingStrategy({ strategies: [ // Try fastest ping first (high performance, but may fail if all gateways are slow) new FastestPingRoutingStrategy({ timeoutMs: 500, gatewaysProvider: new NetworkGatewaysProvider({ ario, sortBy: 'operatorStake', limit: 10, }), }), // Fallback to random selection (guaranteed to work if gateways exist) new RandomRoutingStrategy({ gatewaysProvider: new NetworkGatewaysProvider({ ario, sortBy: 'operatorStake', limit: 20, // Use more gateways for fallback }), }), ], }), }); // Example 2: Preferred gateway with multi-tier fallback const preferredWayfinder = createWayfinderClient({ routingStrategy: new CompositeRoutingStrategy({ strategies: [ // First, try your preferred gateway new StaticRoutingStrategy({ gateway: 'https://my-preferred-gateway.com' }), // If that fails, try fastest ping from top-tier gateways new FastestPingRoutingStrategy({ timeoutMs: 1000, gatewaysProvider: new NetworkGatewaysProvider({ ario, sortBy: 'operatorStake', limit: 5, // Only top 5 gateways }), }), // Final fallback: any random gateway from a larger pool new RandomRoutingStrategy({ gatewaysProvider: new NetworkGatewaysProvider({ ario, limit: 50, // Larger pool for maximum availability }), }), ], }), }); ``` # Telemetry (/sdks/wayfinder/wayfinder-core/telemetry) Wayfinder can optionally emit OpenTelemetry spans for every request. **By default, telemetry is disabled**. You can control this behavior with the `telemetry` option. ```typescript const wayfinder = createWayfinderClient({ // other settings... telemetrySettings: { enabled: true, sampleRate: 0.1, // 10% sampling exporterUrl: 'https://your-otel-exporter', clientName: 'my-app', clientVersion: '1.0.0', }, }); ``` # Verification Strategies (/sdks/wayfinder/wayfinder-core/verification-strategies) Wayfinder includes verification mechanisms to ensure the integrity of retrieved data. Verification strategies offer different trade-offs between complexity, performance, and security. | Verifier | Complexity | Performance | Security | Description | | ------------------------------- | ---------- | ----------- | -------- | ------------------------------------------------------------------------------------------------------------ | | `RemoteVerificationStrategy` | Low | Low | Low | Checks the `x-ar-io-verified` header from the gateway that returned the data. If `true`, the data is considered verified and trusted. | | `HashVerificationStrategy` | Low | High | Low | Computes the SHA-256 hash of the returned data and comparing it to the hash of a **trusted gateway** (_**recommended for most users**_). | | `DataRootVerificationStrategy` | Medium | Medium | Low | Computes the data root for the transaction (most useful for L1 transactions) and compares it to the data root provided by a **trusted gateway**. | | `SignatureVerificationStrategy` | Medium | Medium | Medium | - **ANS-104 Data Items**: Fetches signature components (owner, signature type, tags, etc.) from trusted gateways using range requests, then verifies signatures against the data payload using deep hash calculations following the ANS-104 standard.- **L1 Transactions**: Retrieves transaction metadata from gateway /tx/\ endpoints, computes the data root from the provided data stream, and verifies the signature using Arweave's cryptographic verification. | #### RemoteVerificationStrategy This strategy is used to verify data by checking the `x-ar-io-verified` header from the gateway that returned the data. If the header is set to `true`, the data is considered verified and trusted. This strategy is only recommended for users fetching data from their own gateways and want to avoid the overhead of the other verification strategies. ```javascript const wayfinder = new Wayfinder({ verificationSettings: { // no trusted gateways are required for this strategy enabled: true, strategy: new RemoteVerificationStrategy(), }, }); ``` #### HashVerificationStrategy Verifies data integrity using SHA-256 hash comparison. This is the default verification strategy and is recommended for most users looking for a balance between security and performance. ```javascript const wayfinder = new Wayfinder({ verificationSettings: { enabled: true, strategy: new HashVerificationStrategy({ trustedGateways: [new URL('https://turbo-gateway.com')], }), }, }); ``` #### DataRootVerificationStrategy Verifies data integrity using Arweave by computing the data root for the transaction. This is useful for L1 transactions and is recommended for users who want to ensure the integrity of their data. ```javascript const wayfinder = new Wayfinder({ verificationSettings: { enabled: true, strategy: new DataRootVerificationStrategy({ trustedGateways: [new URL('https://turbo-gateway.com')], }), }, }); ``` #### SignatureVerificationStrategy Verifies signatures of Arweave transactions and data items. Headers are retrieved from trusted gateways for use during verification. For a transaction, its data root is computed while streaming its data and then utilized alongside its headers for verification. For data items, the ANS-104 deep hash method of signature verification is used. ```javascript const wayfinder = new Wayfinder({ verificationSettings: { enabled: true, strategy: new SignatureVerificationStrategy({ trustedGateways: [new URL('https://turbo-gateway.com')], }), }, }); ``` # x402 Payments (/sdks/wayfinder/wayfinder-core/x402-payments) Wayfinder can be configured to work with the [x402 payment protocol](https://docs.ar.io/learn/gateways/x402-payments#what-is-x402) for paid gateway services and higher rate limits. This allows you to seamlessly make requests that may require payment without having to manually handle payment flows. To get started, install the `@ar.io/wayfinder-x402-fetch` package. The `@ar.io/wayfinder-x402-fetch` package is a simple wrapper of the [x402-fetch](https://github.com/coinbase/x402/tree/1d4c253aef959b73b46d42e7f0ccf30c7ce64955/typescript/packages/x402-fetch) library, which creates a fetch implementation to automatically handles x402 payment flows. You can use this fetch implementation with Wayfinder to enable x402 payments for your requests. ```javascript // Set up your wallet for x402 payments const privateKey = process.env.X402_PRIVATE_KEY; // Your private key const account = privateKeyToAccount(privateKey); // Create x402-enabled fetch implementation const x402Fetch = createX402Fetch({ walletClient: account, }); // Create Wayfinder client with x402 fetch to handle payments const wayfinder = createWayfinderClient({ fetch: x402Fetch, routingSettings: { // Configure to use x402-enabled gateways strategy: new StaticRoutingStrategy({ gateway: 'https://paid-gateway.example.com', }), }, }); // Requests will now automatically handle x402 payments const response = await wayfinder.request('ar://transaction-id'); ``` **How it works:** 1. When a gateway returns a `402 Payment Required` status 2. The x402 fetch automatically handles the payment flow 3. The request is retried with payment credentials 4. You get access to premium gateway services **Use cases:** - Higher rate limits on data requests - Access to premium gateway features - Supporting gateway operators through payments To learn more about x402 payments, visit the [x402 documentation](https://docs.ar.io/learn/gateways/x402-payments). # useWayfinderRequest (/sdks/wayfinder/wayfinder-react/(hooks)/usewayfinderrequest) Fetch the data via wayfinder, and optionally verify the data. ```tsx function WayfinderData({ txId }: { txId: string }) { const request = useWayfinderRequest(); const [data, setData] = useState\(null); const [dataLoading, setDataLoading] = useState(false); const [dataError, setDataError] = useState\(null); useEffect(() => { (async () => { try { setDataLoading(true); setDataError(null); // fetch the data for the txId using wayfinder const response = await request(`ar://${txId}`, { verificationSettings: { enabled: true, // enable verification on the request strict: true, // don't use the data if it's not verified }, }); const data = await response.arrayBuffer(); // or response.json() if you want to parse the data as JSON setData(data); } catch (error) { setDataError(error as Error); } finally { setDataLoading(false); } })(); }, [request, txId]); if (dataError) { return Error loading data: {dataError.message}; } if (dataLoading) { return Loading data...; } if (!data) { return No data; } return ( {data} ); } ``` # useWayfinderUrl (/sdks/wayfinder/wayfinder-react/(hooks)/usewayfinderurl) Get a dynamic URL for an existing `ar://` URL or legacy `arweave.net`/`arweave.dev` URL. Example: ```tsx function WayfinderImage({ txId }: { txId: string }) { const { resolvedUrl, isLoading, error } = useWayfinderUrl({ txId }); if (error) { return Error resolving URL: {error.message}; } if (isLoading) { return Resolving URL...; } return ( ); } ``` # Wayfinder React (/sdks/wayfinder/wayfinder-react) A set of React hooks and components for integrating Wayfinder, the decentralized data access system for Arweave. Wayfinder-react wraps the functionality of wayfinder-core in user-friendly React components and hooks, making it easy to integrate ar.io network functionality into your React applications with built-in loading states, error handling, and caching. ## Quick Start ### Install Wayfinder React ```npm npm install @ar.io/wayfinder-react @ar.io/wayfinder-core @ar.io/sdk ``` ### Install polyfills (required for web environments) Crypto polyfills are required for web environments due to the use of `crypto`, `buffer` and `process` types in wayfinder-react dependencies (i.e. `arbundles`). ```npm npm install --save-dev vite-plugin-node-polyfills ``` ```js // vite.config.js plugins: [ nodePolyfills({ globals: { Buffer: true, global: true, process: true, }, }), ], }); ``` Configure your bundler (Webpack, Vite, Rollup, etc.) to provide polyfills for `crypto`, `process`, and `buffer`. Refer to your bundler's documentation for polyfill configuration. ### Setup the provider ```jsx // App.tsx function App() { return ( ); } ``` ### Use the available hooks ```typescript function WayfinderImage({ txId }: { txId: string }) { const { resolvedUrl, isLoading, error } = useWayfinderUrl({ txId }); if (error) { return Error resolving URL: {error.message}; } if (isLoading) { return Resolving URL...; } return ( ); } ``` ## Next Steps } title="Hooks Reference" description="Comprehensive guide to all available React hooks" href="/sdks/wayfinder/wayfinder-react/usewayfinderrequest" /> } title="Wayfinder Core" description="Learn about the underlying core library" href="/sdks/wayfinder/wayfinder-core" />