PaulRosca PaulRosca

Retrofitting arweave-js for Streaming Uploads

NOTE: The code documented in this gist is now available as a package here.

Since the release of the v2 transaction format, Arweave theoretically supports data transactions of infinite size. However, common interfaces for creating Arweave transactions such as arweave-js are limited in the size of transactions they can create due to their implementation requiring all transaction data be eagerly loaded into memory. This bounds them to their runtime memory limitations (eg. approx. ~2GB for Node) and prevents complete utility of Arweave's capabilities.

This document outlines some work to make arweave-js support creating transactions in a streaming manner, unbounded by runtime memory limitations.

Beyond Scope:

Browser support: Browsers and NodeJS do not have a common Streams API. A browser implementation of the below using this should not differ

	//----------------------------------------------------------------
	// ------------------- ServerSide(NODE.JS) -------------------
	//----------------------------------------------------------------
	function encrypt(message){
	const KEY = crypto.randomBytes(32)
	const IV = crypto.randomBytes(16)
	const ALGORITHM = 'aes-256-gcm';

	const cipher = crypto.createCipheriv(ALGORITHM, KEY, IV);
	let encrypted = cipher.update(message, 'utf8', 'hex');