Latest funding of 140 million dollars, interpreting the Decentralization storage protocol Walrus

Developed by the Sui team, an independent PoS chain, new governance token WAL, potential airdrop opportunity

Written by: Alex Liu, Foresight News

This article was originally published on September 25, 2024.

The decentralized storage network Arweave launched its computing layer AO, successfully causing a rebound in the price, ecosystem, and popularity of AR coin, which can be seen as a turnaround victory. As a general-purpose computing chain, what kind of waves will Sui create by launching the decentralized storage network Walrus?

Background Introduction

Team

The development company behind Solana is called Solana Labs, the development company behind Aptos is called Aptos Labs, and the development company behind Sui is called Mysten Labs (that's how unique it is). Most of the founders and employees of Mysten Labs come from the blockchain project Diem, which was dissolved by Facebook (now Meta).

Walrus is the latest product classified as "protocol, platform" by Mysten Labs, and it is a decentralized storage network. The English meaning of Walrus is "sea elephant", and its official website has slogans such as "growing strong like a walrus" and "adapting powerfully like a walrus", conveying the reliability and usability of the protocol as a storage system.

The connection with Sui

Walrus is built on Sui and utilizes Sui to coordinate the sale of storage space and metadata. However, using Walrus does not require building applications or products on Sui, and the brand new governance token WAL will serve as a Utility Token, not SUI.

Competitive Product Comparison

Decentralized storage protocols are generally divided into two major categories. The first category is fully replicated systems, with the main competitors in this field being Filecoin and Arweave, which are typical representatives of this type of system. The main advantage of this type is the complete file availability on storage nodes, which allows for easy access and migration of files even if a storage node goes offline. This setup creates a permissionless environment because storage nodes do not need to rely on each other to recover files.

The reliability of such systems depends on the robustness of the selected storage nodes. Under the classic one-third static adversary model and the assumption of an infinite pool of candidate storage nodes, achieving "twelve nines" security (i.e., a probability of losing file access of less than 10^-12) requires storing more than 25 replicas on the network. This results in a 25-fold storage overhead. In addition, there is the potential issue of a Sybil attack, where malicious actors can pretend to have multiple replicas of stored files, undermining the integrity of the system.

The second type of decentralized storage service uses Reed-Solomon (RS) encoding. RS encoding divides the file into smaller parts, called slices, with each slice representing a portion of the original file. As long as the total size of the slice is greater than the size of the original file, the original file can be decoded. RS coding also has its drawbacks. The encoding and decoding processes rely on domain manipulation, polynomial evaluation, and interpolation, which are computationally expensive. These operations are only practical if the size of the domain and the number of slices are relatively small, limiting the size of the encoded file and the number of participating storage nodes, which would otherwise become very expensive and limit the degree of decentralization. Another problem is that when a storage node goes offline and needs to be replaced by another, unlike a fully replicated system, data cannot simply be replicated from one node to another. The RS-encoded system requires all existing storage nodes to send their tiles to a replacement node, which then recovers the missing tiles. However, this process causes O(|blob|) of data to travel over the network. Frequent restore operations reduce the storage savings that can be achieved through reduced replication.

Challenges Facing Storage

Regardless of the replication protocol used, all existing decentralized storage systems still face two additional challenges:

1. Continuous challenges are needed to ensure that storage nodes retain data without discarding it. This is crucial in an open decentralized system that provides storage payments, but currently, this practice limits the scalability of the system because each file requires a separate challenge.
2. Storage nodes need to coordinate: they need to know who is in the system, which files have paid storage fees, implement participation incentive mechanisms, and manage challenges as well as mechanisms to mitigate abuse. This is why each of the above systems has implemented a custom blockchain to execute transactions and introduced cryptocurrency outside of the storage protocol.

Core Innovation

In the face of these challenges, what innovations does Walrus bring that can offer different solutions for decentralized storage?

In simple terms:

By employing innovative erasure coding technology, Walrus is able to quickly and robustly encode unstructured data blocks into smaller shards, which are stored distributedly across a network of storage nodes. Even if up to two-thirds of the shards are lost, the original data block can be rapidly reconstructed using partial shards. This becomes possible while maintaining a replication factor of only 4 to 5 times, comparable to existing cloud services, with the advantages of decentralization and greater fault tolerance.

Specifically:

Walrus has launched RedStuff, a brand new 2D coding algorithm, specifically designed for Byzantine Fault Tolerance. RedStuff is based on fountain codes, combining the advantages of fast operation and high reliability.

RedStuff encodes data into primary and secondary slices through simple operations (mainly XOR, exclusive OR operation). These slices are distributed across storage nodes, with each node holding a unique combination. For encoding in different dimensions, RedStuff uses different thresholds. The primary dimension adopts a recovery threshold of f+1, which allows for asynchronous writing, as only 2f+1 signatures are needed to prove the availability of the data block, which has already formed a replication factor of 3.

The second dimension uses a recovery threshold of 2f+1, which achieves asynchronous storage proof for the first time while only introducing 1.5 times additional replication, resulting in a final total replication factor of less than 5 times. More importantly, lost slices can be restored based on the amount of lost data, thereby saving bandwidth, all thanks to 2D coding.

The advantages of RedStuff include: Compared to RS coding, using simple XOR operations allows for faster encoding/decoding speeds; due to low storage overhead, the system can scale to hundreds of nodes and possesses high elasticity and fault tolerance, ensuring data recovery even in the event of Byzantine failures.

As a permissionless protocol, Walrus is equipped with an efficient committee reconfiguration protocol to address the natural attrition of storage nodes, ensuring the continuous availability of data. When a new committee replaces the current committee between two epochs, the reconfiguration protocol ensures that all data blocks that have exceeded the Point of Availability (PoA) remain accessible. RedStuff's 2D encoding makes state migration more efficient, allowing other nodes to recover lost slices even if some nodes are unavailable.

Node 1 and Node 3 help Node 4 recover slice data

Walrus introduces an asynchronous challenge protocol to verify whether nodes correctly store data. This protocol allows for efficient storage proofs, ensuring data availability without relying on network assumptions, with costs scaling logarithmically with the number of stored files.

The economic model of Walrus is based on staking, combining reward and penalty mechanisms. The innovative storage certification mechanism expands logarithmically with the number of stored files, reducing the cost of proving file storage.

In summary, Walrus is built around the RedStuff protocol, offering a scalable, resilient, and economically viable decentralized storage solution that can provide high authenticity, integrity, auditability, and availability at a reasonable cost.

All of this is thanks to Sui as the control layer of Walrus. Having a scalable, programmable, and secure infrastructure as the coordination layer allows it to focus on the core issues of decentralized storage.

Potential Airdrop

Walrus will launch an independent Token WAL, with Utility including staking and governance. How can one obtain the WAL Airdrop? Refer to the method of obtaining AO; holding SUI may be one of the ways.

Walrus is expected to launch its testnet soon, with the mainnet launch date yet to be determined. Currently, you can visit the official documentation to learn how to use Walrus to deploy your own website.

Source:

Walrus White Paper:

Walrus: A decentralized storage and DA protocol that can build L2 and large storage based on Sui:

Mysten Labs researcher X thread: