Replication and Erasure Coding for Object Storage

The simplest form of protection for data redundancy is replication. Replication uses copy-based data protection where complete copies of a piece of content are made and distributed across nodes or sub-clusters. Because replication stores data contiguously on disk once the first bit is identified, content is delivered rapidly and efficiently without the need for rehydration.

Replication can be as simple or complex as required, allowing you to create an offsite disaster recovery (DR) cluster or even create N-way replication for collaboration and data locality. Since data is replicated on a domain-by-domain basis, you can choose what data to replicate and to where. Both synchronous and asynchronous replication are supported in Swarm.

One or more additional copies of an “original” object are created and maintained to be available should the original somehow become lost. For example, even in a catastrophic disk failure, the entire Swarm cluster participates in an active recovery that quickly restores the cluster to full replication in a relatively short amount of time. By making the recovery fast, Swarm shortens the window of time during which another disk failure might impact the cluster. If you choose three replicas for an object, the two remaining replicas effectively guarantee that a third replica is rapidly made.

Erasure coding provides enterprise-grade data protection at a lower storage footprint, and is ideal for moderately sized files and large content stores. Using erasure coding in Swarm breaks a file into multiple data segments and then computes parity segments. This results in a total number of segments that use fewer capacity and operational resources when compared to an additional copy.

Files are split into multiple data segments (K) and additional parity segments (P) based on the content of the data segments. This results in (M) total segments (K + P = M) being distributed to M different Swarm nodes or sub-clusters.

Erasure coding is similar to RAID on a per-object level. Should any number of drives fail, all of the nodes in Swarm work in parallel to heal the situation as quickly as possible, so data is fully protected once again. You can have data segments on multiple drives and parity segments on multiple drives. The cluster can be configured to meet any requirements for uptime.

Swarm also analyzes the health of the drives to see if there are any issue to ensure data is copied and protected proactively. This process can detect potential issues before a drive fails.