The vSAN Cost of Surviving Double Drive Failure

Every storage system faces the same risk. A second drive failure that arrives before the first failed drive finishes rebuilding can take data offline and turn a hardware problem into a restore-from-backup event. The odds are statistically low, but the consequences are severe enough that many IT professionals configure their environments to survive a double drive failure. That protection has always cost money, and the bill is climbing. AI demand for memory and flash is driving data center inflation, with price increases of up to 5X on some flash media, and every extra drive a protection scheme consumes now costs far more than it did when the cluster was designed.

Key Takeaways

• Double drive failure protection in vSAN ESA means RAID 6, VCF licensing and at least six NVMe hosts, with seven needed in practice.
• RAID 6 rebuilds put parity math and data movement on the same servers running production VMs, so IT trades application performance against rebuild speed.
• ESA stops at two failures. Server-side designs make a third failure more likely when hosts go down for maintenance or a fault.
• Remote parity uses the copy of data already kept at the DR site to supply missing blocks inline, on lower-cost media such as hard drives.
• Remote parity works only when one platform coordinates the VM, storage, cache, snapshots and replication.

Double drive failure protection is especially problematic for server-side storage like vSAN. There are three ways to deliver it. A replication factor of three (RF3) keeps three full copies of every block, which has a high capacity cost that data center inflation makes higher still. Distributed RAID or erasure coding eases the capacity cost, but it raises hardware requirements and adds I/O load to servers already sharing CPU, memory and network with the applications running in their virtual machines.

Key Terms

Double Drive Failure

A second drive failure that occurs before the first failed drive finishes rebuilding. Without protection for two failures, it takes data offline.

Replication Factor 3 (RF3)

A three-way distributed mirror that keeps three full copies of every block, at three times the raw capacity.

RAID 6 (4+2)

An erasure-coded layout with four data components and two parity components. It survives two failures at 1.5 times the raw capacity and needs at least six hosts in vSAN.

Performance Leg

The mirrored component vSAN ESA adds to each RAID 5 or RAID 6 object to absorb incoming writes before they reach the parity stripe. At two failures to tolerate, it holds three copies.

Remote Parity

A protection method that uses a platform-managed copy of data, at the DR site or local, to supply missing blocks to production in real time after a multi-drive or multi-server failure.

The third option, remote parity, is less obvious. It treats the copy of data already sitting at the disaster recovery (DR) site as a parity source and feeds missing blocks from it back into production. Remote parity is parity by role rather than by math. It serves real blocks from a copy instead of recalculating them from a stripe, so a failure puts no parity calculations on production CPUs. This article looks at all three options.

How vSAN ESA Protects Against Double Drive Failure

vSAN Express Storage Architecture (ESA) handles double drive failure with RAID 6, a 4+2 erasure-coded layout. The capacity overhead is only 1.5X, so it looks good on paper next to the 3X of a triple mirror. ESA also addressed the write penalty that used to come with RAID 6 on vSAN. Each object includes a mirrored “performance leg” that absorbs incoming writes before they are written to the parity stripe, and at two failures to tolerate that leg holds three copies.

The performance leg draws its space from the same NVMe drives as the capacity leg, so it needs no dedicated cache tier, but it still consumes flash. ESA supports only NVMe, the drive type hit hardest by the current flash and memory supercycle.

The Cost Before You Buy a Single Drive

The performance gains in ESA come with a severe price. The first cost is licensing. Most vSAN customers move to VMware Cloud Foundation (VCF) to get ESA and the vSAN capacity entitlement they need, and that move can raise infrastructure licensing costs by 5X or more. High-capacity environments pay again once they exhaust the relatively low per-core capacity entitlement included with VCF.

Customers running vSAN Original Storage Architecture (OSA) remain supported, but new vSAN development is going to ESA. Higher standard VMware pricing, much higher VCF pricing and a stalled OSA roadmap have many of these customers looking for a VMware alternative. For vSAN customers, that search quickly becomes a search for a vSAN alternative, since the storage leaves with the hypervisor. OSA customers who stay and want double drive protection live with the write penalty of OSA’s RAID 6 implementation. They also live with the OSA disk group design, where each group has a single cache drive. If that cache drive fails, the entire disk group goes offline.

The Hardware Bill

Beyond licensing, protecting against double drive failure in vSAN ESA means buying more servers. RAID 6 requires a minimum of six hosts for the 4+2 layout. Seven is the practical number, so there is a spare host to rebuild onto and return the environment to a fully protected state after a server failure. A midsize enterprise running four or five hosts has to grow the cluster, at today’s NVMe prices, just to get double drive protection.

Even then, ESA stops at two. A third concurrent failure exceeds what RAID 6 tolerates. Triple failures are rarer than double failures, but they happen, and they are more likely in server-side storage designs, where two or three hosts down for maintenance or a hardware fault take every drive in those hosts offline at once.

The Rebuild Problem

OSA and ESA both have issues from a rebuild perspective. The 1.5X capacity figure, a 50% savings over a triple mirror, means nothing in a failed state. Rebuilds take longer, and the network absorbs heavy traffic as data moves between hosts. CPU utilization climbs as the cluster recalculates parity on the same servers running production VMs. The load is high enough that IT often has to choose between acceptable application performance and a fast rebuild that returns the environment to a fully protected state.

The Not So Obvious Alternative

At face value, a distributed mirror is the wrong choice in an era of data center inflation. It costs another 0.5X in capacity over RAID 6. For that capacity the organization gets faster rebuilds and no parity math competing with production applications. A mirror on its own protects against only a single drive or server failure, though. Beyond that, there is a loss of data access, and IT will likely need to restore from backup.

Most server-side storage systems also offer RF3, a three-way distributed mirror. It fits some workloads, but at 3X capacity it is hard to justify for most of them at today’s flash prices.

The better answer is remote parity, which uses a copy of data IT has to create anyway and has that copy supply missing blocks to production in real time. The platform itself has to create and manage this copy, so it knows which data went missing because of the failure and where to find it. An external replication or backup tool lacks the access to the storage layer to serve blocks inline to a running VM.

The ideal location for that copy is the DR site. The organization already has, or should have, a DR site with data replicating to it. If the storage software can read that data in real time, it can return blocks to production inline. The copy adds no new capacity cost, and the only penalty is the latency of pulling blocks across the wire. Those are small block transfers, limited to the blocks applications or users request at that moment. If latency is a concern, the same repository can also run locally to remove the network hop.

Remote parity protects beyond dual drive failure. It covers multiple drive failures and even full server failures that exceed a triple mirror or a +3 erasure code. It also brings media flexibility. vSAN RAID requires its extra capacity from like drives, and in ESA that means NVMe SSDs selling at an absolute premium. The remote parity copy can sit on hard drives instead of flash, at the DR site or locally, driving costs down further.

Requirements for Remote Parity

The number one requirement is a robust snapshot mechanism that takes snapshots frequently and pushes them to the DR site without affecting production performance. The second is an efficient RAM cache that holds data changes between replication cycles. Efficiency matters more than cache size. RAM is also selling at a premium.

The storage software also has to know where to look. When a VM requests data that is no longer on the primary or secondary copy, the software checks the RAM cache first and then fetches the block across the wire. A distributed mirror is required as well. It spreads copies of data across all available drives, so even in a multi-drive failure much of a VM’s data remains available locally, which limits how much has to come from the remote parity system.

All of this requires tight coordination across the VM, the local storage, the cache, data distribution across drives, snapshots and replication. Linking those through separate applications, even from one vendor like VMware with ESXi, vSAN, NSX and a separate replication product, adds too much latency and overhead.

VergeOS and ioGuardian as an Example

Full disclosure: I founded Storage Switzerland, and today I work at VergeIO, the company behind VergeOS.

VergeOS from VergeIO is a working example of remote parity. VergeOS runs the hypervisor, VergeFS distributed storage and networking in a single code base, so the storage layer knows which VM owns each block and where every copy lives. VergeFS protects data with a distributed mirror across all drives in the instance, with global inline deduplication and frequent, low-impact snapshots.

ioGuardian is a separate VergeOS instance, at the DR site or local, that holds a deduplicated copy of production data. When a multi-drive or multi-node failure leaves VergeFS without a block, VergeFS requests that block from ioGuardian inline, and the VM keeps running through the repair. ioGuardian runs on lower-cost media, including hard drives. VergeIO describes the process in more detail in Surviving Cascading Drive Failure.

StorageSwiss Take

Double drive failure protection has become one of the most expensive parts of a vSAN environment. It means VCF licensing, six or seven NVMe hosts, and rebuilds that compete with production for CPU and network. Remote parity gets protection beyond two failures from a copy of data the organization already maintains for DR, on cheaper media, with no parity math running on production servers. vSAN customers planning their VMware exit should add it to their evaluation criteria.

To see the exit paths open to vSAN customers compared in detail, along with a live demonstration, join the TruthInIT webinar Leaving VMware Means Leaving vSAN: Your VMware Exit Is Really a Storage Decision on Thursday, November 12, at 1:00 PM ET. Register here.

Frequently Asked Questions

How does vSAN ESA protect against a double drive failure?

ESA uses RAID 6, a 4+2 erasure-coded layout, with a mirrored performance leg of three copies that absorbs writes before they reach the parity stripe. Both live on NVMe drives.

How many hosts does vSAN RAID 6 need?

At least six for the 4+2 layout. Seven is the practical number, so a spare host is available to rebuild onto after a server failure.

Why are RAID 6 rebuilds a problem in server-side storage?

The cluster recalculates parity and moves rebuild data on the same servers and network that run production VMs, so IT has to choose between application performance and a fast return to full protection.

What is remote parity?

Remote parity uses a platform-managed copy of data, usually the one already kept at the DR site, to supply missing blocks to production inline after a multi-drive or multi-server failure. It plays the role of parity by serving real blocks instead of recalculating them.

Can the remote parity copy run on hard drives?

Yes. The copy serves only the blocks a failure leaves missing, so it can sit on lower-cost hard drives at the DR site or locally instead of NVMe flash.