Scale-out storage systems tend to protect data from media failure in one of two ways; they either use replication or erasure coding. Replication creates a user-defined number of copies of data, typically a minimum of three, as it is created or updated and replicates those copies to other nodes in the cluster. The concern with replication is its consumption of storage, generally 3X the core capacity. The other option is erasure coding which is a parity based protection scheme that stripes data across the cluster’s nodes. Erasure coding only requires 1.5X to 1.75X of additional capacity for copies to safeguard a customer’s data against multiple node failures.
Erasure coding, at first glance, seems like the obvious choice. It offers the same level of protection and doesn’t consume the same amount of capacity. Erasure coding, however, does have several challenges that IT professionals need to be aware of before deciding on the technology.
Erasure Coding Performance Problems
The biggest challenge with erasure coding is the performance toll it takes on the storage system. Each segment of data written to the storage system needs to have the erasure coding algorithm run on it and then the storage system needs to write the sub-segments plus parity information across every node in the cluster. Erasure coding also extracts a heavy toll on CPUs which can impact performance.
Secondary storage systems, where erasure coding first appeared, are less susceptible to the performance impact of erasure coding. First, performance is not the top concern for these systems. Second, these systems don’t typically see the same level of write IO that production systems experience.
Implementing erasure coding on a scale-out primary storage system can impact performance, and it can especially increase latency. If the storage software uses replication, then it only has to copy data to the user-defined number of nodes. Since there is no parity calculation, the CPU requirements are significantly lower.
Erasure coding also means that every read IO has to come from multiple nodes in the cluster. Typically, the scale-out architecture funnels the data sub-segments together, and then data is sent through the node that the application is accessing. If the storage software is efficient, the software can direct the accessing client or application to one of the nodes that have the complete data set, thus lowering the amount of network activity required for a read IO.
Finally, if there is a media or node failure, erasure coding creates another performance problem because parity is recalculated on the fly as data is accessed until a background rebuild completes. The system also has to balance processing power, and available IO allocated to recovering from the failure versus serving IO requests. Replication only has to make a copy of the impacted data from a known good node to a node with available capacity.
Erasure Coding Expansion Problems
Another challenge with erasure coding occurs when IT adds a node to the storage cluster; the storage software includes the new nodes in its software stripe, forcing a time and resource consuming rebalancing of data. If the storage cluster is using replication, then the storage software can choose to use the new node for new data, or it can rebalance data from nodes that are nearing capacity.
Over the years, as scale-out architecture expands, the new nodes have new processing and storage capabilities. Scale-out systems with erasure coding tend to treat all nodes and media equally. They can’t differentiate between nodes. However, replication combined with node-aware storage software can differentiate. It can use this intelligence to put first copies of performance demanding volumes on the best performing nodes and volumes that require less performance on older or lower performing nodes.
StorageSwiss Take
Erasure coding only provides a capacity advantage over replication, but the ever decreasing cost of storage combined with data efficiency techniques like deduplication are making the capacity savings less important. Replication enables storage software vendors to extract maximum performance from the underlying hardware while minimizing network traffic. It also enables them to be more “node-aware” which allows the intelligent placement of volumes based on access characteristics.
Scale-out primary storage architectures are a vital part of data center modernization but they need to extract maximum performance from each node and media in those nodes to deliver the performance that modern, and traditional applications expect. To learn more about the impact of storage on data center modernization watch our on demand webinar “Overcoming the Storage Roadblock to Data Center Modernization.”
George Crump is the Chief Marketing Officer at VergeIO, the leader in Ultraconverged Infrastructure. Prior to VergeIO he was Chief Product Strategist at StorONE. Before assuming roles with innovative technology vendors, George spent almost 14 years as the founder and lead analyst at Storage Switzerland. In his spare time, he continues to write blogs on Storage Switzerland to educate IT professionals on all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought-after public speaker. With over 30 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS, SAN, Virtualization, Cloud, and Enterprise Flash. Before founding Storage Switzerland, he was CTO at one of the nation's largest storage integrators, where he was in charge of technology testing, integration, and product selection.
StorageSwiss.com - The Home of Storage Switzerland 