What is Scale-Out NVMe?

The first step for NVMe flash is to use them internally in servers and storage systems. However, there is another aspect of NVMe, NVMe over Fabrics (NVMe-F) that promises high-speed communication between devices and also promises a new way to scale storage infrastructures.

In the same way that iSCSI and Fibre Channel (FC) transmit SCSI over IP of FC networks today, NVMe-F promises to do the same with the NVMe protocol. Both Ethernet and FC networks are continually increasing the overall speed and quality of the connection. That means they can take advantage of the increased command set and queue depth of NVMe, especially since those architectures will communicate with NVMe storage. The problem is that most data centers don’t need this level of performance just yet. But, there is an area of connectivity that does, intra-connectivity between storage systems.

Understanding Scale-Out Architectures

Most storage systems today offer some form of scale-out capability. These are typically presented as a cluster, which is coupled either loosely or tightly. A loosely coupled cluster means that the data is intact on a single storage system (or node) and almost exclusively that node handles data protection. The storage software seamlessly manages multiple storage systems and can easily move workloads between systems as needed. A tightly coupled cluster segments data and stripes it across nodes in the cluster while the cluster handles data protection. It either creates two (or three) copies of each segment or makes sure the segments are on different nodes, or it stripes segments across the nodes and generates the parity.

The loosely coupled cluster can scale more efficiently as each node can be taken to its maximum capacity before another node is added. It also can scale with different types of nodes. For example, there could be nodes that provide very high performance while others provide very high capacity. The software could automatically place workloads on the right node based on that workload’s IO profile.

The Scale-Out Challenge

The problem with scale-out designs is the amount of traffic required for intra-node communications. Tightly coupled clusters are constantly sending small bits of information to other nodes, but loosely coupled clusters, while they aren’t constantly sending data to other nodes, are sending larger amounts when they do. The responsiveness of the intra-node communication is critical to the user experience.

Most of these communications between nodes are done via traditional IP protocols, rather than storage protocols. The efficiency of the transfers, especially when the nodes involved are all-flash, will increase latency, which limits effective node growth.

NVMe-F for Scale-Out Storage

NVMe-F changes the game for intra-storage node communications. Instead of an IP protocol designed to send emails, it uses a storage protocol designed specifically for memory transfers. The efficiency of these transfers means data can move through the various nodes in the cluster as efficiently as an internal transfer can. The result should be much greater scale.

Scale-out of the Future Powered by NVMe

This highly efficient networking should allow storage system vendors to design dynamic storage environments that are created on the fly, as an application needs resources. Imagine a fabric of loosely coupled components; racks of storage processors, high-performance NVMe drives and high capacity SSD drives. As a default, the application data may sit on the high capacity SSD drives, then during a busy day performance demands increase. As the demand increases, the storage software seamlessly moves data to the high-performance drives and begins to assign more storage processors to the task. Once the demand increase subsides, the data moves back to high capacity SSDs, reducing the number of processors assigned to the task. Essentially, they are returned to the fabric for use by another application.

The thought of building up and tearing down a storage system on the fly, as demand warrants, is untenable with current networking technology. However, with NVMe-F, where components are connected, at speeds and latencies that rival internal connections, the fabric approach becomes very realistic.


Storage systems with NVMe drives are already available on the market, increasing storage performance while lowering latency. The next step is to use NVMe to not only solve the limitations of scale-out storage, but also to lay the foundation for a storage fabric that dynamically shifts as workloads demands.

Sponsored by Tegile

Watch On Demand


PREVIOUS BLOG: How Will All-Flash Need to Change to Support NVMe?

NEXT BLOG: What is End to End NVMe?

George Crump is the Chief Marketing Officer at VergeIO, the leader in Ultraconverged Infrastructure. Prior to VergeIO he was Chief Product Strategist at StorONE. Before assuming roles with innovative technology vendors, George spent almost 14 years as the founder and lead analyst at Storage Switzerland. In his spare time, he continues to write blogs on Storage Switzerland to educate IT professionals on all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought-after public speaker. With over 30 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS, SAN, Virtualization, Cloud, and Enterprise Flash. Before founding Storage Switzerland, he was CTO at one of the nation's largest storage integrators, where he was in charge of technology testing, integration, and product selection.

Tagged with: , , , , , , , , ,
Posted in Blog

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 25,514 other subscribers
Blog Stats
%d bloggers like this: