How Will All-Flash Need to Change to Support NVMe?

There are three points within the storage infrastructure that will use NVMe; internal to the storage system, intra-connectivity between storage systems and inter-connectivity to physical servers. The first point of adoption will be in the storage system itself. As vendors refresh the hardware, on which they run their storage software, part of that refresh will be to leverage internal NVMe. Utilizing NVMe in the storage system is the most seamless of the three points and provides the most immediate return.

Why NVMe in the Storage Server

The storage server, even with all-flash systems, eventually becomes the choke point. Server counts can scale and virtual machines on those servers can increase in density all creating additional IO. Each server can have multiple network connections and switches can scale to support all those connections. However, on the other side of the network switch is the storage system, which has a limited amount of networking resources.

The network shortage though is not the network connectivity from the storage system to the switch; it is internal to the storage system itself. Traditionally this network is based on serial attached SCSI (SAS), which has a limited amount of commands and queues it can support.

A SAS based all-flash array essentially has to process each IO serially. While flash allows this serial IO processing to occur very fast, it is still done one at a time, which means even increases in SAS bandwidth won’t make much impact. NVMe, on the other hand, can process thousands of IOs at the same time, which means with the same bandwidth it can process more IO requests from multiple hosts faster than SAS.

Since most data centers have an increasingly mixed IO workload with plenty of parallel IO streams, they are ideal for NVMe. Most all-flash environments, as they refresh to NVMe-based systems, will likely see an increase in per workload performance if that workload is IO intensive. More importantly, these environments will see an increase in their ability to scale their data centers further with the same amount of storage systems.

How to put NVMe in the Storage Server?

Most all-flash vendors are making a clean break from SAS-based all-flash to pure NVMe-based systems. Because NVMe drives and components are more expensive, they will continue to keep SAS in their product lines so they can support customers that don’t need the extreme performance of NVMe. As a result, many data centers may end up with two systems, NVMe-based all-flash for IO intensive workloads and a SAS-based all-flash array for everything else.

An alternative is to create a new style of hybrid array. Hybrid arrays were popular in the early days of flash. They integrated flash-based storage with hard drive based storage automatically moving data between the two storage types and IO demand as required.

A modern implementation of the hybrid design is to mix NVMe based flash storage with SAS-based flash storage. The advantage is the vendor can deliver a single system that cost-effectively meets the demands of all workloads. They can also load up the SAS-based storage with more capacity than the SCSI connection will support, since most of the actual IO will be on the NVMe, and the SAS SSD’s use is for less active data.

The concern over Flash/HDD hybrid designs was the performance impact if the data requested was not on flash but instead on hard disk. The potential for a significant variation in performance is what lead many organizations to adopt all-flash. The NVMe/SAS hybrid designs have no such concern since not only are both media types flash, the SAS won’t be as busy since it is no longer responsible for all the IO. That means a request for the SAS tier will have almost private access to the SAS bus and it will respond to the IO request instantly because of the flash media.


NVMe in the storage server will be most data center’s first encounter with NVMe. While those organizations may not need the added NVMe performance on a per-workload basis, an NVMe-based system should allow them to scale the storage system across more workloads with a greater mixture of IO types. It is important though that all-flash vendors design these systems so that they are affordable today and applicable across the entire data center. Organizations don’t want another niche product that can only solve one problem.

Sponsored by Tegile

Watch On Demand

Previous Blog: Understanding NVMe             Next Blog: What is Scale-Out NVMe?

Twelve years ago George Crump founded Storage Switzerland with one simple goal; to educate IT professionals about all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought after public speaker. With over 25 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS and SAN, Virtualization, Cloud and Enterprise Flash. Prior to founding Storage Switzerland he was CTO at one of the nation's largest storage integrators where he was in charge of technology testing, integration and product selection.

Tagged with: , , , , , , , ,
Posted in Blog

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 22,133 other followers

Blog Stats
%d bloggers like this: