NVMe is a new protocol that most data centers will use within the next several years in addition to SCSI. The path to NVMe will not require an infrastructure overhaul. Instead it will be a gradual, almost seamless process. And even when the data center is mostly NVMe most data center networks will still also transport a lot of SCSI traffic.
NVMe Refresher
NVMe is a protocol designed specifically to communicate with solid state storage devices like flash. SCSI, its predecessor, was designed for hard disk drives, which means it would only queue a limited amount of IO and only send the appropriate amount of IO commands. Since flash drives are much lower latency than hard disk drives, they can process much higher queue depths and IO commands and as a result were constantly waiting on the protocol itself to feed them more work to do. NVMe solves this problem by increasing both queue depth and the number of commands.
The Path to NVMe for Today’s Data Center
Most data centers are enjoying a performance surplus thanks to the introduction of flash into their storage architectures. For the most part flash solves the immediate storage performance problem. But with the addition of workloads and as developers take advantage of the available flash performance, bottlenecks will become apparent.
The first step down the path to the NVMe data center is for flash arrays to use the protocol as an upgrade to their internal connectivity. Internally most storage systems, today, use Serial Attached SCSI (SAS) to communicate with the flash drives. While 12Gbps flash provides adequate bandwidth, it is hampered by the latency within the SCSI protocol. NVMe is PCIe connected and take full advantage of the low latency of flash.
Flash arrays using NVMe is the logical first step. IT will add additional workloads to the flash array long before developers have a chance to optimize code for flash storage. The increase in workloads means more physical systems, running more virtual machines will connect into the storage infrastructure. All that IO will land on the storage system and it is the first place the data centers are likely to see a bottleneck. Using NVMe internally alleviates that bottleneck.
The next step for NVMe is for the protocol to step outside of the flash array. The means leveraging an extension to NVMe called NVMe Over Fabrics (NVMe-F). Most storage networks whether they are IP based or SCSI based communicate via SCSI, which means it also has the same latencies described above. NVMe-F removes those latencies and enables the storage network to be almost as efficient as internal storage.
The first part of stepping out of the storage system will be for scale-out storage systems to use NVMe-F to provide high speed, efficient communication between nodes. Clustered storage systems transmit a lot of IO in the form of data and metadata between nodes within the cluster to make sure that everything stays in sync and is well protected. These networks can also bottleneck down and increase latency. NVMe-F should enable them to scale further and more efficiency than in the past.
The second part of stepping out of the storage system will involve interconnecting the storage system to the physical hosts. Today, most applications can’t generate enough IO to max out standard communications. But as those applications are optimized for flash and as the number of virtual machines per physical server increases then the IO capabilities of the server to storage interconnect will become an issue.
The communication between physical server and the storage system is not typically bandwidth constrained. For most environments, there is plenty of bandwidth available and for those that are constrained there are several bandwidth increases coming in both FC and IP. The issue is, again, the efficiency of that communication. NVMe-F allows the physical server to “load up” the network with enough data and IO commands to fully utilize the increased bandwidth.
Timing the NVMe Transition
Many data centers are ready for the first NVMe step – internal storage system NVMe connectivity – right now. The challenge is there is a price premium for implementing NVMe and most data centers don’t need all their workloads hosted on an NVMe based all-flash array. Look for systems that utilize both NVMe and SAS (where the NVMe drives and network are used as a high performance tier that front ends SAS attached) high capacity SSD drives.
The step to using NVMe-F to interconnect multiple scale-out storage systems is likely to occur within the next six to eight months. Some data centers are experiencing IO bottlenecks within the scale-out cluster and more data centers are heading that direction. Within eight months there will be a pent up demand for high performance, efficient connectivity between storage nodes.
While NVMe will still be in its infancy at the point that scale-out storage systems will start to leverage it, the use case will be very controlled. The scale-out storage vendor will provide the network turnkey using specific adaptors that it is confident work and it can easily support. This initial use case of NVMe-F is important for the industry though as it lays the foundation for the final transition to end-to-end connectivity.
The step to end-to-end connectivity where NVMe physical hosts are attaching to an NVMe scale-out storage cluster equipped with nodes that have internally connected NVMe drives is about two to three years out both in terms of need and the technology being ready.
StorageSwiss Take
NVMe is an important development in the optimization of flash and other memory-based storage technologies. Without it the industry was heading toward massive fragmentation as each vendor would have been forced to develop their own flash optimized protocols. NVMe set a standard that all vendors can adhere to, yet it still leaves room for unique vendor value add.
The journey to NVMe for most data centers starts this year and will progress until its eventual conclusion of end-to-end connectivity in a few years. NVMe should enjoy a long life, similar to that of SCSI, SMB and NFS and is a standard that IT professionals can count on.
The place to learn about NVMe is the Flash Memory Summit in Santa Clara, CA., August 8 to 10. It is one of the most educational events of its type. Whether you are developing the next great flash technology or a data center manager looking to understand how to best leverage flash, there are tracks for you.
Twelve years ago George Crump founded Storage Switzerland with one simple goal; to educate IT professionals about all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought after public speaker. With over 25 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS and SAN, Virtualization, Cloud and Enterprise Flash. Prior to founding Storage Switzerland he was CTO at one of the nation's largest storage integrators where he was in charge of technology testing, integration and product selection.
