What is NVMe? And what does it mean for PCIe-SSD?

There are two constants in data center storage; the need for greater performance and the need for greater capacity. Flash based storage devices have become the go-to option to address the first challenge. But application owners and users quickly move from an initial euphoria with flash performance to demanding more. Since the flash NAND is essentially the constant in the equation, the surrounding infrastructure has to evolve to extract optimal performance from the technology. But achieving maximum performance often leads to proprietary architectures and designs. NVMe (Non Volatile Memory) is a new industry standard that enables data centers to realize full flash potential without compatibility headaches.

The PCIe Challenge

Today, PCIe (Peripheral Component Interconnect Express) flash is looked at as the ideal method to achieve optimal flash performance. The latest incarnation of PCIe flash, 3.0, supports transfer rates up to 8GT/s, or just under 1GB/s of bandwidth per lane. In other words, the PCIe infrastructure is ready for flash, but up until now flash has not been ready for PCIe. The way vendors implement flash creates controversy that impacts the industry and IT professionals that try to apply it. PCIe Flash is flash NAND installed on a PCIe board, which gets installed in a PCIe socket inside a server, but PCIe is not a native storage interface. To overcome that problem vendors typically take two approaches when implementing these PCIe SSDs (Solid State Drive).

The first approach is to implement PCIe flash as a SCSI compatible device. The advantage of this approach is that it fits into the existing SCSI storage stack, requires no special drivers and has broad compatibility across operating systems. The disadvantage is that it requires proprietary drives and forces the extremely fast and low latency PCIe architecture to be bottlenecked by the SCSI interface.

The second approach is for the PCIe flash vendor to deliver a card that avoids the SCSI interface altogether. Direct access to the CPU allows for maximum performance and very low latency. But it also required that each vendor develop a proprietary driver to enable the operating system to communicate with the PCIe SSD like it was standard storage.

Despite its low latency and performance advantages, the exclusive nature of this driver caused a few problems. Again, each vendor had to create their driver for each operating system it intended to support. For most vendors this meant, at a minimum, supporting Windows, VMware, and Linux. For customers it meant having a different PCIe SSD vendor for each operating system that they used since some vendors were better at supporting some operating systems than others. It also meant that each vendor’s driver had to be tested and certified for each combination of hardware and operating system.

To solve these problems about 90 IT industry partners came together to create a standard driver that they could all adopt and support. Beyond just providing compatibility, this group also set out to improve the current SAS/SATA standards to make a driver that was more appropriate for flash storage. NVMe was the result of that effort.

What is NVMe?

NVM Express is an optimized, high performance, scalable host controller interface with a streamlined register interface and command set designed for accessing SSDs attached through the PCIe bus. It was designed from the ground up to capitalize on the low latency and parallelism of PCIe SSDs. It is also a protocol, which defines a standard for how the interface is implemented and how the software stack is structured. This enabled the development of standardized driver software for various operating systems, which eliminated the need for the custom drivers required for earlier versions of PCIe SSDs. Now every vendor’s PCIe SSD card should work on every operating system in the environment.

Vendors can focus on creating high-performance hardware and stop worrying about the nuances of supporting individual operating systems. However, the standard does allow for continued innovation and vendors are free to enhance the base functionality of NVMe in the future.

NVMe, unlike SCSI, was built from the ground up specifically to support solid state storage devices. Its streamlined memory interface, command set, and queue design suites the highly virtualized data center perfectly. NVMe also streamlines the software I/O stack by reducing the unnecessary overhead introduced by the SCSI stack. It also supports more queues than standard SCSI, increasing queues to 64,000 up from one queue that the legacy Advanced Host Controller Interface (AHCI) supports. Also, each NVMe queue can support 64,000 commands up from the 32 commands that AHCI supports in its one queue.

The NVMe Performance Advantage

NVMe brings two key benefits to the data centers that implement SSD technology based on it. The first is performance. Since an NVMe SSD does not have to support legacy protocols, its vendors can focus on taking full advantage of memory based storage. Flash, because of its low latency, can handle a multitude of I/O operations at the same time. NVMe’s 64,000 queues by 64,000 commands provide the ability to feed I/O to the device properly. This means that where a SCSI based drive must select one block of data at a time, a NVMe drive can select up to 64,000 blocks. NVMe also can access more data in the same CPU cycle, where SCSI essentially needs to use a cycle for each access.

The NVMe Compatibility Advantage

The second advantage that is probably more relevant to more data centers is the compatibility advantage that NVMe brings. There is now one software interface standard for PCIe SSD vendors to adhere to, and they don’t have to write their own. Again, this interface is designed specifically for memory based storage.

Not only is compatibility a big win for vendors, it is also a big win for IT professionals responsible for implementation. They no longer need to vet vendors based on their compatibility with a particular operating system. Instead, they can look at the specific capabilities and cost of the card to determine which is best for their environment.

Another area of compatibility is the form factor. NVMe is a standard, supporting both PCIe cards and U.2 form factor SSDs. Support for U.2 SSDs means that a storage system can be built using 2.5″ SSD drives for ease of service while not having to give up performance.

What’s Required for NVMe?

The good news is that the requirements for NVMe are very straight forward. In fact, many data centers will have servers in their environment that already support the standard, assuming those servers have PCIe slots available. Users don’t need to buy a special version of their servers because there is no change to ROM Bios or motherboard designs, other a NVMe aware BIOS for bootability. The other side of the compatibility equation is selecting a flash device that supports NVMe. Those cards are arriving at the market now. Storage Switzerland recently completed a product analysis of one such card from OCZ.


NVMe will allow flash to fulfill more of its performance promise. It leverages the PCIe bus in a standardized way and it is specifically optimized for memory based storage. Implementation should be pretty straight forward and vendors are coming to market with these solutions now. NVMe should have a significant impact on the industry and allow the data center to meet new levels of performance demands.

Article Sponsored By: OCZ Storage Solutions

About OCZ Storage Solutions

OCZ Storage Solutions, a Toshiba Company, enables enterprise customers with a portfolio of innovative flash hardware and software solutions that solve unique storage challenges by accelerating application performance, optimizing IT infrastructures, and enhancing overall reliability and manageability. This total solution approach transforms some of the most challenging database, email exchange, virtualization, VDI and cloud applications into growth opportunities for their customers. OCZ is a leader in NVMe based solutions and has been one of the first companies to market with a portfolio of solutions.

Twelve years ago George Crump founded Storage Switzerland with one simple goal; to educate IT professionals about all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought after public speaker. With over 25 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS and SAN, Virtualization, Cloud and Enterprise Flash. Prior to founding Storage Switzerland he was CTO at one of the nation's largest storage integrators where he was in charge of technology testing, integration and product selection.

Tagged with: , , , , , , , , ,
Posted in Article

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 22,223 other followers

Blog Stats
%d bloggers like this: