Is PCIe SSD Always Faster?

As Solid State Disk (SSD) has evolved many ways have been developed for it to be accessed by the applications that need performance acceleration. The connection between the application and the SSD is often made by the most available, least disruptive method. The assumption is that PCIe is the option to choose when you are concerned about performance, but that is not always accurate.


First, why is PCIe assumed to be the fastest available SSD connection technology? The primary reason is that PCIe SSDs, by virtue of the PCIe bus, have essentially direct access to the CPU where the application is running and where the requests are being made. Other SSD connection methodologies have to go through some form of protocol conversion before connecting to a SATA, Fibre, or 10 GbE connection.

The other advantage to PCIe SSD is lower cost, since a separate external appliance or chassis does not need to be purchased. The PCIe card shares the power supply and housing of the server it’s in.

Why Not PCIe SSD?

However, there are limitations to PCIe. It is not always the best connection strategy, even if performance is the number one concern. First, PCIe SSD is limited to the I/O performance of a single PCIe slot. While multiple cards can be installed, more than two is uncommon. The mirroring or striping of multiple cards requires the use of software, either from the operating system or the application, which may induce latency and overhead. There are other limitations in that a single PCIe card doesn’t scale to support very small or very large amounts of data. It also can’t be shared, unless it is moved out of the server and into an appliance.

While total capacity is an obvious limitation of PCIe SSD, its inability to start small is also a problem. For example if a database only needs 25 GB for its hot files, a 100 GB PCIe SSD card would waste 75 GB of premium storage. Since the card can’t be shared the remaining 75 GB can’t be reallocated to another server. For this reason, PCIe storage is ideal for applications that have a reasonably sized performance capacity need (100 GB+) but not a large one (1 TB+), and applications that don’t require shared access.

PCIe vs. Storage System SSD

Environments that need shared access to SSD and variable SSD capacities should use a shared storage system. However, when SSDs are placed in these systems there’s a concern about achieving maximum performance. These storage systems were often designed for mechanical hard drives and contain SSD only as an afterthought. While SSD will still provide a performance advantage over mechanical hard drives, in many cases these traditional storage systems can’t deliver the full potential of SSDs.

The first problem that legacy storage systems face is that most vendors choose HDD form factor SSDs to shorten time-to-market. These are flash modules installed into a container the same size and shape as a traditional HDD. While they’re fine for single server applications, ideally for boot drives, they are not very space efficient when used in the storage system. As a result implementing flash-based storage in a legacy storage system is not as cost-effective as other methods.

The second problem that legacy storage systems face is that they provide many sophisticated storage services, like thin provisioning, snapshots, and replication. Managing these services adds significantly to the level of overhead. When these legacy storage systems supported only traditional hard drives, the latency of those rotating drives provided enough time for the storage system controller to cover the overhead of the other services. Solid state storage has no rotational latency and as a result its performance can quickly overwhelm the capabilities of a legacy storage controller.

Finally, legacy storage systems were not designed for the massively parallel I/O that flash-based storage requires to reach its full performance potential. This has led many vendors and users to blame the storage network as the bottleneck and as a result led them to use PCIe-based SSD. This also means that they have to work around the lack of ‘shareability’ inherent to PCIe SSD.

PCIe vs. Appliance SSD

Appliance-based SSDs are purpose-built devices designed solely to provide flash-based storage. They have a thin storage services layer that does not include the more sophisticated and overhead-laden services that many enterprise storage systems provide. As a result they can focus on performance and density. This brings several benefits to the storage manager and allows them to use flash-based storage more efficiently, while at the same time taking full advantage of its performance capabilities.

First, since appliance-based SSDs are shareable that means they can be more efficient at capacity provisioning. For example, if an application and/or server only has 25 GB of data that’s appropriate for flash storage, an appliance-based SSD can provide a 25 GB volume to just that application, leaving the rest of its capacity available for other servers and/or applications.

Secondly, because these appliances are not forced to use HDD form factor SSDs they can be more space efficient. This means they can have much more dense configurations, with lower power and cooling requirements.

Most importantly, these systems are designed for massively parallel I/O. This means that instead of having a few discrete I/O paths to the storage controller, like a legacy storage system would have, the SSD appliance can have multiple, shared I/O paths providing better bandwidth and throughput capabilities. For example, multiple 8 Gb Fibre Channel segments can be bonded together to deliver, in the case of Texas Memory Systems’ RamSan-630, up to 10 GB per second bandwidth. This bandwidth can challenge the raw performance of PCIe SSD without the limitations caused by the fact that it can’t be shared.


When the latency of appliance-based SSDs is compared to that of PCIe SSDs the delta is measured in microseconds, and in Texas Memory Systems’ case, less than 10 microseconds. For almost all applications in the enterprise 10 µs is not noticeable. As a result the user can get the direct-access-like performance of a PCIe SSD without having to suffer its non-sharing limitations.

There is a home in the data center for both technologies. PCIe is ideal for non-shared situations, where there is a specific capacity requirement and the cost savings of buying a single board solution outweigh the loss of some efficiency. Appliance-based SSDs on the other hand are ideal for shared application use, where the capacity requirements will vary from server to server and the additional cost of a chassis, power supply and interface cards can be compensated for by the efficiency gains inherent to a shared storage system.

The important point is that performance should not be the only consideration when selecting a PCIe-based SSD solution. What Storage Switzerland believes and what Texas Memory Systems’ testing has confirmed is that performance, when the appliance SSD is properly configured to handle parallel I/O, is essentially the same for PCIe-based and appliance-based SSDs. As a result Storage Switzerland believes that both SSD types have their roles and most data centers will eventually acquire both technologies.

Texas Memory Systems is a client of Storage Switzerland

Click Here To Sign Up For Our Newsletter

George Crump is the Chief Marketing Officer of StorONE. Prior to StorONE, George spent almost 14 years as the founder and lead analyst at Storage Switzerland, which StorONE acquired in March of 2020. In his spare time, he continues to write blogs on Storage Switzerland to educate IT professionals on all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought-after public speaker. With over 30 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS, SAN, Virtualization, Cloud, and Enterprise Flash. Prior to founding Storage Switzerland, he was CTO at one of the nation's largest storage integrators where he was in charge of technology testing, integration, and product selection.

Tagged with: , , , ,
Posted in Article

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 21,783 other followers

Blog Stats
%d bloggers like this: